A fine-grained visual language model construction method based on component perception contrast learning

Abstract

This invention discloses a method for constructing a fine-grained visual language model based on component-aware contrastive learning, comprising: acquiring original image data and cropping it to obtain corresponding local image patches; inputting a cue template and local image patches into a pre-trained large language model to output fine-grained text descriptions; assembling the original image data, local image patches, and corresponding fine-grained text descriptions into a dataset; constructing an initial model, which includes a feature extraction module, a text generation module, and an output module; and training the initial model using the dataset to obtain a fine-grained visual language model for recognizing information in images. The method provided by this invention effectively solves the technical problems of weak local perception ability, poor interpretability, and lack of large-scale component-level supervision data in fine-grained recognition tasks of visual language pre-trained models.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to a method for constructing a fine-grained visual language model based on component perception contrastive learning. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, large-scale vision-language pre-training (VLP) models have become the cornerstone of the field of artificial intelligence, leading a paradigm shift in multimodal understanding tasks. Models represented by CLIP (Contrastive Language-Image Pre-training), ALIGN, and SigLIP have acquired powerful visual semantic representation capabilities through large-scale contrastive learning on hundreds of millions of internet image-text pairs. The core idea of ​​these models is to map images and text to the same high-dimensional feature space and use a contrastive loss function to shorten the feature distance of matched pairs and widen the feature distance of unmatched pairs.

[0003] Although existing VLP models perform well in general scenarios (such as distinguishing between "cat" and "dog", "car" and "airplane"), they show obvious limitations when dealing with tasks that require fine-grained visual understanding, mainly in the following aspects: (1) Loss of local detail perception caused by global alignment mechanism. Traditional CLIP-type models mainly learn the alignment between the global image and the global text. The training data usually comes from images crawled from the Internet and their corresponding Alt-text (alternative text). These texts are often macroscopic descriptions of the image content, such as "a picture of a dog on the grass" or "an airplane under a blue sky". This "coarse-grained" alignment method makes the model tend to focus on the prominent object outlines or background information that occupy the main pixels in the image, while ignoring the local details that are crucial for distinguishing fine-grained categories. In fine-grained visual recognition (FGVR) tasks, the differences between different subcategories are often extremely small. For example, when distinguishing between the Great Crested Flycatcher and the White-browed Chat-tyrant, two birds with very similar appearances, the key differences may lie only in the subtle textures on the wing feathers, the shape of the beak, or the distribution of color patches around the eyes. Because the text in the pre-training data lacks detailed referential descriptions of these specific parts, the model cannot obtain enough supervision signals during training to learn to pay attention to these subtle features. This leads to confusion when the model faces fine-grained classification because it cannot capture local discriminative features. (2) Lack of interpretability in model decisions. Since the model makes judgments based on global features, it is difficult to determine whether the model makes a classification decision based on the core features of the object or background noise. Related studies have used visualization techniques such as Gradient Weighted Class Activation Mapping (Grad-CAM) to find that the attention heatmap of the CLIP model is often scattered in the background (such as tree bark, grass, and blue sky) rather than the target object itself. For example, when identifying a bird species, a model might classify it as a "forest bird" simply because there are leaves in the background, rather than based on its plumage features. This "shortcut learning" phenomenon severely reduces the model's credibility, especially in high-risk applications such as medical diagnosis, autonomous driving, and security monitoring, where users cannot trust a model with the wrong focus.

[0004] Patent document CN120950741A discloses a method and apparatus for constructing a multimodal database based on visual language model collaborative routing, comprising: parsing original document data to obtain multimodal element data, and determining the element structure information corresponding to the multimodal element data; arranging the multimodal element data based on the element structure information to obtain target document data; segmenting the target document data to obtain a structured dataset, and determining target key data based on the structured data in the structured dataset; and constructing a multimodal database based on the target key data, wherein the multimodal database is used for data retrieval.

[0005] Patent document CN120517442A discloses a vehicle and a method for generating driving trajectories, an electronic device, a storage medium, and a product. The method includes inputting the bird's-eye view features of a vehicle's surround-view image into a computer vision model to obtain multiple candidate initial driving trajectories; selecting a target visual language model according to driving instructions; generating trajectory correction prompts based on each candidate initial driving trajectory, the surround-view image, driving status information, and driving instructions; and using the target visual language model to correct each candidate initial driving trajectory. The driving cost of each candidate driving trajectory is calculated based on different types of driving cost weight coefficients determined by the surround-view image. Finally, the candidate driving trajectory with the lowest driving cost is output as the vehicle's driving trajectory at the next moment. Summary of the Invention

[0006] The purpose of this invention is to provide a method for constructing a fine-grained visual language model based on component-aware contrastive learning. This method can effectively solve the technical problems of weak local perception ability, poor interpretability, and lack of large-scale component-level supervision data in fine-grained recognition tasks of visual language pre-trained models.

[0007] To achieve the objectives of this invention, the following technical solution is provided: a method for constructing a fine-grained visual language model based on component-aware contrastive learning, comprising the following steps: The original image data is acquired, and each component of the target element in the original image data is cropped to obtain the corresponding local image patch; The prompt template and local image patches are input into a pre-trained large language model to output a fine-grained text description of the content of the local image patches; The dataset consists of the original image data, local image patches, and corresponding fine-grained text descriptions. Build an initial model, which includes a feature extraction module, a text generation module, and an output module; The feature extraction module is used to crop the input raw image data to obtain corresponding local image patches, and extract global visual features and local visual features of local image patches from the raw image data. The text generation module generates corresponding global text description candidate sets and local text candidate sets based on the input original image data and local image patches, and extracts global text feature candidate sets from the global text description candidate sets and local text feature candidate sets from the local text candidate sets. The output module performs similarity matching on a global text feature candidate set based on global visual features to obtain the global text feature with the highest similarity, and performs similarity matching on a local text candidate set based on local visual features to obtain the local text feature with the highest similarity. Based on the global and local text features obtained from matching, a fine-grained text description containing both global and local text descriptions is generated. The initial model is trained using the dataset to obtain a fine-grained visual language model for recognizing information in images.

[0008] Specifically, the pre-trained large language model adopts the Gemma 3 model or Llama4 Scout model, which are open source models available online.

[0009] Specifically, the original image data includes a first dataset with local labels and a second dataset without local labels from the internet; For the image data in the first dataset, the regions are cropped using the coordinates corresponding to the local labels to obtain the corresponding local image patches; For the image data in the second dataset, an inference scan is performed using an open vocabulary object detection model to output predicted bounding boxes. The image data is then cropped using the predicted bounding boxes to obtain corresponding local image patches.

[0010] Specifically, the prompt template includes character settings, task instructions, context information, and a few sample examples.

[0011] Specifically, the feature extraction module is built using a visual Transformer architecture.

[0012] Specifically, the text generation module adopts the Transformer architecture and extends the maximum sequence length supported by its positional encoding.

[0013] Specifically, a multi-stage training strategy is adopted during the training process. The multi-stage training strategy includes two stages, the details of which are as follows: The first stage involves pre-training the initial model using a large-scale full-graph dataset containing fine-grained descriptions; The second stage involves fine-tuning the pre-trained initial model using the dataset.

[0014] Specifically, during training, global contrastive loss and component-level contrastive loss are used to adjust the model parameters.

[0015] Specifically, during training, the optimization objective is a weighted sum of the global contrastive loss and the component-level contrastive loss, expressed as follows: ; in, Indicates the first A visual feature, Indicates the first Text features, Indicates temperature parameter, Indicates the number of samples in the batch.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By explicitly introducing component-level supervision, the problem of traditional models being insensitive to subtle visual differences is effectively solved. At the same time, based on specific training strategies, the model can accurately focus on the discriminative parts of an object, rather than relying on background noise, thus providing a decision-making basis that conforms to human cognition and improving the reliability of the model. Attached Figure Description

[0017] Figure 1 This is a flowchart of the fine-grained visual language model construction method based on component perception contrastive learning provided in this embodiment; Figure 2 This is a schematic diagram of the text prompt template provided in this embodiment; Figure 3 This is a comparison diagram of the effects of the method provided in this embodiment and the traditional method. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown in this embodiment, a fine-grained visual language model construction method based on component-aware contrastive learning is provided, including the following steps: Acquire raw image data (e.g., images of birds, cars, airplanes, etc.), and crop the components of the target elements in the raw image data (e.g., bird wings, airplane engines, car logos, etc.) to obtain corresponding local image patches; The prompt template and local image patches are input into a pre-trained large language model (such as Gemma 3, Llama 4Scout) to output a fine-grained text description of the content of the local image patches (e.g., "A western meadow tyrant perches on a barbed wire fence. The bird has a grey head and back, a bright yellow belly and chest, dark grey-brown wings, a long, dark brown tail with white outer tail feathers, a small black beak, and dark eyes. The bird is slightly turned to the right, showing its side profile."). The dataset consists of the original image data, local image patches, and corresponding fine-grained text descriptions. Build an initial model, which includes a feature extraction module, a text generation module, and an output module; The feature extraction module is used to crop the input raw image data to obtain corresponding local image patches, and to extract global visual features and local visual features of local image patches from the raw image data. In this embodiment, a Vision Transformer (ViT) architecture (such as ViT-B / 16 or ViT-L / 14) is used to process the raw full image input and the cropped component image input in parallel. For each input, the Token in its output sequence is taken as the visual feature vector.

[0020] Specifically, the original image data and local image blocks are normalized in size, normalized in pixels, and divided into image blocks, respectively, and then input into the visual encoder. The visual encoder performs feature encoding on the input image, takes its global representation vector or pooled output vector, and performs linear projection and normalization processing to obtain the global visual features corresponding to the original image data and the local visual features corresponding to each local image block.

[0021] The text generation module generates corresponding global text description candidate sets and local text candidate sets based on the input raw image data and local image patches. It then extracts global text feature candidate sets from the global text description candidate sets and local text feature candidate sets from the local text candidate sets. In this embodiment, a Transformer architecture is used, and its positional embeddings support a maximum sequence length increased from the standard 77 tokens to 248 tokens or longer to adapt to the fine-grained descriptions of long texts generated by this invention. The text encoder processes three different granularities of text input: short titles, long titles containing rich context, and component descriptions targeting specific regions. That is, after processing, the input text outputs corresponding text feature vectors. All feature vectors undergo L2 normalization before similarity calculation.

[0022] Specifically, for each generated or preset text description, the text description is first converted into a word sequence by a word segmenter, then input into a text encoder for encoding, and the output vector corresponding to the text end marker or the pooled output vector is taken, and then processed by linear projection and normalization to obtain the corresponding text features.

[0023] The global text description candidate set includes short text titles and long text descriptions for the entire image, such as "A bird with a yellow belly stands on a wire fence" and "A bird with a gray head, yellow chest and belly, and dark wings perches on a wire fence"; the local text description candidate set includes local descriptions for each part of the target element, such as "The beak is dark black and short and thick" and "The wings are dark gray-brown with fine feather texture" and "The tail feathers are long and the outer feathers are white". The output module performs similarity matching on a global text feature candidate set based on global visual features to obtain the global text feature with the highest similarity, and performs similarity matching on a local text candidate set based on local visual features to obtain the local text feature with the highest similarity. Based on the global and local text features obtained from matching, a fine-grained text description containing both global and local text descriptions is generated. The initial model is trained using the dataset to obtain a fine-grained visual language model for recognizing information in images.

[0024] More specifically, for datasets with manually labeled component bounding boxes (such as CUB-200-2011, Birdsnap), local image patches of each discriminative component (such as head, wings, beak, feet, etc.) are directly cropped from the original high-resolution image based on the labeled coordinates. For datasets lacking component annotations (such as Stanford Cars, FGVC-Aircraft), a pre-trained open-vocabulary object detection model (such as OV-DINO) is used for annotation. First, a list of textual prompts containing the names of key components of the target object (such as "air intake grille", "wheel hub", "tail wing") is constructed based on domain knowledge; then, this list is input into the detection model to infer the image, automatically locate and generate predicted bounding boxes for each component; finally, local image patches of the components are cropped based on the predicted bounding boxes.

[0025] To address the lack of detailed component-level descriptions in existing datasets, this embodiment provides a data generation pipeline. The process begins with the selection and preparation of data sources, choosing publicly available datasets covering various fine-grained categories such as birds, cars, and airplanes as the base image library. These datasets include those with manually labeled components (such as CUB-200-2011) and those without (such as Stanford Cars). Next, intelligent component localization and extraction are performed, employing differentiated processing strategies for different types of data sources.

[0026] In this embodiment, the specific steps of the process are as follows: Based on the accurate extraction of prior knowledge, for datasets with manual annotation, the key point coordinates or component bounding boxes provided in their metadata are directly read. The bounding boxes are expanded by a certain proportion to retain local contextual information, and then the corresponding component image blocks are cropped from the original image.

[0027] For datasets lacking annotations, a generalized extraction based on open vocabulary detection is employed, utilizing open vocabulary object detection models (such as OV-DINO). First, a list of textual cues containing discriminative parts (e.g., bird beaks, wings, airplane engines, car logos, etc.) is constructed for each object category. This list is then input into the detection model to perform inference scanning on the images, outputting predicted bounding boxes. Finally, noise and redundant boxes are filtered by setting a confidence threshold and applying non-maximum suppression, and the part images are cropped based on the filtered coordinates. Subsequently, fine-grained description generation based on a multimodal large language model is performed, using advanced large language models (such as Gemma 3 or Llama 4) as the description generator. Structured cue templates are designed, including role settings (e.g., an ornithologist), task instructions (requiring attention to visual attributes such as color and shape), contextual information, and few-shot examples. The cropped part images and cue templates are input into the model to generate accurate textual descriptions of the parts.

[0028] Text descriptions are generated using pre-trained large language models (such as Gemma 3, Llama 4 Scout). First, a dedicated text prompt template is designed, containing "role setting" (e.g., setting the model as an ornithologist or automotive expert), "task instructions" (requiring the generation of detailed descriptions of visual attributes), "contextual information" (e.g., object category), and "few-shot examples." Then, cropped part image patches and the prompt template are input into the multimodal large language model. The model generates corresponding fine-grained text descriptions based on the visual content (e.g., "The beak is dark black, short, thick, and straight, with a sharp tip"). Finally, the part image patches are paired with the generated text descriptions to construct a part-level labeled dataset containing approximately 1 million pairs of samples. The text prompt template used in this embodiment is as follows: Figure 2 As shown.

[0029] To enable the model to simultaneously grasp both macroscopic semantics and microscopic details, this embodiment provides a joint loss function, which specifically includes: Global contrast loss calculation: For images in a batch, calculate the contrast loss (InfoNCE Loss) between the full image visual features and the short text title features, and between the full image visual features and the long text title features.

[0030] The full-image visual features are obtained by inputting the original image data into a visual encoder, followed by pooling, linear projection, and normalization. The short text title features and long text title features are obtained by inputting the corresponding short text title and long text description into a text encoder, followed by text representation extraction, linear projection, and normalization. This loss is designed to maintain the model's ability to understand the overall scene and its zero-shot generalization ability.

[0031] For each component extracted from the image, the contrast loss between its independent visual features and the corresponding generative textual description features is calculated.

[0032] In this process, the visual features of the components are obtained by inputting cropped local image patches into a visual encoder, followed by pooling, linear projection, and normalization. The generative textual description features of the components are obtained by inputting the corresponding textual descriptions into a text encoder, followed by text representation extraction, linear projection, and normalization. During this process, descriptions of other components in the same image and descriptions of components in other images are treated as negative samples. This loss-forced model closely aligns specific local visual patterns with their corresponding semantic descriptions in the feature space, thereby achieving fine-grained semantic alignment.

[0033] The overall optimization objective is a weighted sum of the global contrast loss and the component-level contrast loss. By adjusting the weight parameters, the contributions of the two are balanced. The loss function is calculated as follows: ; in, Indicates the first A visual feature, Indicates the first Text features, Indicates temperature parameter, Indicates the number of samples in the batch.

[0034] For the model training process, this embodiment adopts a phased training strategy. The first phase is global refinement pre-training, trained on a large-scale full-image dataset (such as FineHARD) containing fine-grained descriptions, using only global contrastive loss. The aim is to adapt the model to the expanded text length and initially establish awareness of fine-grained attributes. The second phase is component-aware fine-tuning, trained on the component-level image-text pair dataset constructed in this invention, using the full joint loss function, focusing on injecting component-aware capabilities. During the inference phase, the model receives the input image and performs classification or retrieval by calculating the similarity between image features and text features of candidate categories.

[0035] like Figure 3 As shown, the image recognition results generated by the method provided in this embodiment and CLIP are respectively. It can be seen that the method provided in this embodiment can accurately focus on the discriminative components of the object, thus verifying the effectiveness of the method of the present invention.

[0036] To comprehensively evaluate the method provided in this embodiment, six publicly available fine-grained perception benchmark datasets were selected, comprehensively covering various scenarios: CUB-200-2011 (Birds), which contains 200 species of birds, is mainly distinguished by minute features such as the shape of the beak and the texture of the feathers; FGVC Aircraft (Aircraft), which contains 102 aircraft models; Stanford Dogs (Dogs), which contains 120 dog breeds, mainly relying on the recognition of hair texture and ear shape; Oxford-IIIT Pets (Pets), covering 37 common cat and dog breeds; NABirds (North American Birds), a North American bird dataset, which contains 400 categories, is larger in scale and more detailed in classification than CUB, and is extremely challenging; Oxford Flowers (Flowers), which contains 102 flower species, mainly classified according to the shape of the petals and the distribution of the colors of the stamens.

[0037] The experimental comparison results are shown in Table 1, and the indicators in the table are classification accuracies.

[0038]

[0039] As shown in Table 1, under various fine-grained recognition datasets, the accuracy of the present invention exceeds the comparison method, proving the effectiveness of the fine-grained visual language model based on part-aware contrastive learning proposed by the present invention; among them, the first comparison method: the method in the literature "Learning Transferable Visual Models From Natural Language Supervision" (authors Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger and Ilya Sutskever, published in the International conference on machine learning in 2021), this method learns the representations of image-text alignment through large-scale visual language pre-training.

[0040] Comparison Method 2: The method in the paper "EVA-CLIP: Improved Training Techniques for CLIP atScale" (authors Sun, Quan and Fang, Yuxin and Wu, Ledell and Wang, Xinlong and Cao, Yue, published in arXiv PrePrint in 2023) achieves stronger performance on large-scale data by improving optimizer stability and data augmentation strategies.

[0041] Comparison Method 3: The method in the paper "Long-CLIP: Unlocking the Long-Text Capability of CLIP" (authors Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang and Jiaqi Wang, published at the 2024 European conference on computer vision), which focuses on solving the long text input limitation of CLIP by supporting longer text descriptions through extended positional encoding.

[0042] Comparison Method 4: The method in the paper "FG-CLIP: Fine-Grained Visual and Textual Alignment" (authors Xie, Chunyu, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng and Yuhui Yin, published at the International Conference on Machine Learning in 2025) uses the FineHARD dataset for training and introduces hard negative sample mining to improve the model's fine-grained perception ability.

[0043] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0044] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0045] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing a fine-grained visual language model based on component-aware contrastive learning, characterized in that, Includes the following steps: The original image data is acquired, and each component of the target element in the original image data is cropped to obtain the corresponding local image patch; The prompt template and local image patches are input into a pre-trained large language model to output a fine-grained text description of the content of the local image patches; The dataset consists of the original image data, local image patches, and corresponding fine-grained text descriptions. Build an initial model, which includes a feature extraction module, a text generation module, and an output module; The feature extraction module is used to crop the input raw image data to obtain corresponding local image patches, and extract global visual features and local visual features of local image patches from the raw image data. The text generation module generates corresponding global text description candidate sets and local text candidate sets based on the input original image data and local image patches, and extracts global text feature candidate sets from the global text description candidate sets and local text feature candidate sets from the local text candidate sets. The output module performs similarity matching on a global text feature candidate set based on global visual features to obtain the global text feature with the highest similarity, and performs similarity matching on a local text candidate set based on local visual features to obtain the local text feature with the highest similarity. Based on the global and local text features obtained from matching, a fine-grained text description containing both global and local text descriptions is generated. The initial model is trained using the dataset to obtain a fine-grained visual language model for recognizing information in images.

2. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, The original image data includes a first dataset with local labels and a second dataset without local labels from the internet; For the image data in the first dataset, the regions are cropped using the coordinates corresponding to the local labels to obtain the corresponding local image patches; For the image data in the second dataset, an inference scan is performed using an open vocabulary object detection model to output predicted bounding boxes. The image data is then cropped using the predicted bounding boxes to obtain corresponding local image patches.

3. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, The prompt template includes character settings, task instructions, context information, and a few sample examples.

4. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, The feature extraction module is built using a visual Transformer architecture.

5. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, The text generation module adopts the Transformer architecture and extends the maximum sequence length supported by its positional encoding.

6. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, A multi-stage training strategy is employed during the training process. This multi-stage training strategy comprises two stages, the details of which are as follows: The first stage involves pre-training the initial model using a large-scale full-graph dataset containing fine-grained descriptions; The second stage involves fine-tuning the pre-trained initial model using the dataset.

7. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 1, characterized in that, During training, global contrastive loss and component-level contrastive loss are used to adjust the model parameters.

8. The method for constructing a fine-grained visual language model based on component-aware contrastive learning according to claim 7, characterized in that, During training, the optimization objective is a weighted sum of the global contrastive loss and the component-level contrastive loss, expressed as follows: ;in, Indicates the first A visual feature, Indicates the first Text features, Indicates temperature parameter, Indicates the number of samples in the batch.

Images