File search method based on improved CLIP model
By introducing the dynamic language adapter LoRA module and hybrid granularity feature pyramid into the CLIP model, and combining progressive training and hard sample mining, the problems of insufficient multilingual support and visual feature extraction of the CLIP model are solved, thereby improving the retrieval accuracy and efficiency of video structuring tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-31
AI Technical Summary
The existing CLIP model has shortcomings in multilingual support, fine-grained visual feature extraction, and negative sample utilization, which limits its application in the field of video structuring.
By inserting the LoRA module, a dynamic language adapter, into the CLIP model, introducing a hybrid granularity feature pyramid, and employing progressive training and hard sample mining strategies, the model's multilingual support and training speed are improved.
It improves retrieval accuracy and efficiency in video structuring tasks, supports text search for images with custom attributes, and expands the model's ease of use and applicability.
Smart Images
Figure CN121765102A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular to a text search image method based on an improved CLIP model. Background Technology
[0002] Video structuring refers to building a structured platform for video big data based on the attributes and characteristics of people, vehicles, objects, colors, numbers, and other features presented in video footage. It is widely used in increasingly diverse scenarios such as intelligent transportation, public safety, retail analytics, and smart finance. Traditional video structured analysis result queries rely entirely on manual clicks, which is cumbersome and difficult to use. Furthermore, existing algorithms have limited support for structured attribute categories, making them difficult to extend and preventing users from customizing searches for unsupported attributes. Currently, we are exploring the use of a CLIP-based multimodal large language model to perform text-based image search (text-to-image search), allowing users to search for images with relevant attributes by inputting custom text.
[0003] The existing CLIP model suffers from three shortcomings. First, multilingual support requires training multiple models independently, resulting in low parameter efficiency. The original CLIP model was trained in English and cannot support other languages, limiting its applicability. Second, the fixed-granularity feature extractor for visual images leads to poor fine-grained retrieval performance; for example, ViT (Visual Transformer) loses local details due to global attention. Finally, traditional contrastive learning does not adequately utilize negative samples, resulting in slow convergence. These shortcomings limit the widespread application of the CLIP model in video structuring. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a text-based image search method based on an improved CLIP model. This method is applicable to tasks such as video structuring where text is used to search for images. By improving the CLIP model, the accuracy of text-based image search can be increased, and the training speed of the model can also be improved. This invention can be used in a wide range of video surveillance scenarios such as smart cities and smart parks, enabling the rapid retrieval of corresponding image results from a database based on user-inputted text content with custom attributes, significantly improving retrieval efficiency and ease of use.
[0005] The technical problem solved by this invention is achieved through the following technical solution: A text search image method based on an improved CLIP model includes the following steps: Step 1: Acquire images and related data and perform preprocessing; Step 2: Build an improved CLIP network; Step 3: Train the improved CLIP network using the preprocessed images and related data from Step 1; Step 4: Use the trained and improved CLIP network to search for images using text.
[0006] Moreover, the specific implementation method of step 1 is as follows: collect image and text samples, filter the images and text, match the text samples with the images after filtering, and construct image-text pairs.
[0007] Furthermore, the improved CLIP network in step 2 includes a text feature extraction network and an image feature extraction network. A dynamic language adapter is inserted at the top layer of the text encoder. The structure of the dynamic language adapter is a lightweight LoRA module. Multilingual support is achieved by training only the LoRA module. A hybrid granularity feature pyramid is introduced into the image feature extraction network. At the same time, ViT's [CLS] label is used to mark two sets of hybrid granular features at different levels of global and local patch mean features, corresponding to the two sets of features of fine-grained noun phrase-level embedding and coarse-grained sentence-level embedding in the text module.
[0008] Furthermore, the specific workflow for improving the CLIP network in step 2 is as follows: Step 2.1: Feed the training images into the image feature extraction network. Calculate the mean of the features of all patches in the output layer to obtain the mean feature V1. At the same time, use the [CLS] label of the output layer as the output to obtain the global feature V2. Finally, output two features, V1 and V2. Step 2.2: Input the text corresponding to the image into the text feature extraction network, and use the output of the second to last layer of the network as the input of the last layer and the LoRA layer of the original network to obtain the output features To of the original network and the output features Tm of the LoRA layer. Step 2.3: Determine the language of the text. If it is English, use the feature To output by the original network; if it is another language, use the feature Tm output by the LoRA layer. Step 2.3.1: If it is in English, extract the feature To1 of each phrase in the text from the feature To and calculate the cosine similarity with the mean feature V1 of the image. Extract the global feature To2 of the [CLS] tag from the feature To and calculate the similarity with the global feature V2 of the image. Step 2.3.2: If it is another language, extract the feature Tm1 of each phrase in the text from the feature Tm and calculate the cosine similarity with the mean feature V1 of the image. Extract the global feature Tm2 of the [CLS] tag from the feature Tm and calculate the similarity with the global feature V2 of the image.
[0009] Moreover, the specific implementation method of step 3 is as follows: initialize the parameters using the original CLIP model; Step 3.1: First, freeze the parameters of the image feature extraction network so that they are not updated, and only train the LoRA module of the text feature extraction network to achieve multilingual support; Step 3.2: Unfreeze the image feature extraction network model and fine-tune it using multilingual image-text pairs to improve the ability to extract image and text features; Step 3.3: Perform hard sample mining, select the top 10% of mismatches with the highest similarity, reduce the weight of the loss value, avoid the increase in model learning difficulty due to the presence of images with high semantic similarity in the dataset, and improve the model convergence speed.
[0010] Furthermore, the loss calculation method for the contrastive learning is as follows:
[0011] in: and This represents the features extracted by the feature extraction network from the matched image-text pairs. This indicates the calculation of the cosine similarity between two feature vectors; This represents a temperature parameter used to control the distribution of feature similarity.
[0012] The advantages and positive effects of this invention are: 1. This invention involves acquiring and preprocessing images and related data; constructing an improved CLIP network; training the improved CLIP network using the preprocessed images and related data; and using the trained improved CLIP network to perform text-based image search. This invention is applicable to tasks such as video structuring where text-based image search is suitable. By improving the CLIP model, the accuracy of text-based image search can be increased, and the training speed of the model can be improved. This invention can be used in a wide range of video surveillance scenarios such as smart cities and smart parks, enabling the rapid retrieval of corresponding image results from a database based on user-inputted custom attributes of text content, significantly improving retrieval efficiency and ease of use.
[0013] 2. Traditional structured mapping only supports a limited number of preset attributes and cannot be extended. However, the improved CLIP model of this invention can retrieve any text and supports attributes beyond those preset in the original video structured mapping device, no longer limited by the attribute categories searched.
[0014] 3. By adding the LoRA model, this invention can add multilingual support without changing the performance of the original model, greatly expanding the usability of the model and meeting the needs of users of different languages.
[0015] 4. By improving the CLIP model structure and training strategy, this invention can significantly improve the model's retrieval accuracy and training stability, shorten training time, and accelerate model iteration optimization and deployment. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the model training of the present invention; Figure 2 This is a schematic diagram of the model reasoning of the present invention. Detailed Implementation
[0017] The present invention will be further described in detail below with reference to the accompanying drawings.
[0018] like Figure 1 The invention illustrates the process of acquiring and preprocessing images and related data; constructing an improved CLIP network; training the improved CLIP network using the preprocessed images and related data; and using the trained improved CLIP network to perform text-based image search. This invention is applicable to tasks such as video structuring where text-based image search is suitable. By improving the CLIP model, the accuracy of text-based image search can be increased, and the training speed of the model can be improved. This invention can be used in a wide range of video surveillance scenarios such as smart cities and smart parks, enabling the rapid retrieval of corresponding image results from a database based on user-inputted custom attributes of text content, significantly improving retrieval efficiency and ease of use.
[0019] like Figure 2 The diagram shows the inference architecture of the CLIP model. When the CLIP model is used for text search image tasks, it first extracts the features of the input text, then compares them with the image features already extracted from the database, and finally selects the top K images with the highest similarity as the output, which are considered the retrieved results.
[0020] A text search image method based on an improved CLIP model includes the following steps: Step 1: Acquire images and related data and perform preprocessing.
[0021] The first step is to construct the training sample set. A large number of pre-collected image and text samples can be obtained by downloading open-source multimodal datasets, web scraping, or using private datasets. Text-image matching can be performed to construct over five million image-text pairs. When collecting data, priority should be given to using open-source data or data containing both text and images to improve data preparation speed. For individual texts, open-source text-to-image models can be used to generate corresponding images. For individual images, open-source multimodal large language models can be used to generate text descriptions for the images, or manual text descriptions can be added to achieve better data quality.
[0022] Step 2: Build an improved CLIP network.
[0023] The improved CLIP network includes text feature extraction and image feature extraction networks. The improvement to the text feature extraction network primarily involves inserting a dynamic language adapter at the top layer of the text encoder. Specifically, this is a lightweight LoRA (Low-Rank Adaptation) module, achieving multilingual support by training only the LoRA module. The improvement to the image feature extraction network introduces a hybrid granularity feature pyramid, utilizing two sets of hybrid granular features from different levels: ViT's [CLS] markers (global) and Patch mean features (local). These correspond to the two sets of features in the text module: noun phrase-level embedding (fine-grained) and sentence-level embedding (coarse-grained). The specific workflow is as follows: Step 2.1: Feed the training images into the image feature extraction network. Calculate the mean of the features of all patches in the output layer to obtain the mean feature V1. At the same time, use the [CLS] label of the output layer as the output to obtain the global feature V2. Finally, output two features, V1 and V2. Step 2.2: Input the text corresponding to the image into the text feature extraction network, and use the output of the second to last layer of the network as the input of the last layer and the LoRA layer of the original network to obtain the output features To of the original network and the output features Tm of the LoRA layer. Step 2.3: Determine the language of the text. If it is English, use the feature To output by the original network; if it is another language, use the feature Tm output by the LoRA layer. Step 2.3.1: If it is in English, extract the feature To1 of each phrase in the text from the feature To and calculate the cosine similarity with the mean feature V1 of the image. Extract the global feature To2 of the [CLS] tag from the feature To and calculate the similarity with the global feature V2 of the image. Step 2.3.2: If it is another language, extract the feature Tm1 of each phrase in the text from the feature Tm and calculate the cosine similarity with the mean feature V1 of the image. Extract the global feature Tm2 of the [CLS] tag from the feature Tm and calculate the similarity with the global feature V2 of the image. In this embodiment, the image feature extraction network uses the ViT-B model. The input image is 224×224, and the patch size is 16×16. Each patch is flattened into a 768-dimensional vector (16×16×3 channels). Positional encoding uses 1D learnable position embeddings (non-fixed trigonometric functions), with dimensions consistent with the patch sequence. A 12-layer Transformer encoder structure is adopted. Each layer contains two core modules: Multi-Head Self-Attention (MSA) and Multilayer Perceptron (MLP). The MSA includes 12 attention heads, each with a dimension of 64. The MLP includes two fully connected layers. The encoder uses LayerNorm normalization and residual connections to form a complete encoding structure.
[0024] The text feature extraction network employs the RoBERTa model. Its core component is also a 12-layer Transformer encoder structure. The hidden layer dimension is 512, and the number of attention heads is 12. The input vocabulary size is 49,408 (including special markers). Positional encoding uses learnable positional embeddings with a maximum length of 77. To support multilingual tasks, the vocabulary can be dynamically expanded to 250K words (supporting 100+ languages) using SentencePiece.
[0025] Step 3: Use the preprocessed images and related data from Step 1 to train the improved CLIP network.
[0026] The comparative training adopts a progressive approach, and the entire process is divided into three stages: In Phase 1, the parameters of the image feature extraction network are frozen and not updated. Only the LoRA module of the text feature extraction network is trained to achieve multilingual support. Phase 2 unfreezes the image feature extraction network model and fine-tunes it using multilingual image-text pairs to improve the ability to extract image and text features; Phase 3 involves hard sample mining, selecting the top 10% of mismatched pairs with the highest similarity to reduce the weight of the loss value. This avoids increasing the learning difficulty of the model due to the presence of images with high semantic similarity in the dataset, thereby improving the model's convergence speed.
[0027] The loss calculation method for contrastive learning is as follows:
[0028] in: and This represents the features extracted by the feature extraction network from the matched image-text pairs. This indicates the calculation of the cosine similarity between two feature vectors; This represents a temperature parameter used to control the distribution of feature similarity.
[0029] In this embodiment, the optimizer uses AdamW (a variant of Adam with weight decay), and the peak learning rate is 5e-4, which needs to be adjusted in conjunction with the learning rate scheduling strategy. A weight decay parameter of 0.2 can prevent overfitting, and the gradient clipping threshold is set to 1.0 to stabilize the training process. The batch size is set to 32,768. When using large-scale distributed training, the batch size adjustment should be balanced according to the GPU memory capacity and the number of data parallel nodes. The learning rate scheduling strategy first starts with the first 2,000 steps... Increase the learning rate to the peak value, then use cosine annealing, and after warm-up, decay the rate according to the cosine function. .
[0030] The model is trained iteratively using the BP (backpropagation) algorithm, which reduces the final loss value of the network to a low level.
[0031] Step 4: Use the trained and improved CLIP network to search for images using text.
[0032] The image feature extraction network model trained in the previous step is used to infer the images in the database, generate the feature vector of each image and store it in memory or hard disk for repeated use later.
[0033] The text feature extraction network model trained in the previous step is used to reason about the attribute text to be queried and extract the feature vector of the text.
[0034] Then, the similarity between the text's feature vector and the feature vectors of all images in the database is calculated, and the top K (e.g., 100) images with the highest similarity are selected as the output according to a preset similarity threshold (e.g., 0.5) or directly.
[0035] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.
Claims
1. A method for text-to-image generation based on an improved CLIP model, characterized in that: The method comprises the following steps: Step 1, collecting images and their related data and preprocessing; Step 2, constructing an improved CLIP network; Step 3, training the improved CLIP network using the preprocessed images and their related data in step 1; Step 4, using the trained improved CLIP network to search pictures by text.
2. The method of claim 1, wherein the method is based on an improved CLIP model. The specific implementation method of step 1 is to collect picture and text samples, screen the images and texts, match the text samples with the images after screening, and construct image-text pairs.
3. The method of claim 1, wherein the method is based on an improved CLIP model. The improved CLIP network in step 2 includes a text feature extraction network and an image feature extraction network, wherein a dynamic language adapter is inserted at the top layer of the text encoder, the structure of the dynamic language adapter is a lightweight LoRA module, and multi-language support is realized by training only the LoRA module, and a hybrid granularity feature pyramid is introduced in the image feature extraction network, and the global and local Patch mean features of the [CLS] mark of ViT are used to obtain two groups of mixed granularity features of different levels, corresponding to the two groups of features of the noun phrase level embedding fine granularity and the sentence level embedding coarse granularity of the text module.
4. The method of claim 3, wherein the method is based on an improved CLIP model. The specific workflow of the improved CLIP network in step 2 is as follows: Step 2.1, input the training picture into the picture feature extraction network, calculate the mean value of the features of all Patches at the output layer to obtain the mean value feature V1, and take the [CLS] mark of the output layer as the output to obtain the global feature V2, and finally output two features V1 and V2; Step 2.2, input the text corresponding to the picture into the text feature extraction network, take the output of the second last layer of the network as the input of the last layer of the original network and the LoRA layer to obtain the output feature To of the original network and the output feature Tm of the LoRA layer; Step 2.3, according to the language of the text, if it is English, use the feature To output by the original network, if it is other language, use the feature Tm output by the LoRA layer; Step 2.3.1, if it is English, calculate the cosine similarity between the feature To1 of each phrase in the text and the mean value feature V1 of the image, and calculate the similarity between the global feature To2 of the [CLS] mark and the global feature V2 of the image; Step 2.3.2, if it is other language, calculate the cosine similarity between the feature Tm1 of each phrase in the text and the mean value feature V1 of the image, and calculate the similarity between the global feature Tm2 of the [CLS] mark and the global feature V2 of the image.
5. The method of claim 1, wherein the method is based on an improved CLIP model. The specific implementation method of step 3 is to initialize the parameters of the original CLIP model; Step 3.1, freeze the parameters of the image feature extraction network and do not participate in the update, only train the LoRA module of the text feature extraction network to realize multi-language support; Step 3.2, unfreeze the image feature extraction network model, fine-tune the training using multi-language image-text pairs, and improve the extraction capability of image and text features; Step 3.3, difficult sample mining is performed, the top 10% most similar error pairs are selected, the weight of the loss value is reduced, and the difficulty of model learning due to the existence of images with high semantic similarity in the data set is avoided, and the convergence speed of the model is improved.
6. The method of claim 5, wherein the method is based on an improved CLIP model. The loss calculation method of the contrast learning is: ； wherein: and denote the features of the image-text pair on the match extracted by the feature extraction network; denote the cosine similarity of the two feature vectors; denote the temperature parameter for controlling the distribution of the feature similarity.