Image-text retrieval method for open type target and processing terminal
By combining the open object detection model DINO-X and the CLIP model, and integrating the detection results with feature similarity calculation, the problems of limited retrieval accuracy of the CLIP model and poor generalization ability of open object detection are solved, achieving efficient retrieval of everything and accurate image and text retrieval results.
Patent Information
- Application Number
- CN202511490515.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-17
- Publication Date
- 2026-03-03
AI Technical Summary
In existing technologies, the accuracy of the CLIP model for image and text retrieval is affected by the target size and background complexity. Furthermore, traditional two-stage retrieval models have poor generalization ability for open targets, resulting in inaccurate retrieval results and difficulty in adapting to dynamic category changes.
By combining the open target detection model DINO-X with the CLIP model, and fusing detection results, image features, text features, and confidence scores, similarity calculation is performed using the Faiss vector database to achieve image and text retrieval of open targets.
It enables the retrieval of almost all objects described in natural language, improving retrieval accuracy and user experience. It can dynamically adapt to category changes and reduce the cost of model training and annotation.
Smart Images
Figure CN121597859A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image and text retrieval technology, specifically to an image and text retrieval method and processing terminal capable of performing image and text retrieval on open targets. Background Technology
[0002] Retrieval using the pure CLIP model has limitations. The main limitation is that when using the CLIP model for image-text retrieval, the accuracy of the retrieval results is easily affected by the size of the target (i.e., object) in the image. For small targets or application scenarios with complex backgrounds, the retrieval accuracy is not high, which can easily lead to deviations in the retrieval results and limited retrieval effectiveness.
[0003] To address the limited retrieval results of CLIP models, existing technologies employ non-pure CLIP models, typically using a two-stage retrieval approach combined with a retrieval framework. This approach uses a CNN-based detector (such as YOLO or Faster R-CNN) for the first stage of object detection, and the cropped results are then input into the CLIP model for the second stage of retrieval. While this two-stage approach improves retrieval accuracy, it also has significant limitations. This method can only identify predefined, fixed categories, i.e., closed targets within a specified range. It has poor generalization ability for untrained categories and cannot effectively perform retrieval. This results in a cumbersome process, slow response, and high cost, making it unsuitable for real-world applications requiring retrieval of dynamically changing categories.
[0004] Therefore, there is a need for a method that can overcome the inaccuracy of traditional pure CLIP retrieval, while also overcoming the limitation of traditional retrieval models in terms of category, which prevents them from retrieving open targets. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and processing terminal for image and text retrieval of open targets, which can solve the problems described in the background art.
[0006] The technical solution to achieve the objective of this invention is: a method for image and text retrieval of open targets, comprising the following steps: Step 1: Obtain several images to be searched, and feed them into an open object detection model for retrieval. Each image yields a detection result, resulting in a set D of detection results. Elements of the result set D include the object's category, confidence score, and location within the image. For each element in the detection result D, based on the position of the target in the image represented by that element, crop out the target image set P; Step 2: Feed each image in the image set P into the CLIP model to extract the image features of the image, thereby obtaining the image features of each image. The image features of all images constitute the image feature set FI. Step 3: Input the categories of the elements in the detection result set D into the CLIP model to extract the text features of the categories. The text features of all categories constitute the category text feature set FC. Step 4: Fuse the confidence scores of the elements in the detection result set D, the image set P, the image feature set FI, and the category text feature set FC to form the image features, text features, confidence scores, and image IDs representing the images to which the elements belong, thus obtaining the fused features. Insert the fused features into the Faiss vector database. Step 5: Receive the search text input by the user and feed it into the text encoding model in the CLIP model to extract the text features FT of the search text. Then, calculate the similarity S between the search text features FT and the fused features in the Faiss vector database using the following formula:
[0007] In the formula, This represents the confidence level of an element in the detection result set D. The cosine similarity between the textual features of an element and the retrieval textual features (FT). The cosine similarity between the image features of an element and the Fourier transform (FT) of the retrieved text features; Step 6: Traverse all images to obtain the similarity S between each target in each image and the search text, thereby obtaining several similarities S under the current search text. Filter out targets with similarity S ≥ preset similarity threshold to obtain a target result set. Select N targets from the target result set sorted by similarity from largest to smallest as search results and output the search results.
[0008] Furthermore, in step 1, images are streamed from n cameras and saved as images, thus obtaining several images to be retrieved.
[0009] Furthermore, the open-ended target detection model is the DINO-X detection model.
[0010] Furthermore, the CLIP model includes an image encoding model and a text encoding model. The image encoding model is used to extract image features from images, and the image features are 512-dimensional feature values. The text encoding model is used to extract text features of categories, and the text features are 512-dimensional text feature values.
[0011] Furthermore, N=20.
[0012] A processing terminal, comprising: Memory, used to store program instructions; A processor for running the program instructions to perform the steps of the image and text retrieval method for open targets.
[0013] The beneficial effects of this invention are as follows: This invention breaks through the limitations of traditional target category retrieval. Compared to traditional detection models that can only retrieve predefined categories (such as COCO's 80 categories), this invention, through a combination of the CLIP model and an open object detection model, can detect and identify the vast majority of objects that can be described in natural language, achieving true "retrieval of everything" and improving user experience. Furthermore, this solution cleverly integrates detection confidence, text-category semantic similarity, and image-text matching degree through a similarity calculation model, effectively balancing the impact of detection quality, semantic relevance, and cross-modal consistency on retrieval results, further ensuring the accuracy of the retrieval. Attached Figure Description
[0014] Figure 1 This is a flowchart illustrating a preferred embodiment of the method of the present invention; Figure 2 This is a schematic diagram of the processing terminal. Detailed Implementation
[0015] The present invention will be further described below with reference to the accompanying drawings and specific embodiments: like Figure 1 As shown, a method for image and text retrieval of open-ended targets includes the following steps: Step 1: Obtain several images to be retrieved and send them to an open object detection model for retrieval. Each image yields a detection result, resulting in a set of detection results D. The elements of the set of detection results D include the object's category, confidence level, and location in the image.
[0016] For each element in the detection result D, based on the position of the target in the image represented by that element, crop out the target image set P.
[0017] It is understandable that the images to be retrieved can be obtained from the videos or images captured by n cameras. For example, the streams from n cameras can be saved as images to obtain several images to be retrieved.
[0018] The target can be an object, such as a pedestrian, vehicle, pet, clothing, etc. The key is that the target is precisely the result to be retrieved; that is, the target is retrieved from the image and text, and its type (pedestrian, vehicle, or other) is identified. The target type identification result is part of the retrieval results. For example, if an image contains a pedestrian and a background, the target is a pedestrian. The elements of the detection result set are the image itself, which includes the target's category (pedestrian), the confidence score for that category, and the target's location within the image.
[0019] It is understandable that the open object detection model refers to the DINO-X detection model, or other models capable of detecting open objects. This capability means that the model can effectively detect objects not included in its training, not just those within a specific range determined during training. It means the model can effectively detect objects even without prior training. The DINO-X detection model can achieve image retrieval at the level of tens of thousands of objects.
[0020] Step 2: Feed each image in the image set P into the CLIP model to extract the image features of the image, thereby obtaining the image features of each image. The image features of all images constitute the image feature set FI.
[0021] It is understandable that the elements in the image feature set FI correspond one-to-one with the elements in the detection result set D.
[0022] CLIP models include image encoding models and text encoding models. Image encoding models can extract image features from images. Image features can be 512-dimensional feature values or feature values of other dimensions.
[0023] Step 3: Input the category of the element in the detection result set D (i.e. the category to which the target belongs) into the CLIP model to extract the text features of the category. The text features of all categories constitute the category text feature set FC.
[0024] It is understandable that the elements in the category text feature set FC and the elements in the image feature set FI are in one-to-one correspondence. They are both based on the image features of each image and the text features of the category to which the image belongs, and are all associated with the same image, thus achieving a one-to-one correspondence.
[0025] Specifically, the text encoding model in the CLIP model is used to extract text features of categories, for example, extracting 512-dimensional text feature values.
[0026] Step 4: Fuse the confidence scores of the elements in the detection result set D, the image set P, the image feature set FI, and the category text feature set FC to form the image features, text features, confidence scores, and image IDs representing the images to which the elements belong, thus obtaining the fused features. Insert the fused features into the Faiss vector database.
[0027] Step 5: Receive the user's input search text (such as commonly used keywords), and feed the search text into the text encoding model in the CLIP model to extract the text features FT of the search text. Then, calculate the similarity S between the search text features FT and the fused features in the Faiss vector database according to the following formula:
[0028] In the formula, This represents the confidence level of an element in the detection result set D. The cosine similarity between the textual features of an element and the retrieval textual features (FT). This represents the cosine similarity between the image features of an element and the FT of the retrieved text features.
[0029] Step 6: Traverse all images to obtain the similarity S between each target (corresponding to an element) and the search text in each image. This will give you several similarities S under the current search text. Filter out targets with similarity S ≥ the preset similarity threshold to obtain a target result set. Select N targets from the target result set, sorted by similarity from largest to smallest, as the search results, and output the search results.
[0030] The value of N can be set as needed. For example, N=20 means selecting the top 20 targets with the highest similarity S as the search results. The target can be identified by its image ID, so the search results are output as images.
[0031] This invention breaks through the limitations of traditional target category retrieval. Compared to traditional detection models that can only retrieve predefined categories (such as COCO's 80 categories), this invention, through a combination of the CLIP model and an open object detection model, can detect and identify the vast majority of objects that can be described in natural language, achieving true "everything retrieval" and improving user experience. Furthermore, this solution cleverly integrates detection confidence, text-category semantic similarity, and image-text matching through a similarity calculation model, effectively balancing the impact of detection quality, semantic relevance, and cross-modal consistency on retrieval results, further ensuring retrieval accuracy.
[0032] like Figure 2 As shown, the present invention also provides a processing terminal 100, which includes: Memory 101 is used to store program instructions; Processor 102 is configured to run the program instructions to perform the steps of the image and text retrieval method for open targets.
[0033] The embodiments disclosed in this specification are merely illustrative of one aspect of the invention, and the scope of protection of the invention is not limited to these embodiments. Any other functionally equivalent embodiments fall within the scope of protection of the invention. Those skilled in the art can make various other corresponding changes and modifications based on the technical solutions and concepts described above, and all such changes and modifications should fall within the scope of protection of the claims of this invention.
Claims
1. A method for image and text retrieval of open-ended targets, characterized in that, Includes the following steps: Step 1: Obtain several images to be searched, and feed them into an open object detection model for retrieval. Each image yields a detection result, resulting in a set D of detection results. Elements of the result set D include the object's category, confidence score, and location within the image. For each element in the detection result D, based on the position of the target in the image represented by that element, crop out the target image set P; Step 2: Feed each image in the image set P into the CLIP model to extract the image features of the image, thereby obtaining the image features of each image. The image features of all images constitute the image feature set FI. Step 3: Input the categories of the elements in the detection result set D into the CLIP model to extract the text features of the categories. The text features of all categories constitute the category text feature set FC. Step 4: Fuse the confidence scores of the elements in the detection result set D, the image set P, the image feature set FI, and the category text feature set FC to form the image features, text features, confidence scores, and image IDs representing the images to which the elements belong, thus obtaining the fused features. Insert the fused features into the Faiss vector database. Step 5: Receive the search text input by the user and feed it into the text encoding model in the CLIP model to extract the text features FT of the search text. Then, calculate the similarity S between the search text features FT and the fused features in the Faiss vector database using the following formula: In the formula, This represents the confidence level of an element in the detection result set D. The cosine similarity between the textual features of an element and the retrieval textual features (FT). The cosine similarity between the image features of an element and the Fourier transform (FT) of the retrieved text features; Step 6: Traverse all images to obtain the similarity S between each target in each image and the search text, thereby obtaining several similarities S under the current search text. Filter out targets with similarity S ≥ preset similarity threshold to obtain a target result set. Select N targets from the target result set sorted by similarity from largest to smallest as search results and output the search results.
2. The image and text retrieval method for open-ended targets according to claim 1, characterized in that, In step 1, images are streamed from n cameras and saved as images, thus obtaining several images to be searched.
3. The image and text retrieval method for open-ended targets according to claim 1, characterized in that, The open-source target detection model is the DINO-X detection model.
4. The image and text retrieval method for open-ended targets according to claim 1, characterized in that, The CLIP model includes an image encoding model and a text encoding model. The image encoding model is used to extract image features from images, which are 512-dimensional feature values. The text encoding model is used to extract text features for categories, which are 512-dimensional text feature values.
5. The image and text retrieval method for open-ended targets according to claim 1, characterized in that, N=20。 6. A processing terminal, characterized in that, It includes: Memory, used to store program instructions; A processor for running the program instructions to perform the steps of the image and text retrieval method for open targets as described in any one of claims 1-5.