Zero sample target detection algorithm based on binary search and CLIP

Abstract

The invention discloses a zero sample target detection algorithm based on binary search and CLIP, and belongs to the technical field of image target detection, and the algorithm specifically comprises the steps: constructing a data set training CLIP model, carrying out the preprocessing of a target image, setting an initial range of boundary search, inputting the preprocessed target image into the CLIP model, and carrying out the recognition of the target image. The method comprises the following steps: determining a primary bounding box of a target by using binary search and a pad method, calculating the similarity between a target image and a target text by using a CLIP model, generating text description information of the target, adjusting and optimizing the primary bounding box in combination with position information of the target in the image, and generating a final bounding box. The method has remarkable advantages in the aspects of processing new category target detection, improving the calculation efficiency and reducing the manual annotation cost.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a zero-sample target detection algorithm based on binary search and CLIP. Background Art

[0002] With the rapid development of deep learning and computer vision technology, image object detection has become an important branch of artificial intelligence research. Image object detection technology can be used in many fields such as security, transportation, and medical care.

[0003] Traditional image target detection technologies include the Faster R-CNN method and the YOLO method. Both methods have excellent performance in the speed and accuracy of target detection and recognition in multi-target detection and complex scene recognition.

[0004] However, both Faster R-CNN and YOLO rely on a large amount of image annotation data for training. As the application scenarios become more complicated, especially when image annotation data is scarce or when processing new categories of targets that have never been seen, the target detection effect of traditional methods is poor and has great limitations. Therefore, the present invention proposes a zero-shot target detection algorithm based on binary search and CLIP, which can achieve efficient and accurate target detection without relying on large-scale annotated data. Summary of the invention

[0005] The purpose of the present invention is to provide a zero-sample target detection algorithm based on binary search and CLIP to solve the following technical problems:

[0006] Traditional image target detection methods, such as the Faster R-CNN method and the YOLO method, rely heavily on a large amount of image annotation data to train the model. Therefore, when faced with situations where image annotation data is scarce or new categories of targets are to be recognized, the detection efficiency and accuracy are not ideal and have certain limitations.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A zero-shot target detection algorithm based on binary search and CLIP includes the following steps:

[0009] S1: Collect sample images and sample texts to build a dataset, use CLIP's image encoder and text encoder to extract the embedding vectors of sample images and sample texts respectively, and perform image-text joint training on the CLIP model based on the embedding vectors;

[0010] S2: Capture the target image and preprocess the target image, set the initial range of boundary search for the target image, input the preprocessed target image into the trained CLIP model, and use binary search and paint method to determine the primary bounding box of the target according to the initial range;

[0011] S3: The CLIP model calculates the similarity between the target image and the target text, and generates text description information of the target. It makes a comprehensive judgment on the text description information of the target and the location information of the target in the image, adjusts the primary bounding box, and generates the final bounding box of the target.

[0012] As a further solution of the present invention: the specific steps included in S1 are:

[0013] The dataset contains N pairs of real sample pairs, each of which consists of a sample image and a unique correctly matched sample text;

[0014] For each pair of sample images and sample texts, the image encoder and text encoder are used to generate their own embedding vectors, and the cosine similarity of the two embedding vectors is calculated to quantify the similarity between the sample images and sample texts.

[0015] The data set also contains N 2 -N wrong sample pairs, CLIP model maximizes the cosine similarity of image embedding and text embedding of N pairs of real sample pairs, while minimizing the cosine similarity of N pairs of real sample pairs 2 - Cosine similarity of image embedding and text embedding of N pairs of error samples.

[0016] As a further solution of the present invention: the specific process of preprocessing the target image in S2 is:

[0017] The preprocessing includes image size unification, color space conversion and image batch processing;

[0018] The method for unifying the image size is: converting all the acquired target images into the same size matching the CLIP model through scaling and cropping operations;

[0019] The color space conversion method is: uniformly converting the color space of the target image into the RGB space;

[0020] The image batch processing method is: after unifying the sizes of all target images, the target images are packaged in different batches, and each batch contains m target images.

[0021] As a further solution of the present invention: the specific process of using binary search in S2 is:

[0022] The initial range includes boundary ranges in four directions: up, down, left, and right. The target image is segmented step by step in the four directions according to the initial range to generate segmented regions, and a binary search is performed on each segmented region.

[0023] Set the similarity threshold and calculate the similarity between each segmented region and the target through the CLIP model;

[0024] If the similarity between the segmented region and the target exceeds the threshold, the segmented region is judged to contain the target and further segmentation search is performed on the segmented region. Otherwise, the segmented region is judged to contain no target.

[0025] By continuously adjusting the upper, lower, left, and right boundary ranges, the actual position of the target is gradually approached and framed, and a primary bounding box is generated.

[0026] As a further solution of the present invention: the specific process of using the paint method in S2 is:

[0027] When performing a binary search on the target image, if it is determined that a segmented area does not contain the target, the paintmethod is used to set the pixel value of the segmented area to black, marking it as a blackened area, and the blackened area will no longer participate in the similarity calculation in subsequent segmentation searches.

[0028] As a further solution of the present invention: the process of generating the final bounding box in S3 is:

[0029] The CLIP model uses an image encoder to extract the embedding vector of the target image in the segmented area, and determines the category of the target in the target image by comparing the similarity between the embedding vector and the embedding vector of the target text, and generates text description information of the target;

[0030] Based on the text description information of the target and the comprehensive judgment of the location information of the target in the image, the primary bounding box is gradually adjusted and optimized, and the final bounding box of the target is generated.

[0031] Beneficial effects of the present invention:

[0032] The present invention utilizes the zero-sample learning characteristics of the CLIP model and the joint embedding learning ability between images and texts to overcome the shortcomings of traditional target detection methods that are extremely dependent on large-scale data annotation. In the case of scarce image annotation data or new category target detection, the present invention can still achieve high-precision detection, and the zero-sample learning ability of the model effectively expands the scope of application of target detection. The present invention introduces a binary search method to efficiently search for the boundary of the target object, and quickly locks the boundary of the target by gradually narrowing the search range. Compared with the traditional sliding window or RPN method, this method significantly reduces the computational overhead and improves the efficiency and accuracy of boundary detection. At the same time, the present invention uses the paintmethod method to deal with the boundary problem, ensuring that the cropped image remains monotonic, avoiding the errors caused by direct cropping, and further improving the accuracy of boundary determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with the accompanying drawings.

[0034] Figure 1 It is a schematic diagram of the process of the present invention;

[0035] Figure 2 It is the overall algorithm flow chart of target image boundary annotation. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1-Figure 2 As shown, the present invention is a zero-sample target detection algorithm based on binary search and CLIP, comprising the following steps:

[0038] S1: Construct a dataset to train the CLIP model. The dataset contains N pairs of samples. Each pair of samples consists of a sample image and a unique matching sample text. Each pair of sample images and sample texts generates their own embedding vectors through an image encoder and a text encoder respectively. The similarity between the sample image and the sample text is quantified by calculating the cosine similarity of the two embedding vectors.

[0039] In the data set, there are N pairs of real samples and N 2 -N wrong sample pairs, CLIP model maximizes the cosine similarity of image embedding and text embedding of N pairs of real sample pairs, while minimizing the cosine similarity of N pairs of real sample pairs 2-The cosine similarity of the image embedding and text embedding of N pairs of error samples is used to complete the joint training of image and text.

[0040] S2: Acquire the target image and preprocess the target image. The preprocessing includes image size unification, color space conversion and image batch processing. The specific process of preprocessing is as follows:

[0041] The method for unifying the image size is: converting all the acquired target images into the same size matching the CLIP model through scaling and cropping operations;

[0042] The color space conversion method is: uniformly converting the color space of the target image into the RGB space;

[0043] The image batch processing method is: after unifying the sizes of all target images, the target images are packaged in different batches, and each batch contains m target images.

[0044] The initial range of boundary search is set for the target image. The initial range includes the boundary ranges in four directions: up, down, left, and right. The target image is segmented step by step in the four directions according to the initial range to generate segmented regions. For each segmented region, the primary bounding box is determined by binary search and paint method. The specific process is as follows:

[0045] Set the similarity threshold and calculate the similarity between each segmented region and the target through the CLIP model;

[0046] If the similarity between the segmented area and the target exceeds the threshold, the segmented area is judged to contain the target and further segmentation search is performed on the segmented area. Otherwise, the segmented area is judged to contain no target. For the segmented area without the target, the pixel value of the segmented area is set to black using the paint method and marked as a blackened area. The blackened area will no longer participate in the subsequent similarity calculation;

[0047] By continuously adjusting the upper, lower, left, and right boundary ranges, the actual position of the target is gradually approached and framed, and a primary bounding box is generated.

[0048] The advantages of binary search and paint method are that they can reduce the interference of irrelevant areas on the judgment of CLIP model and improve the model's attention to the actual target area. Since the effective search area will be reduced after each iteration, the search efficiency is improved.

[0049] When performing the blackening operation, you can choose pixel values ​​of other colors for the blackening operation, but you need to ensure that the blackened pixel values ​​do not affect the normal detection and evaluation of the model on the remaining area.

[0050] S3: The CLIP model uses an image encoder to extract the embedding vector of the target image in the segmented area, and determines the category of the target in the target image by comparing the similarity between the embedding vector and the embedding vector of the target text, and generates text description information of the target;

[0051] According to the text description information of the target and the comprehensive judgment of the position information of the target in the image, the primary bounding box is gradually adjusted and optimized. The optimization algorithm can use algorithms such as gradient descent to generate the final bounding box of the target.

[0052] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0053] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0054] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A zero-shot target detection algorithm based on binary search and CLIP, characterized in that: The following steps are involved: S1: Collect sample images and sample texts to build a dataset, use CLIP's image encoder and text encoder to extract the embedding vectors of sample images and sample texts respectively, and perform image-text joint training on the CLIP model based on the embedding vectors; S2: Capture the target image and preprocess the target image, set the initial range of boundary search for the target image, input the preprocessed target image into the trained CLIP model, and use binary search and paintmethod to determine the primary bounding box of the target according to the initial range; S3: The CLIP model calculates the similarity between the target image and the target text, and generates text description information of the target. It makes a comprehensive judgment on the text description information of the target and the location information of the target in the image, adjusts the primary bounding box, and generates the final bounding box of the target.

2. The zero-shot target detection algorithm based on binary search and CLIP according to claim 1, characterized in that: The specific steps of training the CLIP model in S1 are: The dataset contains N pairs of real sample pairs, each of which consists of a sample image and a unique correctly matched sample text; For each pair of sample images and sample texts, the image encoder and text encoder are used to generate their own embedding vectors, and the cosine similarity of the two embedding vectors is calculated to quantify the similarity between the sample images and sample texts. The data set also contains N 2 -N wrong sample pairs, CLIP model maximizes the cosine similarity of image embedding and text embedding of N pairs of real sample pairs, while minimizing the cosine similarity of N pairs of real sample pairs 2 - Cosine similarity of image embedding and text embedding of N pairs of error samples.

3. The zero-shot target detection algorithm based on binary search and CLIP according to claim 1, characterized in that: The specific process of preprocessing the target image in S2 is: The preprocessing includes image size unification, color space conversion and image batch processing; The method for unifying the image size is: converting all the acquired target images into the same size matching the CLIP model through scaling and cropping operations; The color space conversion method is: uniformly converting the color space of the target image into the RGB space; The image batch processing method is: after unifying the sizes of all target images, the target images are packaged in different batches, and each batch contains m target images.

4. The zero-shot target detection algorithm based on binary search and CLIP according to claim 3, characterized in that: The specific process of using binary search in S2 is: The initial range includes boundary ranges in four directions: up, down, left, and right. The target image is segmented step by step in the four directions according to the initial range to generate segmented regions, and a binary search is performed on each segmented region. Set the similarity threshold and calculate the similarity between each segmented region and the target through the CLIP model; If the similarity between the segmented region and the target exceeds the threshold, the segmented region is judged to contain the target and further segmentation search is performed on the segmented region. Otherwise, the segmented region is judged to contain no target. By continuously adjusting the upper, lower, left, and right boundary ranges, the actual position of the target is gradually approached and framed, and a primary bounding box is generated.

5. The zero-shot target detection algorithm based on binary search and CLIP according to claim 4, characterized in that: The specific process of using the paint method in S2 is: When performing a binary search on the target image, if it is determined that a segmented area does not contain the target, the paint method is used to set the pixel value of the segmented area to black and mark it as a blackened area. The blackened area will no longer participate in the similarity calculation in subsequent segmentation searches.

6. The zero-shot target detection algorithm based on binary search and CLIP according to claim 4, characterized in that: The process of generating the final bounding box in S3 is: The CLIP model uses an image encoder to extract the embedding vector of the target image in the segmented area, and determines the category of the target in the target image by comparing the similarity between the embedding vector and the embedding vector of the target text, and generates text description information of the target; Based on the text description information of the target and the comprehensive judgment of the location information of the target in the image, the primary bounding box is gradually adjusted and optimized, and the final bounding box of the target is generated.

Images