An intelligent image search method based on CLIP and image classification

Abstract

The present invention provides an intelligent image search method based on CLIP and image classification, and relates to the field of image processing technology. The method comprises: obtaining the category of the image to be searched; obtaining the N categories with the highest classification probability and the corresponding probability, and selecting the category to be detected as the prompt word for the image search; obtaining the n-dimensional image feature vector to be searched and the prompt word feature vector through CLIP model encoding; determining the category probability of the category to be detected; determining the weight of the prompt word feature vector; determining the fusion feature vector; creating an image vector warehouse, and obtaining the feature vector of each image in the image vector warehouse through CLIP model encoding; determining the cosine distance between each image in the image vector warehouse and the image to be searched; determining the image in the image vector warehouse that is most similar to the image to be searched. According to the present invention, a specific object image in the search image can be specified, and the features of part of the overall image can be retained at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an intelligent image search method based on CLIP and image classification. Background Art

[0002] In the related art, CN109376256A proposes an image search method and device, wherein the method includes: obtaining an image to be searched; extracting multi-scale features of the image to be searched; determining a hash value of the image to be searched based on the multi-scale features; comparing the multi-scale features of the image to be searched with the multi-scale features of each original image in the hash bucket to which the hash value belongs, to obtain a similar original image corresponding to the image to be searched, thereby reducing the number of original images that need to be compared, shortening the image search time, and improving the search efficiency.

[0003] CN102779157A provides a method and device for searching images, which can improve the accuracy of searching for a gallery image similar to a compared image from a gallery. The method includes: calculating the gradient histogram feature vector and the angle histogram feature vector of the compared image and the gallery image, determining the similarity between the compared image and the gallery image according to the Euclidean distance of the gradient histogram feature vector of the compared image and the gallery image and the Euclidean distance of the angle histogram feature vector of the compared image and the gallery image, and selecting a gallery image similar to the compared image from the gallery according to the similarity.

[0004] Therefore, in the related art, although the overall feature information of the query image can be extracted and compared with a large number of images, the most similar image found by the related art is only similar to the query image as a whole. In the case where a specific object in an image needs to be identified or searched, it is difficult to specify the specific object image in the search image and at the same time retain some of the features of the overall image.

[0005] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0006] The present invention provides an intelligent image search method based on CLIP and image classification, which can solve the technical problem that it is difficult to specify a specific object image in a search image and at the same time retain the characteristics of part of the overall image.

[0007] According to the present invention, an intelligent image search method based on CLIP and image classification is provided, comprising: inputting an image to be searched into a trained image classification model to obtain a category of the image to be searched; according to the category, obtaining N categories with the highest classification probability and corresponding probabilities, and selecting the category to be detected as a prompt word for image search, wherein if there is no category to be detected in the N categories, the prompt word is inputted custom, and N is a positive integer; encoding the image to be searched and the prompt word respectively through the CLIP model to obtain an n-dimensional feature vector of the image to be searched and a feature vector of the prompt word; according to the N The method comprises the following steps: determining the probability corresponding to the category, determining the category probability of the category to be detected; determining the weight of the prompt word feature vector according to the category probability; determining the fused feature vector according to the weight, the feature vector of the image to be searched and the feature vector of the prompt word; creating an image vector warehouse, and obtaining the feature vector of each image in the image vector warehouse through CLIP model encoding; determining the cosine distance between each image in the image vector warehouse and the image to be searched according to the fused feature vector and the feature vector of each image in the image vector warehouse; and determining the image in the image vector warehouse that is most similar to the image to be searched according to the cosine distance.

[0008] According to the present invention, the category probability of the category to be detected is determined according to the probabilities corresponding to the N categories, including: according to the formula Determine the category probability Softmax(z i ), where z i is the probability of the i-th category, e is the base of the natural logarithm, N is the number of categories, i≤N, and both i and N are positive integers.

[0009] According to the present invention, the weight of the prompt word feature vector is determined according to the category probability, including: according to the formula α=0.5·Softmax(z i )+0.25 determines the weight α of the prompt word feature vector, where Softmax(z i ) is the category probability of the category to be detected.

[0010] According to the present invention, determining a fusion feature vector according to the weight, the feature vector of the image to be searched and the feature vector of the prompt word includes: according to the formula v=(1-α)V image +αV class Determine the fusion feature vector v, where V image is the feature vector of the image to be searched, V class is the feature vector of the prompt word.

[0011] According to the present invention, the cosine distance between each image in the image vector warehouse and the image to be searched is determined based on the fused feature vector and the feature vector of each image in the image vector warehouse, including: obtaining the multidimensional feature value of the fused feature vector; obtaining the multidimensional feature value of the feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the fused feature vector; and determining the cosine distance between each image in the image vector warehouse and the image to be searched based on the multidimensional feature value of the fused feature vector and the multidimensional feature value of the feature vector of each image in the image vector warehouse.

[0012] According to the present invention, the cosine distance between each image in the image vector warehouse and the image to be searched is determined according to the multidimensional eigenvalue of the fused eigenvector and the multidimensional eigenvalue of the eigenvector of each image in the image vector warehouse, including: determining the cosine distance between each image in the image vector warehouse and the image to be searched according to the formula Determine the cosine distance d(v,V) between the first image in the image vector warehouse and the image to be searched I ), where V I is the feature vector of the Ith image in the image vector warehouse, v j is the j-th eigenvalue of the fused eigenvector, V j is the j-th eigenvalue of the I-th image in the image vector warehouse, n is the dimension of the fused feature vector and the feature vector of each image in the image vector warehouse, j≤n, and I, j and n are all positive integers.

[0013] According to the present invention, based on the cosine distance, the image in the image vector warehouse that is most similar to the image to be searched is determined, including: sorting the cosine distance between each image in the image vector warehouse and the image to be searched from small to large; the image ranked first is the image in the image vector warehouse that is most similar to the image to be searched.

[0014] Technical effect: According to the present invention, by using the image classification model and the CLIP model, the content of the image to be searched can be accurately identified, and the most relevant category can be selected as the search prompt word, thereby improving the accuracy of the search results. Fusion of the prompt word feature vector and the image feature vector is helpful to comprehensively consider the image content and the search prompt word, and can specify the specific object image in the search image, while retaining the characteristics of part of the overall image. When determining the category probability of the category to be detected, the category probability of the category to be detected can be determined by converting the probabilities of N categories into a probability distribution between 0 and 1, and the sum of the probabilities is 1, thereby improving the accuracy in classification and recognition. When determining the weight of the prompt word feature vector, the weight of the prompt word feature vector can be controlled between 0.25 and 0.75 to reduce the interference of the prompt word feature vector with too large or too small weight on the fused feature vector, so that the weight of the prompt word feature vector is within a reasonable range. When determining the fused feature vector, the fused feature vector can be determined by weighted summation of the feature vector of the image to be searched and the feature vector of the prompt word, so that the effect of feature fusion is better, and specific objects in the image can be identified or searched more effectively, so that the specific object image in the search image can be specified, and at the same time, the characteristics of part of the overall image can be retained to improve the quality of the search results. When determining the cosine distance between each image in the image vector warehouse and the image to be searched, the cosine distance between the image to be searched and each image in the image vector warehouse can be determined based on the vector distance between the fused feature vector and the feature vector of each image in the image vector warehouse, which is helpful for effective similarity comparison in image search and recognition, so as to find the most similar image.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of a process flow of an intelligent image search method based on CLIP and image classification according to an embodiment of the present invention is exemplarily shown;

[0018] Figure 2 An image vector warehouse creation diagram according to an embodiment of the present invention is exemplarily shown;

[0019] Figure 3 The basic framework diagram of the intelligent image search method based on CLIP and image classification according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 A flow chart of an intelligent image search method based on CLIP and image classification according to an embodiment of the present invention is exemplarily shown, the method comprising: step S101, inputting the image to be searched into a trained image classification model to obtain the category of the image to be searched; step S102, according to the category, obtaining N categories with the highest classification probability and the corresponding probabilities, and selecting the category to be detected as the prompt word for image search, wherein if there is no category to be detected in the N categories, the prompt word is inputted by user, and N is a positive integer; step S103, encoding the image to be searched and the prompt word respectively through the CLIP model to obtain an n-dimensional feature vector of the image to be searched and a feature vector of the prompt word; step S104, according to The probabilities corresponding to the N categories are used to determine the category probability of the category to be detected; step S105, according to the category probability, determine the weight of the prompt word feature vector; step S106, according to the weight, the feature vector of the image to be searched and the feature vector of the prompt word, determine the fused feature vector; step S107, create an image vector warehouse, and obtain the feature vector of each image in the image vector warehouse through CLIP model encoding; step S108, according to the fused feature vector and the feature vector of each image in the image vector warehouse, determine the cosine distance between each image in the image vector warehouse and the image to be searched; step S109, according to the cosine distance, determine the image in the image vector warehouse that is most similar to the image to be searched.

[0023] According to the intelligent image search method based on CLIP and image classification according to the embodiment of the present invention, by using the image classification model and the CLIP model, the content of the image to be searched can be accurately identified, and the most relevant category can be selected as the search prompt word, thereby improving the accuracy of the search results. Fusion of the prompt word feature vector and the image feature vector helps to comprehensively consider the image content and the search prompt word, and can specify a specific object image in the search image while retaining the characteristics of part of the overall image.

[0024] According to an embodiment of the present invention, in step S101, the classification of the image to be searched may be performed using a ResNet image classification model trained using an images dataset to classify the image to be searched.

[0025] According to an embodiment of the present invention, in step S102, the category to be detected is selected from the N categories with the highest classification probability as a prompt word for image search. If the category to be detected is not included in the N categories, the user defines the prompt word.

[0026] According to one embodiment of the present invention, in step S103, CLIP is a multimodal model that encodes the text and image of the prompt word respectively and converts them into feature vectors. For example, using Image Embedding and Text Embedding of the CLIP model, Image Embedding converts the image to be searched into an n-dimensional feature vector of the image to be searched, and Text Embedding converts the prompt word into an n-dimensional feature vector of the prompt word.

[0027] According to one embodiment of the present invention, in step S104, a new probability, namely, a category probability, is calculated using a Softmax function, which can convert the probabilities of N categories into a probability distribution between 0 and 1, and the sum of the probabilities is 1, so as to further process and analyze the search image.

[0028] According to an embodiment of the present invention, determining the category probability of the category to be detected according to the probabilities corresponding to the N categories includes: determining the category probability Softmax(z i ),

[0029]

[0030] Among them, z i is the probability of the i-th category, e is the base of the natural logarithm, N is the number of categories, i≤N, and both i and N are positive integers.

[0031] According to one embodiment of the present invention, in formula (1), It is the ratio of the exponential function value of the probability of the i-th category to the sum of the exponential function values ​​of the probabilities corresponding to the N categories, that is, the category probability of the category to be detected, which is a number between 0 and 1.

[0032] In this way, the probabilities of N categories can be converted into a probability distribution between 0 and 1, and the sum of the probabilities is 1, so as to determine the category probability of the category to be detected, thereby improving the accuracy in classification and recognition.

[0033] According to an embodiment of the present invention, in step S105, the weight of the prompt word feature vector is determined according to the category probability, and the weight may reflect the importance or relevance of the prompt word feature vector.

[0034] According to an embodiment of the present invention, determining the weight of the prompt word feature vector according to the category probability includes: determining the weight α of the prompt word feature vector according to formula (2),

[0035] α=0.5·Softmax(z i )+0.25 (2)

[0036] Among them, Softmax(z i ) is the category probability of the category to be detected.

[0037] According to one embodiment of the present invention, in formula (2), 0.5·Softmax(z i )+0.25 means that the category probability range of the category to be detected is controlled between 0.25 and 0.75 as the weight of the prompt word feature vector. The higher the category probability of the category to be detected, the higher the proportion of the prompt word feature vector.

[0038] In this way, the weight of the prompt word feature vector can be controlled between 0.25 and 0.75 to reduce the interference of the prompt word feature vector with too large or too small weight on the fusion feature vector, so that the weight of the prompt word feature vector is within a reasonable range.

[0039] According to one embodiment of the present invention, in step S106, a fused feature vector is determined based on the weight, the feature vector of the image to be searched and the prompt word feature vector, that is, the prompt word feature vector and the feature vector of the image to be searched are fused by the weight to obtain a fused feature vector.

[0040] According to an embodiment of the present invention, determining a fused feature vector according to the weight, the feature vector of the image to be searched and the feature vector of the prompt word includes: determining the fused feature vector v according to formula (3),

[0041] v=(1-α)V image+αV class (3)

[0042] Among them, V image is the feature vector of the image to be searched, V class is the feature vector of the prompt word.

[0043] According to one embodiment of the present invention, in formula (3), (1-α)V image +αV class It represents the result of weighted summation of the feature vector of the image to be searched and the feature vector of the prompt word, that is, the fused feature vector. For example, if the weight is 1, the fused feature vector is the feature vector of the prompt word, and the features of the image to be searched are not retained. If the weight is 0, the fused feature vector is the feature vector of the image to be searched, which is the traditional image search method. Therefore, the weight is controlled between 0.25 and 0.75.

[0044] In this way, the fused feature vector can be determined by weighted summation of the feature vector of the image to be searched and the feature vector of the prompt word, so that the effect of feature fusion is better and specific objects in the image can be identified or searched more effectively. Therefore, the image of a specific object in the search image can be specified and at the same time, some features of the overall image can be retained, thereby improving the quality of the search results.

[0045] According to one embodiment of the present invention, in step S107, Figure 2 The image vector warehouse creation diagram according to an embodiment of the present invention is exemplarily shown. The purpose of creating the image vector warehouse is to effectively store and search for images. By extracting the feature vector of the image and recording its path, efficient image retrieval can be achieved. The feature vector of the image is extracted using CLIP's Image Embedding to obtain an n-dimensional feature vector for each image. After extracting the feature vector of the image, the feature vector is recorded, and a mapping relationship between the image path and the feature vector is established for subsequent retrieval and matching.

[0046] According to an embodiment of the present invention, in step S108, the cosine distance can be used to calculate the distance between two vectors. By calculating the cosine distance, the similarity value between each image in the image vector warehouse and the image to be searched can be obtained.

[0047] According to one embodiment of the present invention, step S108 includes: obtaining the multidimensional feature value of the fused feature vector; obtaining the multidimensional feature value of the feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the fused feature vector; determining the cosine distance between each image in the image vector warehouse and the image to be searched based on the multidimensional feature value of the fused feature vector and the multidimensional feature value of the feature vector of each image in the image vector warehouse.

[0048] According to one embodiment of the present invention, the cosine distance between each image in the image vector warehouse and the image to be searched is determined according to the multidimensional eigenvalue of the fused feature vector and the multidimensional eigenvalue of the feature vector of each image in the image vector warehouse, including: determining the cosine distance d(v,V) between the Ith image in the image vector warehouse and the image to be searched according to formula (4): I ),

[0049]

[0050] Among them, V I is the feature vector of the Ith image in the image vector warehouse, v j is the j-th eigenvalue of the fused eigenvector, V j is the j-th eigenvalue of the I-th image in the image vector warehouse, n is the dimension of the fused feature vector and the feature vector of each image in the image vector warehouse, j≤n, and I, j and n are all positive integers.

[0051] According to one embodiment of the present invention, in formula (4), is the cosine distance between the feature vector of the Ith image in the image vector warehouse and the fused feature vector. The smaller the cosine distance, the more similar the two feature vectors are. is the numerical calculation of the cosine distance, It represents the sum of the eigenvalues ​​of the corresponding dimensions of the feature vector of the image to be searched and the feature vector of the Ith image in the image vector warehouse. It is the product of the feature vector of the image to be searched and the modulus of the feature vector of the image in the image vector warehouse.

[0052] In this way, the cosine distance between the image to be searched and each image in the image vector warehouse can be determined based on the vector distance between the fused feature vector and the feature vector of each image in the image vector warehouse, which helps to perform effective similarity comparison in image search and recognition, thereby finding the most similar image.

[0053] According to an embodiment of the present invention, in step S109, the images may be sorted or screened by cosine distance, so as to find the image in the image vector warehouse that is most similar to the image to be searched.

[0054] According to an embodiment of the present invention, step S109 includes: sorting the co-line distance between each image in the image vector warehouse and the image to be searched from small to large; the image ranked first is the image in the image vector warehouse that is most similar to the image to be searched.

[0055] According to one embodiment of the present invention, the image ranked first in the sorting list, i.e., the image with the smallest cosine distance, is the image in the image vector warehouse that is most similar to the image to be searched. This image has the highest similarity with the image to be searched and can be regarded as the most matching image.

[0056] According to the intelligent image search method based on CLIP and image classification of the embodiment of the present invention, by using the image classification model and the CLIP model, the content of the image to be searched can be accurately identified, and the most relevant category can be selected as the search prompt word, thereby improving the accuracy of the search results. The prompt word feature vector and the image feature vector are fused to facilitate comprehensive consideration of the image content and the search prompt word, and a specific object image in the search image can be specified, while retaining the characteristics of part of the overall image. When determining the category probability of the category to be detected, the category probability of the category to be detected can be determined by converting the probabilities of N categories into a probability distribution between 0 and 1, and the probability sum is 1, thereby improving the accuracy in classification and recognition. When determining the weight of the prompt word feature vector, the weight of the prompt word feature vector can be controlled between 0.25 and 0.75 to reduce the interference of the prompt word feature vector with too large or too small weight on the fusion feature vector, so that the weight of the prompt word feature vector is within a reasonable range. When determining the fused feature vector, the fused feature vector can be determined by weighted summation of the feature vector of the image to be searched and the feature vector of the prompt word, so that the effect of feature fusion is better, and specific objects in the image can be identified or searched more effectively, so that the specific object image in the search image can be specified, and at the same time, the characteristics of part of the overall image can be retained to improve the quality of the search results. When determining the cosine distance between each image in the image vector warehouse and the image to be searched, the cosine distance between the image to be searched and each image in the image vector warehouse can be determined based on the vector distance between the fused feature vector and the feature vector of each image in the image vector warehouse, which is helpful for effective similarity comparison in image search and recognition, so as to find the most similar image.

[0057] Figure 3 The basic framework diagram of the intelligent image search method based on CLIP and image classification according to an embodiment of the present invention is exemplarily shown. The prompt word is obtained using the image classification model, and the CLIP multimodal model is used to perform a comprehensive image search of the prompt word and the picture.

[0058] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims

1. An intelligent image search method based on CLIP and image classification, characterized in that: include: Input the image to be searched into the trained image classification model to obtain the category of the image to be searched; According to the categories, N categories with the highest classification probabilities and corresponding probabilities are obtained, and the categories to be detected are selected as prompt words for image search, wherein if there is no category to be detected among the N categories, the prompt words are inputted by user, and N is a positive integer; Encode the image to be searched and the prompt word through the CLIP model respectively to obtain an n-dimensional feature vector of the image to be searched and a feature vector of the prompt word; Determine the category probability of the category to be detected according to the probabilities corresponding to the N categories; Determining the weight of the prompt word feature vector according to the category probability; Determine a fusion feature vector according to the weight, the feature vector of the image to be searched and the feature vector of the prompt word; Create an image vector warehouse and obtain the feature vector of each image in the image vector warehouse through CLIP model encoding; Determine the cosine distance between each image in the image vector warehouse and the image to be searched according to the fused feature vector and the feature vector of each image in the image vector warehouse; According to the cosine distance, an image in the image vector warehouse that is most similar to the image to be searched is determined.

2. The intelligent image search method based on CLIP and image classification according to claim 1, characterized in that: Determining the category probability of the category to be detected according to the probabilities corresponding to the N categories includes: According to the formula Determine the category probability Softmax(z i ), where z i is the probability of the i-th category, e is the base of the natural logarithm, N is the number of categories, i≤N, and both i and N are positive integers.

3. The intelligent image search method based on CLIP and image classification according to claim 1, characterized in that: Determining the weight of the prompt word feature vector according to the category probability includes: According to the formula α=0.5·Softmax(z i )+0.25 Determine the weight α of the prompt word feature vector, where Softmax(z i ) is the category probability of the category to be detected.

4. The intelligent image search method based on CLIP and image classification according to claim 1, characterized in that: Determining a fusion feature vector according to the weight, the feature vector of the image to be searched, and the feature vector of the prompt word includes: According to the formula v=(1-α)V image +αV class Determine the fusion feature vector v, where V image is the feature vector of the image to be searched, V class is the feature vector of the prompt word.

5. The intelligent image search method based on CLIP and image classification according to claim 1, characterized in that: Determining the cosine distance between each image in the image vector warehouse and the image to be searched according to the fused feature vector and the feature vector of each image in the image vector warehouse, including: Obtaining multidimensional eigenvalues ​​of the fused eigenvector; Acquire a multidimensional feature value of a feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the fused feature vector; The cosine distance between each image in the image vector warehouse and the image to be searched is determined according to the multidimensional eigenvalue of the fused eigenvalue and the multidimensional eigenvalue of the eigenvector of each image in the image vector warehouse.

6. The intelligent image search method based on CLIP and image classification according to claim 5, characterized in that: Determining the cosine distance between each image in the image vector warehouse and the image to be searched according to the multidimensional eigenvalue of the fused eigenvector and the multidimensional eigenvalue of the eigenvector of each image in the image vector warehouse includes: According to the formula Determine the cosine distance d(v,V) between the first image in the image vector warehouse and the image to be searched I ), where V I is the feature vector of the Ith image in the image vector warehouse, v j is the j-th eigenvalue of the fused eigenvector, V j is the j-th eigenvalue of the I-th image in the image vector warehouse, n is the dimension of the fused feature vector and the feature vector of each image in the image vector warehouse, j≤n, and I, j and n are all positive integers.

7. The intelligent image search method based on CLIP and image classification according to claim 1, characterized in that: Determining the image in the image vector warehouse that is most similar to the image to be searched according to the cosine distance includes: Sorting the co-line distance between each image in the image vector warehouse and the image to be searched from small to large; The image ranked first is the image in the image vector warehouse that is most similar to the image to be searched.

Images