SAM-based intelligent image searching method
Through SAM segmentation model and Gaussian convolution filter blur processing, the problem of difficult to highlight the detailed characteristics of the search object in complex backgrounds is solved, and the effect of improving accuracy and efficiency in image search is achieved.
Patent Information
- Application Number
- CN202410945548.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-15
AI Technical Summary
The prior art is difficult to highlight the detailed features of the search object in a complex background and retain part of the features of the surrounding objects, resulting in a decrease in search accuracy and efficiency.
Using the intelligent SAM-based graph search method, panoramic segmentation of the search image through the SAM segmentation model, select the target object, and generate the object part clear surrounding object blur or clear surrounding black according to the segmentation mask and Gaussian convolution filter blur process. The image of the target object extracts the feature vector of the target object, and calculates the cosine distance from the target object in the image vector warehouse to find the most similar image.
It improves the recognition and accuracy of the search object, and can search for specific objects more effectively in complex backgrounds, reduces interference with surrounding objects, highlights the detailed characteristics of the search object, and retains some of the characteristics of the surrounding objects.
Smart Images

Figure CN119066220A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an intelligent image search method based on SAM. Background Art
[0002] In the related art, CN116204671A provides an image search method and device, which relates to the field of artificial intelligence technology, specifically, to the technical fields of computer vision, image processing, deep learning, etc. The image search method disclosed herein includes: performing an image search on an image to be searched to obtain a first search result from a first database; extracting text information from the image to be searched; performing a text search based on the extracted text information to obtain a second search result from a second database; and outputting a search result for the image to be searched based on the first search result and the second search result.
[0003] CN115248871A provides an image search method and device, the method comprising: obtaining a decoupled feature vector of a face image to be searched; inputting the decoupled feature vector into a target classifier to obtain the feature category to which the face image to be searched belongs; determining the first middle-level feature vector of the face image to be searched according to the decoupled feature vector and the feature category to which the face image to be searched belongs; matching the first middle-level feature vector with the second middle-level feature vector of each image in the image set to be searched to obtain a target face image corresponding to the face image to be searched. This invention obtains the decoupled feature vector of the face image, constructs a series of middle-level features based on the decoupled feature vector, and uses them for face search, thereby solving the problems in the prior art that when the amount of face data increases, the face search efficiency drops sharply, and the search accuracy is caused by key point occlusion and inaccurate key point positioning, thereby improving the speed and accuracy of face search.
[0004] Therefore, in the related art, although the similarity between the overall feature vector of the query image and the vector in the database can be calculated to find the most similar image, the overall feature vector cannot accurately capture the details of the target object. When searching for a specific object in a complex background, it is difficult to highlight the detailed features of the search object and retain some features of the surrounding objects.
[0005] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0006] The present invention provides an intelligent image search method based on SAM, which can solve the technical problem that it is difficult to highlight the detailed features of the search object and retain part of the features of the surrounding objects.
[0007] According to the present invention, a SAM-based intelligent image search method is provided, comprising: inputting an image to be searched into a SAM segmentation model to obtain segmentation masks of all objects in the image to be searched, wherein the segmentation mask and the image to be searched are matrices of the same size, and the elements of the segmentation mask matrix are 0 or 1, 0 being the background and 1 being the target object; selecting an object to be searched according to the object to be searched to obtain a segmentation mask corresponding to the search object image; determining a target object image according to the image to be searched and the segmentation mask corresponding to the search object image; obtaining a feature vector of the target object image according to the target object image; creating an image vector warehouse to obtain a feature vector of each image in the image vector warehouse; determining a cosine distance between each image in the image vector warehouse and the target object image according to the feature vector of the target object image and the feature vector of each image in the image vector warehouse; and determining an image in the image vector warehouse that is most similar to the target object image according to the cosine distance.
[0008] According to the present invention, the target object image is determined according to the image to be searched and the segmentation mask corresponding to the search object image, including: selecting whether the search object image needs to consider the features of the surrounding objects; when the search object image needs to consider the features of the surrounding objects, blurring the image to be searched to determine the blurred image; determining an image in which the object part is clear and the surrounding objects are blurred according to the blurred image, the image to be searched, and the segmentation mask corresponding to the search object image; determining the target object image as the image in which the object part is clear and the surrounding objects are blurred; when the search object image does not need to consider the features of the surrounding objects, determining an image in which the object part is clear and the surrounding objects are black according to the image to be searched and the segmentation mask corresponding to the search object image; finding out the image contour in which the pixels of the image in which the object part is clear and the surrounding objects are black are not 0, and then extracting the image with the pixels not 0 to obtain an image containing only the object; determining the target object image as the image containing only the object.
[0009] According to the present invention, when the search object image needs to consider surrounding objects, the image to be searched is blurred to determine the blurred image, including: obtaining the horizontal and vertical coordinates of the pixels of the image to be searched; setting the Gaussian convolution kernel size, and obtaining the horizontal and vertical coordinates of the Gaussian convolution kernel; determining the blurred image according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the Gaussian convolution kernel size and the horizontal and vertical coordinates of the Gaussian convolution kernel.
[0010] According to the present invention, determining a blurred image according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the size of the Gaussian convolution kernel and the horizontal and vertical coordinates of the Gaussian convolution kernel comprises: determining a blurred image according to the formula Determine the blurred image I blur , where x is the horizontal coordinate of the image pixel to be searched, y is the vertical coordinate of the image pixel to be searched, K is the Gaussian convolution kernel, w is the horizontal coordinate of the Gaussian convolution kernel, h is the vertical coordinate of the Gaussian convolution kernel, t is the size of the Gaussian convolution kernel, k is the value of the parity of t, I input The image to be searched.
[0011] According to the present invention, based on the segmentation masks corresponding to the blurred image, the image to be searched and the search object image, determining an image in which the object is partially clear and the surrounding objects are blurred comprises: according to the formula I1=mask·I input +(1-mask)·I blur , determine the image I1 where the object is partially clear and the surrounding objects are blurred, where mask is the segmentation mask corresponding to the search object image, I input is the image to be searched, I blur For blurred images.
[0012] According to the present invention, when the search object image does not need to consider the features of the surrounding objects, the image with a clear object part and a black surrounding is determined according to the segmentation mask corresponding to the image to be searched and the image of the search object, including: according to the formula I2 = mask·I input , determine the image I2 where the object is clear and the surrounding is black, where mask is the segmentation mask corresponding to the search object image, I input The image to be searched.
[0013] According to the present invention, the cosine distance between each image in the image vector warehouse and the target object image is determined based on the feature vector of the target object image and the feature vector of each image in the image vector warehouse, including: obtaining the multidimensional feature value of the feature vector of the target object image; obtaining the multidimensional feature value of the feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the feature vector of the target object image; determining the cosine distance between each image in the image vector warehouse and the target object image based on the multidimensional feature value of the feature vector of the target object image and the multidimensional feature value of the feature vector of each image in the image vector warehouse.
[0014] According to the present invention, according to the multidimensional eigenvalues of the eigenvector of the target object image and the multidimensional eigenvalues of the eigenvector of each image in the image vector warehouse, determining the cosine distance between each image in the image vector warehouse and the target object image comprises: determining the cosine distance between each image in the image vector warehouse and the target object image according to the formula Determine the cosine distance d(v, V) between the first image in the image vector warehouse and the target object image I), where v is the feature vector of the target object image, V I is the feature vector of the Ith image in the image vector warehouse, v i is the i-th eigenvalue of the eigenvector of the target object image, V i is the eigenvalue of the i-th dimension of the i-th image in the image vector warehouse, n is the dimension of the feature vector of the target object image and the feature vector of each image in the image vector warehouse, i≤n, and I, i and n are all positive integers.
[0015] According to the present invention, based on the cosine distance, determining the image in the image vector warehouse that is most similar to the target object image includes: sorting the cosine distance between each image in the image vector warehouse and the target object image from small to large; the image ranked first is the image in the image vector warehouse that is most similar to the target object image.
[0016] Technical effect: According to the present invention, by using the SAM segmentation model, the image to be searched can be panorama segmented, and the image of any object in the image to be searched can be selected and extracted, and the most similar image can be searched for. For images in which the search object is clear and the surrounding objects are blurred, the detailed features of the search object can be highlighted, and part of the features of the surrounding objects can be retained, thereby improving the search accuracy, and more effectively searching for specific objects in complex backgrounds. When determining a blurred image, the image to be searched can be blurred based on the Gaussian convolution filter blur formula to determine a blurred image, which can effectively reduce the image quality and reduce the interference of surrounding objects, thereby helping to retain part of the features of surrounding objects. When determining an image in which the object is partially clear and the surrounding objects are blurred, the image to be searched and the blurred image can be mixed by a segmentation mask to determine an image in which the object is partially clear and the surrounding objects are blurred, in which the search object is partially clear and the surrounding objects are blurred, the search object in the image can be made more prominent, the interference of surrounding objects can be reduced, and it helps to improve the recognition of the search object. When determining an image in which the object is partially clear and the surrounding area is black, the pixel values of the portion corresponding to the search object in the image to be searched can be retained through a segmentation mask, and the other portions can be set to black. The image in which the object is partially clear and the surrounding area is set to black can highlight the search object and make it more prominent in the image, while the surrounding area can be set to black, thereby reducing background interference. When determining the cosine distance between the first image in the image vector warehouse and the target object image, the cosine distance between the target object image and each image in the image vector warehouse can be determined based on the vector distance between the feature vector of the target object image and the feature vector of each image in the image vector warehouse, which is helpful for effective similarity comparison in image search and recognition, thereby finding the most similar image in the image vector warehouse.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.
[0019] Figure 1 A schematic diagram of a process flow of a SAM-based intelligent image search method according to an embodiment of the present invention is exemplarily shown;
[0020] Figure 2 An image vector warehouse creation diagram according to an embodiment of the present invention is exemplarily shown;
[0021] Figure 3 The basic framework diagram of the SAM-based intelligent image search method according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0024] Figure 1A flow chart of an intelligent image search method based on SAM according to an embodiment of the present invention is exemplarily shown, the method comprising: step S101, inputting the image to be searched into the SAM segmentation model, obtaining the segmentation mask of all objects in the image to be searched, wherein the segmentation mask and the image to be searched are matrices of the same size, and the elements of the segmentation mask matrix are 0 or 1, 0 is the background, and 1 is the target object; step S102, selecting the object to be searched according to the object, and obtaining the segmentation mask corresponding to the search object image; step S103, according to the image to be searched, and the Search for a segmentation mask corresponding to the object image to determine the target object image; step S104, obtain a feature vector of the target object image based on the target object image; step S105, create an image vector warehouse to obtain a feature vector of each image in the image vector warehouse; step S106, determine a cosine distance between each image in the image vector warehouse and the target object image based on the feature vector of the target object image and the feature vector of each image in the image vector warehouse; step S107, determine an image in the image vector warehouse that is most similar to the target object image based on the cosine distance.
[0025] According to the SAM-based intelligent image search method of the embodiment of the present invention, by using the SAM segmentation model, the image to be searched can be panorama segmented, and the image of any object in the image to be searched can be selected and extracted, and the most similar image can be searched. For images with clear search objects and blurred surrounding objects, the detailed features of the search objects can be highlighted, and some features of the surrounding objects can be retained, thereby improving the search accuracy and more effectively searching for specific objects in complex backgrounds.
[0026] According to one embodiment of the present invention, in step S101, the SAM segmentation model has the capabilities of instance segmentation and semantic segmentation, wherein instance segmentation can identify and divide each object instance in the image, and semantic segmentation can mark the corresponding category label for each pixel in the image. The SAM segmentation model segments all objects in the image to be searched, and generates segmentation masks for all objects. The segmentation mask is a matrix of the same size as the image to be searched, and the elements of the segmentation mask matrix are 0 or 1. If the value of the corresponding position of the segmentation mask matrix is 0, the pixel at the corresponding position of the image to be searched is the background, and if the value of the corresponding position of the segmentation mask matrix is 1, the pixel at the corresponding position of the image to be searched is the target object. By segmenting the mask matrix, the background part and the target object in the image to be searched can be clearly distinguished.
[0027] According to an embodiment of the present invention, in step S102, an image of a selected object is extracted from the image to be searched to form a search object image, and a segmentation mask corresponding to the search object image is obtained.
[0028] According to an embodiment of the present invention, in step S103, object recognition is further performed based on the image to be searched and the segmentation mask corresponding to the search object image, which helps to accurately extract the required target object image.
[0029] According to one embodiment of the present invention, step S103 includes: selecting whether the search object image needs to consider the features of surrounding objects; when the search object image needs to consider the features of surrounding objects, blurring the image to be searched to determine a blurred image; determining an image in which the object part is clear and the surrounding objects are blurred based on the blurred image, the image to be searched, and the segmentation mask corresponding to the search object image; determining the target object image as an image in which the object part is clear and the surrounding objects are blurred; when the search object image does not need to consider the features of surrounding objects, determining an image in which the object part is clear and the surrounding objects are black based on the image to be searched and the segmentation mask corresponding to the search object image; finding out the image contour in which the pixels of the image in which the object part is clear and the surrounding objects are black are not 0, and then extracting the image with non-0 pixels to obtain an image containing only the object; determining the target object image as the image containing only the object.
[0030] According to one embodiment of the present invention, it is selected whether the search object image needs to consider the features of the surrounding objects, that is, when processing a larger search object, it is selected that the features of the surrounding objects need to be considered, and the desired image can be correctly searched. When processing a smaller object or when the features of the surrounding objects do not need to be considered, it is only necessary to focus on the features of the smaller object, and it is selected not to consider the features of the surrounding objects. In the case of selecting the need to consider the features of the surrounding objects, the features of some surrounding objects are retained, but the features of the surrounding objects cannot be too prominent. Therefore, the surrounding objects are blurred, which can highlight the detailed features of the search object and retain some features of some surrounding objects. The search image is blurred using the Gaussian convolution filter blur formula to obtain a blurred image. Using the segmentation mask corresponding to the search object image, the clear object and the blurred surrounding objects are merged to obtain an image in which the object is partially clear and the surrounding objects are blurred. When the search object image is small, or there is no need to consider the features of surrounding objects, the search object can be cut out by the segmentation mask corresponding to the search object image as the target object image, removing all the features of the surrounding objects. An object in the image is irregular, so first use the findContours function to cut out the search object, and the image must be a rectangle, so add black pixels around it to make the target object just a rectangle.
[0031] According to one embodiment of the present invention, when the search object image needs to consider surrounding objects, the image to be searched is blurred to determine the blurred image, including: obtaining the horizontal and vertical coordinates of the pixels of the image to be searched; setting the Gaussian convolution kernel size, and obtaining the horizontal and vertical coordinates of the Gaussian convolution kernel; determining the blurred image according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the Gaussian convolution kernel size and the horizontal and vertical coordinates of the Gaussian convolution kernel.
[0032] According to an embodiment of the present invention, the horizontal and vertical coordinates of each pixel in the image to be searched are obtained, so that each pixel can be accurately located and processed in subsequent processing. The Gaussian convolution kernel is used for image blurring, and the size of the Gaussian convolution kernel determines the intensity of the blurring effect.
[0033] According to an embodiment of the present invention, determining a blurred image according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the size of the Gaussian convolution kernel and the horizontal and vertical coordinates of the Gaussian convolution kernel comprises: determining the blurred image I according to formulas (1) and (2): blur ,
[0034]
[0035] Where x is the horizontal coordinate of the pixel in the image to be searched, y is the vertical coordinate of the pixel in the image to be searched, K is the Gaussian convolution kernel, w is the horizontal coordinate of the Gaussian convolution kernel, h is the vertical coordinate of the Gaussian convolution kernel, t is the size of the Gaussian convolution kernel, k is the value of the parity of t, I input The image to be searched.
[0036] According to one embodiment of the present invention, in formula (2), It means that if the Gaussian convolution kernel size is an even number, k is the Gaussian convolution kernel size. If the Gaussian convolution kernel size is an odd number, k is the integer of the Gaussian convolution kernel size minus 1. For example, when the Gaussian convolution kernel size is 3, the Gaussian convolution kernel size is an odd number, k is 2. When the Gaussian convolution kernel size is 4, the Gaussian convolution kernel size is an even number, k is 4. In formula (1), It means to perform weighted summation on all pixels in the convolution window, and perform weighted processing on the pixel values through the Gaussian convolution kernel K to obtain the blurred pixel values, that is, the blurred image. The larger the Gaussian convolution kernel, the more obvious the image blurring effect. On the contrary, the smaller the Gaussian convolution kernel, the lighter the blurring effect and the more image details can be retained.
[0037] In this way, the search image can be blurred based on the Gaussian convolution filter blur formula to determine the blurred image, which can effectively reduce the image quality and reduce the interference of surrounding objects, thereby helping to retain some features of the surrounding objects.
[0038] According to an embodiment of the present invention, determining an image in which the object is partially clear and the surrounding objects are blurred according to the segmentation masks corresponding to the blurred image, the image to be searched and the search object image includes: determining an image I1 in which the object is partially clear and the surrounding objects are blurred according to formula (3),
[0039] I1=mask·I input +(1-mask)·I blur (3)
[0040] Among them, mask is the segmentation mask corresponding to the search object image, I input is the image to be searched, I blur For blurred images.
[0041] According to one embodiment of the present invention, in formula (3), mask·I input Indicates that the pixels corresponding to the search object in the search image are retained, that is, the clear object part is retained. (1-mask)·I blur It means that the pixels other than the part corresponding to the search object in the blurred image are blurred, that is, the surrounding objects are blurred. The above two items are added together to form an image in which the object part is clear and the surrounding objects are blurred.
[0042] In this way, the image to be searched and the blurred image can be mixed through the segmentation mask to determine an image in which the object is partially clear and the surrounding objects are blurred. In this image, the search object is partially clear and the surrounding objects are blurred. This can make the search object in the image more prominent and reduce the interference of surrounding objects, which helps to improve the recognition of the search object.
[0043] According to an embodiment of the present invention, when the search object image does not need to consider the features of the surrounding objects, determining an image in which the object is partially clear and the surrounding is black according to the segmentation masks corresponding to the image to be searched and the image of the search object includes: determining an image I2 in which the object is partially clear and the surrounding is black according to formula (4);
[0044] I2=mask·I input (4)
[0045] Among them, mask is the segmentation mask corresponding to the search object image, I input The image to be searched.
[0046] According to one embodiment of the present invention, in formula (4), mask·I input It means that the pixels corresponding to the search object in the image to be searched are retained, that is, the image with the object part being clear and the surrounding part being black.
[0047] In this way, the pixel values of the part corresponding to the search object in the image to be searched can be retained through the segmentation mask, and the other parts can be set to black. The image with the object part clear and the surrounding black can be determined. The search object can be highlighted to make it more prominent in the image, while the surrounding area can be set to black, thereby reducing background interference.
[0048] According to an embodiment of the present invention, in step S104, the target object image is placed in an Image Embedding model to obtain a feature vector of the target object image. For example, the Image Encoder of CLIP is used as Image Embedding.
[0049] According to one embodiment of the present invention, in step S105, Figure 2 The image vector warehouse creation diagram according to an embodiment of the present invention is exemplarily shown. The purpose of creating the image vector warehouse is to effectively store and search for images. By extracting the feature vector of the image and recording its path, efficient image retrieval can be achieved. Each image in the image vector warehouse is placed in the Image Embedding model (the same model as the target object image is used). For example, the image feature vector can be extracted using CLIP's Image Embedding. After extracting the feature vector of the image, the feature vector is recorded, and a mapping relationship between the image path and the feature vector is established for subsequent retrieval and matching.
[0050] According to an embodiment of the present invention, in step S106, the cosine distance can be used to calculate the distance between two vectors. By calculating the cosine distance, the similarity between each image in the image vector warehouse and the target object image can be obtained.
[0051] According to one embodiment of the present invention, step S106 includes: obtaining the multidimensional eigenvalue of the feature vector of the target object image; obtaining the multidimensional eigenvalue of the feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the feature vector of the target object image; determining the cosine distance between each image in the image vector warehouse and the target object image based on the multidimensional eigenvalue of the feature vector of the target object image and the multidimensional eigenvalue of the feature vector of each image in the image vector warehouse.
[0052] According to one embodiment of the present invention, based on the multidimensional eigenvalues of the eigenvalues of the target object image's eigenvalues and the eigenvalues of the eigenvalues of each image in the image vector warehouse, determining the cosine distance between each image in the image vector warehouse and the target object image comprises: determining the cosine distance d(v, V) between the first image in the image vector warehouse and the target object image according to formula (5): I ),
[0053]
[0054] Among them, v is the feature vector of the target object image, V I is the feature vector of the Ith image in the image vector warehouse, v i is the i-th eigenvalue of the eigenvector of the target object image, V i is the eigenvalue of the i-th dimension of the i-th image in the image vector warehouse, n is the dimension of the feature vector of the target object image and the feature vector of each image in the image vector warehouse, i≤n, and I, i and n are all positive integers.
[0055] According to one embodiment of the present invention, in formula (5), is the cosine distance between the feature vector of the first image in the image vector warehouse and the feature vector of the target object image, that is, the cosine distance between the first image in the image vector warehouse and the target object image. The smaller the cosine distance, the more similar the feature vectors of the two images are. is the numerical calculation of the cosine distance, The sum of the product of the feature vector of the target object image and the feature value of the corresponding dimension of the feature vector of the I-th image in the image vector warehouse, It is the product of the feature vector of the target object image and the modulus of the feature vector of the Ith image in the image vector warehouse.
[0056] In this way, the cosine distance between the target object image and each image in the image vector warehouse can be determined based on the vector distance between the feature vector of the target object image and the feature vector of each image in the image vector warehouse, which helps to make effective similarity comparisons in image search and recognition, thereby finding the most similar image in the image vector warehouse.
[0057] According to an embodiment of the present invention, in step S107, the images in the image vector warehouse may be sorted or screened by cosine distance, so as to find the image in the image vector warehouse that is most similar to the target object image.
[0058] According to an embodiment of the present invention, step S107 includes: sorting the co-line distance between each image in the image vector warehouse and the target object image from small to large; the image ranked first is the image in the image vector warehouse that is most similar to the target object image.
[0059] According to one embodiment of the present invention, the image ranked first in the sorting list, i.e., the image with the smallest cosine distance, is the image in the image vector warehouse that is most similar to the target object image. This image has the highest similarity to the target object image and can be regarded as the most matching image.
[0060] According to the SAM-based intelligent image search method of the embodiment of the present invention, by using the SAM segmentation model, the image to be searched can be panorama segmented, and the image of any object in the image to be searched can be selected and extracted, and the most similar image can be searched for. For images in which the search object is clear and the surrounding objects are blurred, the detailed features of the search object can be highlighted, and part of the features of the surrounding objects can be retained, thereby improving the search accuracy, and more effectively searching for specific objects in complex backgrounds. When determining a blurred image, the image to be searched can be blurred based on the Gaussian convolution filter blur formula to determine a blurred image, which can effectively reduce the image quality and reduce the interference of surrounding objects, thereby helping to retain part of the features of surrounding objects. When determining an image in which the object is partially clear and the surrounding objects are blurred, the image to be searched and the blurred image can be mixed by a segmentation mask, thereby determining an image in which the object is partially clear and the surrounding objects are blurred, in which the search object is partially clear and the surrounding objects are blurred, the search object in the image can be made more prominent, the interference of surrounding objects can be reduced, and it helps to improve the recognition of the search object. When determining an image in which the object is partially clear and the surrounding area is black, the pixel values of the portion corresponding to the search object in the image to be searched can be retained through a segmentation mask, and the other portions can be set to black. The image in which the object is partially clear and the surrounding area is set to black can highlight the search object and make it more prominent in the image, while the surrounding area can be set to black, thereby reducing background interference. When determining the cosine distance between the first image in the image vector warehouse and the target object image, the cosine distance between the target object image and each image in the image vector warehouse can be determined based on the vector distance between the feature vector of the target object image and the feature vector of each image in the image vector warehouse, which is helpful for effective similarity comparison in image search and recognition, thereby finding the most similar image in the image vector warehouse.
[0061] Figure 3 The basic framework diagram of the SAM-based intelligent image search method according to an embodiment of the present invention is exemplarily shown. Using the SAM segmentation model, all objects in the input image can be segmented, that is, the image of any object in the image can be selected and extracted to search for the most similar image.
[0062] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.
Claims
1. An intelligent image search method based on SAM, characterized in that: include: Input the image to be searched into the SAM segmentation model to obtain the segmentation mask of all objects in the image to be searched, wherein the segmentation mask and the image to be searched are matrices of the same size, and the elements of the segmentation mask matrix are 0 or 1, 0 represents the background, and 1 represents the target object; According to the object, an object to be searched is selected to obtain a segmentation mask corresponding to the image of the search object; Determine a target object image according to the image to be searched and the segmentation mask corresponding to the search object image; According to the target object image, obtaining a feature vector of the target object image; Create an image vector warehouse and obtain the feature vector of each image in the image vector warehouse; Determine a cosine distance between each image in the image vector warehouse and the target object image according to a feature vector of the target object image and a feature vector of each image in the image vector warehouse; An image in the image vector warehouse that is most similar to the target object image is determined according to the cosine distance.
2. The SAM-based intelligent image search method according to claim 1, characterized in that: Determining a target object image according to the image to be searched and a segmentation mask corresponding to the search object image includes: Selecting whether the search object image needs to consider surrounding object features; When the search object image needs to consider the features of surrounding objects, blurring the image to be searched to determine a blurred image; Determine an image in which the object is partially clear and surrounding objects are blurred according to the blurred image, the image to be searched, and the segmentation mask corresponding to the search object image; Determine the target object image as an image in which the object is partially clear and surrounding objects are blurred; When the search object image does not need to consider the features of surrounding objects, an image in which the object portion is clear and the surrounding portion is black is determined according to the image to be searched and the segmentation mask corresponding to the search object image; Find out the image contour where the pixel of the image with the object portion being clear and the surrounding being black is not 0, and then extract the image where the pixel is not 0 to obtain an image containing only the object; The target object image is determined as the image containing only the object.
3. The SAM-based intelligent image search method according to claim 2, characterized in that: When the search object image needs to consider surrounding objects, blurring the image to be searched to determine a blurred image, including: Get the horizontal and vertical coordinates of the pixels of the image to be searched; Set the Gaussian convolution kernel size and obtain the horizontal and vertical coordinates of the Gaussian convolution kernel; A blurred image is determined according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the size of the Gaussian convolution kernel and the horizontal and vertical coordinates of the Gaussian convolution kernel.
4. The SAM-based intelligent image search method according to claim 3, characterized in that: Determining a blurred image according to the image to be searched, the horizontal and vertical coordinates of the pixels of the image to be searched, the size of the Gaussian convolution kernel, and the horizontal and vertical coordinates of the Gaussian convolution kernel includes: According to the formula Determine the blurred image I blur , where x is the horizontal coordinate of the image pixel to be searched, y is the vertical coordinate of the image pixel to be searched, K is the Gaussian convolution kernel, w is the horizontal coordinate of the Gaussian convolution kernel, h is the vertical coordinate of the Gaussian convolution kernel, t is the size of the Gaussian convolution kernel, k is the value of the parity of t, I input The image to be searched.
5. The SAM-based intelligent image search method according to claim 2, characterized in that: Determining an image in which an object portion is clear and surrounding objects are blurred according to the segmentation masks corresponding to the blurred image, the image to be searched, and the search object image, comprises: According to the formula I1=mask·I input +(1-mask)·I blur Determine the image I1 where the object is partially clear and the surrounding objects are blurred, where mask is the segmentation mask corresponding to the search object image, I input is the image to be searched, I blur For blurred images.
6. The SAM-based intelligent image search method according to claim 2, characterized in that: When the search object image does not need to consider the features of surrounding objects, determining an image in which the object is partially clear and the surrounding is black according to the segmentation masks corresponding to the image to be searched and the image of the search object, comprises: According to the formula I2=mask·I input Determine the image I2 where the object is partially clear and the surrounding is black, where mask is the segmentation mask corresponding to the search object image, and I input The image to be searched.
7. The SAM-based intelligent image search method according to claim 1, characterized in that: Determining a cosine distance between each image in the image vector warehouse and the target object image according to a feature vector of the target object image and a feature vector of each image in the image vector warehouse includes: Obtaining multi-dimensional eigenvalues of a eigenvector of the target object image; Acquire a multidimensional feature value of a feature vector of each image in the image vector warehouse, wherein the dimension of the feature vector of each image in the image vector warehouse is consistent with the dimension of the feature vector of the target object image; The cosine distance between each image in the image vector warehouse and the target object image is determined according to the multidimensional eigenvalue of the eigenvalue of the eigenvector of the target object image and the multidimensional eigenvalue of the eigenvector of each image in the image vector warehouse.
8. The SAM-based intelligent image search method according to claim 7, characterized in that: Determining the cosine distance between each image in the image vector warehouse and the target object image according to the multidimensional feature value of the feature vector of the target object image and the multidimensional feature value of the feature vector of each image in the image vector warehouse includes: According to the formula Determine the cosine distance d(v, V) between the first image in the image vector warehouse and the target object image I ), where v is the feature vector of the target object image, V I is the feature vector of the Ith image in the image vector warehouse, v i is the i-th eigenvalue of the eigenvector of the target object image, V i is the eigenvalue of the i-th dimension of the i-th image in the image vector warehouse, n is the dimension of the feature vector of the target object image and the feature vector of each image in the image vector warehouse, i≤n, and I, i and n are all positive integers.
9. The SAM-based intelligent image search method according to claim 1, characterized in that: Determining, according to the cosine distance, an image in the image vector warehouse that is most similar to the target object image, comprising: Sorting the co-line distance between each image in the image vector warehouse and the target object image from small to large; The image ranked first is the image in the image vector warehouse that is most similar to the target object image.
Citation Information
Patent Citations
Image retrieval method and system based on image segmentation and fuzzy pattern recognition
CN109213886A
Image searching method and device
CN115248871A
Augmented reality method and system for three-dimensional object and storage medium
CN117197405A
Medical image fine-grained classification method based on guidance of large segmentation model
CN118135325A
Fine-grained image retrieval method and device, electronic equipment and storage medium
CN118193773A
