Image retrieval method, system and equipment for enhancing fine-grained object retrieval performance and medium

Abstract

The invention relates to the technical field of image retrieval, and provides an image retrieval method, system, equipment and medium for enhancing fine-grained object retrieval performance, the method comprises the following steps: processing an original image through a multi-scale image interception technology, generating a target image and a slice set, inputting the images into an encoder, constructing an image vector library, and meanwhile, inputting the target image and the slice set into a database; the multi-modal large model carries out semantic analysis on images and slices, generates text description and constructs a text vector library, in the retrieval stage, visual similarity matching is carried out based on image vectors, text similarity matching is carried out based on the text vector library, results are integrated to obtain a candidate set, and finally, a final retrieval result is obtained through weighted fusion and sorting. According to the method, the accuracy of fine-grained object retrieval is remarkably improved, the comprehensiveness of retrieval is enhanced by combining visual and text information, and the correlation of results is improved, so that a user can find required information more quickly and more accurately.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of image retrieval, and particularly relates to an image retrieval method, system, device and medium for enhancing fine-grained object retrieval performance. BACKGROUND

[0002] With the continuous progress of computer technology, the management and application of image data in search systems have become increasingly important. Introducing image information into question and answer systems and computer vision tasks can help significantly improve the accuracy and intelligence level of the system. However, in current image information retrieval practices, there are still some key challenges.

[0003] The commonly used image retrieval method currently mainly relies on the embedding vector generated by the image encoding model to realize matching in the way of "image searching image". This way is difficult to support the function of "text searching image", that is, to find related images according to the text description. In addition, due to the different sizes of images and the fact that the target object may only occupy a small part of the image, the information proportion of the object in the whole image is small, which reduces the similarity matching effect between the whole images and easily causes the omission of related images.

[0004] Therefore, by proposing an image retrieval method for enhancing fine-grained object retrieval performance, it is a feasible and preferred way to improve the accuracy and semantic understanding ability of image retrieval. SUMMARY

[0005] To solve the above problems, the present disclosure provides an image retrieval method, system, device and medium for enhancing fine-grained object retrieval performance, which realizes multi-level understanding of image content by introducing an object-based multi-scale slicing mechanism and combining a Paligemma model to generate image semantic description.

[0006] In a first aspect, the present disclosure provides an image retrieval method for enhancing fine-grained object retrieval performance, the method comprising, processing the original image based on a multi-scale image intercepting technology to obtain a target image and a target image slice set; inputting the target image and the target image slice set into an image encoder to establish an image vector library; performing semantic analysis on the target image and the target image slice set by using a multi-modal large model to generate corresponding text descriptions, vector encoding each text description, and establishing a text vector library; performing visual similarity matching on the query vector and the image vector library and semantic similarity matching on the query vector and the text vector library to obtain a visual candidate image set and a semantic candidate image set respectively, and generating a visual similarity score and a semantic similarity score; The visual similarity score and the semantic similarity score are weighted and fused to obtain a final similarity score, and the visual candidate image set and the semantic candidate image set are sorted according to the final similarity score to obtain a final retrieval result.

[0007] Further, The original image is processed based on a multi-scale image cropping technology to obtain a target image and a target image slice set, specifically including: The original image is uniformly resized to obtain a target image; The target image is identified by using an image segmentation model to obtain each target object and target position information corresponding to each target object; The target image is cropped based on a preset size parameter to obtain an initial image slice set; The object inclusion relationship between each initial image slice is determined based on each target object and its target position information, and the initial image slice set is de-duplicated based on the object inclusion relationship to obtain the target image slice set.

[0008] Further, The target image and the target image slice set are input into an image encoder to establish an image vector library, specifically including: The target image and the target image slice set are input into the image encoder to extract corresponding visual features; The extracted visual features are fused to establish the image vector library.

[0009] Further, The query vector is matched with the image vector library for visual similarity to obtain a visual candidate image set, and a visual similarity score is generated, specifically including: The query vector is used to search in the image vector library to obtain an initial candidate image set; The visual similarity scores of each candidate image in the initial candidate image set and the query vector are calculated respectively, and the visual candidate image set is determined according to the visual similarity scores.

[0010] Further, The query vector is matched with the text vector library for semantic similarity to obtain a semantic candidate image set, and a semantic similarity score is generated, specifically including: The query vector is used to search in the text vector library to obtain an initial candidate text description set; determine actual similarities between the query vector and each of the text description information in the text vector library by using a text similarity algorithm, and determine a second initial candidate text description set according to the actual similarities; perform deduplication processing on the first initial candidate text description set and the second initial candidate text description set to obtain an actual recall result; calculate semantic similarity scores for the actual recall result by using a text rerank model, and determine the semantic candidate image set according to the semantic similarity scores.

[0011] Further, perform weighted fusion on the visual similarity scores and the semantic similarity scores to obtain final similarity scores, and sort the visual candidate image set and the semantic candidate image set according to the final similarity scores to obtain a final retrieval result, specifically including: obtain image reverse rankings of each final candidate image in the visual candidate image set and text reverse rankings of each final candidate image in the semantic candidate image set; determine the final similarity scores according to the image reverse rankings and the text reverse rankings.

[0012] In a second aspect, based on the same inventive concept, the present disclosure provides an image retrieval system for enhancing the performance of fine-grained object retrieval, The system comprises: a multi-scale image cropping module that processes an original image based on a multi-scale image cropping technique to obtain a target image and a target image slice set; an image feature extraction module that inputs the target image and the target image slice set into an image encoder to establish an image vector library; an image semantic description generation module that performs semantic analysis on the target image and the target image slice set by using a multi-modal large model to generate corresponding text descriptions, vector-encodes each text description, and establishes a text vector library; a primary retrieval module that performs visual similarity matching between a query vector and the image vector library, and semantic similarity matching between the query vector and the text vector library, to obtain a visual candidate image set and a semantic candidate image set, and generates visual similarity scores and semantic similarity scores; a secondary retrieval module that performs weighted fusion on the visual similarity scores and the semantic similarity scores to obtain final similarity scores, and sorts the visual candidate image set and the semantic candidate image set according to the final similarity scores to obtain a final retrieval result.

[0013] In a third aspect, the present disclosure provides an electronic device, comprising at least one processor and at least one memory electrically connected; The memory is electrically connected with the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the image retrieval methods for enhancing fine-grained object retrieval performance as described above.

[0014] In a fourth aspect, the present disclosure provides a computer storage medium, wherein the computer readable storage medium stores a computer program; The computer program is executed by the processor to implement any of the image retrieval methods for enhancing fine-grained object retrieval performance as described above.

[0015] In a fifth aspect, the present disclosure provides a computer program product stored in at least one storage medium; The computer program product comprises a plurality of instructions for causing at least one electronic device to perform any of the image retrieval methods for enhancing fine-grained object retrieval performance as described above.

[0016] Compared with the prior art, the present disclosure has the following advantages: 1. The image segmentation model is used in combination with the minimum size constraint condition to accurately identify and extract each object in the image, ensuring the integrity of each object; at the same time, a multi-scale cutting strategy is adopted to obtain image content from different granularities to enhance the ability to capture local details.

[0017] 2. The original image and its multiple slices are respectively input into an image encoder to extract visual features at different levels and fuse them to form a more representative comprehensive image vector representation, thereby improving the accuracy of similarity calculation between images.

[0018] 3. A multi-modal large model such as Paligemma is used to automatically generate high-quality text descriptions for each image, establish a mapping relationship between the image and its semantic information, and then construct an image semantic information library, greatly expanding the application scenarios of image retrieval.

[0019] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or will be learned by practice of the present disclosure. The objects and other advantages of the present disclosure will be realized and achieved by the structures indicated in the specification, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A flowchart of an image retrieval method for enhancing fine-grained object retrieval performance of the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will clearly and completely explain the technical solutions in the embodiments of the present disclosure with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present disclosure.

[0023] Figure 1 An image retrieval method for enhancing fine-grained object retrieval performance according to the embodiments of the present disclosure is shown. As shown in Figure 1 The image retrieval method for enhancing fine-grained object retrieval performance of the embodiments of the present disclosure includes the following steps: S1, processing the original image based on a multi-scale image cropping technique to obtain a target image and a target image slice set; In the embodiments of the present disclosure, the step S1 specifically includes: S11, uniformly adjusting the size of the original image to obtain a target image; S12, identifying the target image using an image segmentation model to obtain each target object and the target position information corresponding to each target object; S13, performing multi-scale cropping on the target image according to a preset size parameter to obtain an initial image slice set; S14, determining the object inclusion relationship between each initial image slice based on each target object and its target position information, and performing deduplication processing on the initial image slice set according to the object inclusion relationship to obtain the target image slice set.

[0024] In the specific implementation process, the specific process of cropping based on the multi-scale image cropping technique includes: 1. Uniformly adjusting the size of the original input picture to 1024X1024 to obtain a target image; 2. Obtaining the specific position information of each target object in the target image based on an image segmentation model; 3. Step size of 16 in image size of 512X512, 224X224, 112X112, 32X32, intercept in image, get each image size intercepted target image slice; 4. De-duplication processing is performed on each target image slice, and target image slices with semantic independence and content irreplaceability are retained, and target image slices completely contained or combinable by other target image slices are removed.

[0025] For any image size intercepted target image slice: 1) Count the number of objects in each target image slice; 2) Traverse all the split target image slices; 3) Use the result of the segmentation model for each target image slice to count the number of target objects contained therein and the class and location information of each target object.

[0026] 4) Determine the object inclusion relationship between the target image slices. If all target objects in target image slice A can be found in target image slice B with the same class and location completely contained, it is considered that target image slice B completely contains the content of target image slice A; At this time, it can be determined that target image slice A is a "subset" of target image slice B, and target image slice A can be deleted.

[0027] 5) Two-by-two comparison and de-duplication: compare all target image slices two by two: if target image slice B contains all the target objects of target image slice A → delete target image slice A; if target image slice A contains all the target objects of target image slice B → delete target image slice B; if neither contains the other → keep both target image slices.

[0028] 6) Interval merging optimization: there is a group of continuous split target image slices: the i-th to the j-th; if the i-th target image slice and the j-th target image slice together can cover all the target objects in the middle target image slices, that is, the contents of the i-th target image slice and the j-th target image slice contain all the target objects appearing in the i-th to the j-th target image slices; then the i+1-th to the j-1-th target image slices can be deleted, and only the i-th and j-th target image slices are retained.

[0029] 7) Finally, the target image slice set is retained, and after the above steps, a set of non-redundant and complementary information target image slices is obtained; these images will be used for subsequent feature extraction and index construction.

[0030] For example, assume that there are the following three target segmentation images: target image slice A: contains the object {cat}; target image slice B: contains the objects {cat, window}; target image slice C: contains the objects {window, table}; according to the rule: target image slice A is completely contained by target image slice B → delete target image slice A; there is no mutual inclusion relationship between target image slice B and target image slice C → both are retained; the final retained target image slices are: B, C.

[0031] S2, input the target image and the target image slice set into an image encoder to establish an image vector library; In an embodiment of the present disclosure, the step S2 specifically comprises: S21, input the target image and the target image slice set into the image encoder to extract corresponding visual features; S22, fuse the extracted visual features to establish the image vector library.

[0032] In the embodiment, the original image and the obtained target image slice set without redundancy and complementary information are input into the image encoder of the Paligemma model to form a comprehensive image vector representation containing multi-level information, and in the specific implementation process, the mapping relationship from the target image slice to the original image is obtained.

[0033] S3, perform semantic analysis on the target image and the target image slice set by using a multi-modal large model to generate corresponding text descriptions, vector encode the text descriptions, and establish a text vector library; In the embodiment, in order to more efficiently extract semantic information in the image, an advanced Paligemma model is adopted to process the original image and the multi-scale slice (a small image set without redundancy and complementary information) thereof, to automatically generate detailed image descriptions, to allow understanding of image content from two dimensions of vision and semantics, to provide rich text information support for the subsequent retrieval process, and finally to vector encode the semantic information and establish a mapping relationship between the semantics and the image. The Paligemma model is uniformly selected for image encoding and text description generation, which can ensure alignment between the image and the text semantics.

[0034] Through S2 and S3, the image vector library of the original image and the multi-scale slice and the text vector library have been generated, and the original image information can be corresponded to by searching the image or semantic information of the multi-scale slice.

[0035] S4, performing visual similarity matching of the query vector with the image vector library and semantic similarity matching of the query vector with the text vector library to obtain a visual candidate image set and a semantic candidate image set respectively, and generating a visual similarity score and a semantic similarity score; In the embodiments of the present disclosure, the step S4 specifically comprises: S41, searching in the image vector library by using the query vector to obtain an initial candidate image set; S42, calculating a visual similarity score of each candidate image in the initial candidate image set with the query vector respectively, and determining the visual candidate image set according to the visual similarity scores; S43, searching in the text vector library by using the query vector to obtain an initial candidate text description set; S44, determining an actual similarity between the query vector and each text description information in the text vector library by using a text similarity algorithm, and determining a second initial candidate text description set according to the actual similarities; S45, performing a deduplication processing on the first initial candidate text description set and the second initial candidate text description set to obtain an actual recall result; S46, calculating a semantic similarity score of the actual recall result by using a text rerank model, and determining the semantic candidate image set according to the semantic similarity score.

[0036] S5, performing a weighted fusion on the visual similarity score and the semantic similarity score to obtain a final similarity score, and performing a sorting on the visual candidate image set and the semantic candidate image set according to the final similarity score to obtain a final retrieval result; In the embodiments of the present disclosure, the step S5 specifically comprises: S51, obtaining an image reverse ranking of each final candidate image in the visual candidate image set and a text reverse ranking of each final candidate image in the semantic candidate image set; S52, determining the final similarity score according to the image reverse ranking and the text reverse ranking.

[0037] In the specific implementation process, the query vector can be text data or image data; 1) Text-to-image search, when the query vector is text, the query text is converted into a vector representation form by a text encoding model; 2) Image-to-image search, when the query vector is an image, the query image is converted into a vector representation form by an image encoding model, and a Paligemma model is used to generate text description information to prepare for subsequent text retrieval.

[0038] 3) Image Similarity Matching: Utilize the aforementioned text vectors (for text-to-image search) / image vectors (for image-to-image search) to find the most relevant image information in a pre-built image vector library. For each original image retrieved (which may contain multiple different size slices), calculate the visual similarity score between the original image and the query vector based on the similarity calculation formula.

[0039] Where the similarity calculation formula is:

[0040] Where, is the average value of slice similarity; The calculation is the variance (or dispersion), which measures the difference between the similarity of each slice. If all slice similarities are very close, the variance is small; otherwise, if the similarity difference is large, the variance is large.

[0041] In the implementation process, the following factors need to be considered when calculating the similarity of the original image based on multiple size slices: The influence of single slice similarity: Each size slice contributes to the final similarity.

[0042] The influence of slice number: Finding more size slices should increase the overall similarity.

[0043] Balancing mechanism: Ensure that in some cases (e.g. when all slice similarities are low), the overall similarity will not be falsely increased due to the large number of slices.

[0044] 4) Text-based search: Perform semantic search in the text vector library to find the most relevant text information closest to the input text semantics (text-to-image search, input text semantics is the original query text; image-to-image search, input text semantics is the Paligemma model generated text description information). At the same time, perform text similarity evaluation using text similarity algorithms (such as Elasticsearch's text similarity mechanism) to evaluate the similarity between the query text and the text description information. Remove duplicate text description information obtained from the two search methods as the recall result. Reordering: Finally, apply the text rerank model to reorder the above recall results to obtain the similarity score of the recall result; finally, according to the similarity calculation formula, calculate the similarity score of the original image corresponding to the text description dimension.

[0045] 5) Fusion ranking algorithm of visual similarity score and semantic similarity score. For each image, add its reverse ranking in the image list and the text description list as the fusion score of the image. That is, the i-th retrieved image, the fusion similarity is:

[0046] Wherein, k is the weight of image list ranking, (1-k) is the weight of text description list, if searching image by text, since the text information is relatively more accurate, then k=0.4, 1-k=0.6, the ranking of text information part occupies a larger weight. If searching image by image, then k=0.6, the weight of picture part occupies a larger weight. is the ranking of the i-th image in the picture list, is the ranking of the i-th image in the text description list.

[0047] In the embodiment, through the above-mentioned object-based multi-scale image cropping and encoding and the dual retrieval mechanism combining image and semantic features, the performance of the image retrieval system is significantly improved, especially in the aspect of fine-grained object retrieval. In addition, the method also shows great potential in the intelligent question answering scene, which can make full use of image information and improve the overall effect of the question answering system.

[0048] Based on the same inventive concept, the disclosure embodiments also provide an image retrieval system for enhancing the performance of fine-grained object retrieval corresponding to the above-mentioned method, comprising: A multi-scale image cropping module, which processes the original image based on multi-scale image cropping technology to obtain a target image and a target image slice set; An image feature extraction module, which inputs the target image and the target image slice set into an image encoder to establish an image vector library; An image semantic description generation module, which performs semantic analysis on the target image and the target image slice set using a multi-modal large model to generate corresponding text descriptions, vector-encodes each text description, and establishes a text vector library; A primary retrieval module, which performs visual similarity matching between a query vector and the image vector library, and semantic similarity matching between the query vector and the text vector library, respectively obtains a visual candidate image set and a semantic candidate image set, and generates a visual similarity score and a semantic similarity score; A secondary retrieval module, which performs weighted fusion on the visual similarity score and the semantic similarity score to obtain a final similarity score, and sorts the visual candidate image set and the semantic candidate image set according to the final similarity score to obtain a final retrieval result.

[0049] Based on the same inventive concept, the disclosure also provides an electronic device correspondingly. The electronic device of the embodiments of the disclosure comprises at least one processor and at least one memory electrically connected, wherein the memory is electrically connected with the processor, and the memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the image retrieval method for enhancing the performance of fine-grained object retrieval as described above.

[0050] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between the lines, and the indirect connection mode can also be applied to the embodiments of the disclosure as long as the purpose of the disclosure is achieved.

[0051] Based on the same inventive concept, the disclosure also provides a computer storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image retrieval method for enhancing the performance of fine-grained object retrieval as described above.

[0052] Based on the same inventive concept, the disclosure also provides a computer program product stored in at least one storage medium, wherein the computer program product comprises a plurality of instructions for causing at least one computer device to execute the image retrieval method for enhancing the performance of fine-grained object retrieval as described above.

[0053] Although the disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the disclosure.

Claims

1. An image retrieval method for enhancing fine-grained object retrieval performance, characterized in that, the method comprises: processing the original image based on a multi-scale image cropping technique to obtain a target image and a target image slice set; inputting the target image and the target image slice set into an image encoder to establish an image vector library; performing semantic analysis on the target image and the target image slice set using a multi-modal large model to generate corresponding text descriptions, vector encoding each text description, and establishing a text vector library; matching a query vector with the image vector library for visual similarity and with the text vector library for semantic similarity, respectively obtaining a visual candidate image set and a semantic candidate image set, and generating visual similarity scores and semantic similarity scores; weighting and fusing the visual similarity scores and the semantic similarity scores to obtain final similarity scores, and sorting the visual candidate image set and the semantic candidate image set according to the final similarity scores to obtain the final retrieval result.

2. The method of claim 1, characterized in that, processing the original image based on a multi-scale image cropping technique to obtain a target image and a target image slice set, specifically comprising: uniformly resizing the original image to obtain a target image; using an image segmentation model to identify the target image to obtain each target object and the target object's corresponding target position information; performing multi-scale cropping on the target image according to preset size parameters to obtain an initial image slice set; determining the object inclusion relationship between each initial image slice based on each target object and its target position information, and performing de-duplication processing on the initial image slice set according to the object inclusion relationship to obtain the target image slice set.

3. The method of claim 2, characterized in that, inputting the target image and the target image slice set into an image encoder to establish an image vector library, specifically comprising: inputting the target image and the target image slice set into the image encoder to extract corresponding visual features; fusing each visual feature to establish the image vector library.

4. The method of claim 3, characterized in that, matching a query vector with the image vector library for visual similarity to obtain a visual candidate image set and generate visual similarity scores, specifically comprising: using the query vector to search in the image vector library to obtain an initial candidate image set; calculating the visual similarity scores of each candidate image in the initial candidate image set with the query vector, and determining the visual candidate image set according to the visual similarity scores.

5. The method of claim 4, characterized in that, matching a query vector with the text vector library for semantic similarity to obtain a semantic candidate image set and generate semantic similarity scores, specifically comprising: using the query vector to search in the text vector library to obtain a set of initial candidate text descriptions; determine actual similarities between the query vector and each of the text description information in the text vector library by using a text similarity algorithm, and determine a second initial candidate text description set according to the actual similarities; perform deduplication processing on the first initial candidate text description set and the second initial candidate text description set to obtain an actual recall result; calculate a semantic similarity score for the actual recall result by using a text rerank model, and determine the semantic candidate image set according to the semantic similarity score.

6. The method of claim 5, wherein, weighting and fusing the visual similarity score and the semantic similarity score to obtain a final similarity score, and sorting the visual candidate image set and the semantic candidate image set according to the final similarity score to obtain a final retrieval result, specifically comprising: obtaining an image reverse ranking of each final candidate image in the visual candidate image set and a text reverse ranking of each final candidate image in the semantic candidate image set; determining the final similarity score according to the image reverse ranking and the text reverse ranking.

7. An image retrieval system for enhancing fine-grained object retrieval performance, comprising: a multi-scale image cropping module configured to process an original image based on a multi-scale image cropping technique to obtain a target image and a target image slice set; an image feature extraction module configured to input the target image and the target image slice set into an image encoder to establish an image vector library; an image semantic description generation module configured to perform semantic analysis on the target image and the target image slice set by using a multi-modal large model to generate corresponding text descriptions, and to vectorize each text description to establish a text vector library; a primary retrieval module configured to perform visual similarity matching between a query vector and the image vector library, and semantic similarity matching between the query vector and the text vector library, to obtain a visual candidate image set and a semantic candidate image set, and to generate a visual similarity score and a semantic similarity score; a secondary retrieval module configured to weight and fuse the visual similarity score and the semantic similarity score to obtain a final similarity score, and to sort the visual candidate image set and the semantic candidate image set according to the final similarity score to obtain a final retrieval result.

8. An electronic device comprising at least one processor and at least one memory electrically connected; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image retrieval method for enhancing fine-grained object retrieval performance according to any one of claims 1-6. The memory is electrically connected with the processor, wherein 9. A computer storage medium, comprising: the computer storage medium stores a computer program; the computer program is executed by a processor to implement the image retrieval method for enhancing fine-grained object retrieval performance according to any one of claims 1-6.

10. A computer program product, comprising: ​ The computer program product is stored in at least one storage medium; The computer program product comprises a plurality of instructions for causing at least one electronic device to perform the image retrieval method for enhancing fine-grained object retrieval performance according to any one of claims 1-6.

Images