Image retrieval method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an image retrieval method and device, equipment, a storage medium and a program product. The method comprises the following steps: on the basis of a local feature vector and a global feature vector corresponding to a to-be-retrieved image, matching a plurality of candidate images from a preset retrieval library; calculating a local feature overlap ratio between the to-be-retrieved image and each candidate image based on the local feature vector corresponding to each candidate image and the local feature vector corresponding to the to-be-retrieved image; for each candidate image, based on the corresponding local feature overlap ratio, fusing the global feature vector corresponding to the candidate image and the global feature vector of the to-be-retrieved image to obtain a fused global feature vector corresponding to the candidate image; and calculating the similarity between the global feature vector corresponding to the to-be-retrieved image and the fused global feature vector corresponding to each candidate image, and determining a retrieval result of the to-be-retrieved image based on the similarity. According to the method, the retrieval accuracy can be improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image retrieval method, device, equipment, storage medium and program product. Background Art

[0002] In traditional technology, image retrieval solutions are implemented based on vector retrieval of local features or global semantic features. Semantic feature extraction is first performed to convert the entire image into a one-dimensional feature vector, and then these feature vectors are stored in an efficient vector index library. When a user initiates a retrieval query, the system quickly selects the most similar images and returns them to the user by calculating the vector similarity between the image to be retrieved and the images in the library.

[0003] The above solutions usually represent images based on local features or global semantic features of a single dimensional feature, ignoring other equally important feature information in the image, and also not considering the fusion of multiple features, and the image representation ability is insufficient. This single processing method not only limits the comprehensive mining of image content information, but also fails to achieve the recognition of image detail texture. In particular, when faced with images with significant differences in local features, due to the lack of effective capture and comparison of local details, the retrieval effect is often poor and cannot meet the requirements of accurate retrieval. Summary of the invention

[0004] The embodiments of the present application provide an image retrieval method, apparatus, device, storage medium and program product to solve the problem in that traditional technologies use local features or global semantic features based on single-dimensional features to represent images, which cannot meet accurate retrieval requirements.

[0005] In a first aspect, an embodiment of the present application provides an image retrieval method, comprising:

[0006] Based on the local feature vector and the global feature vector corresponding to the image to be retrieved, multiple candidate images are matched from a preset retrieval library;

[0007] Calculating the local feature overlap between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved;

[0008] For each candidate image, based on the corresponding local feature coincidence, the global feature vector corresponding to the candidate image and the global feature vector of the image to be retrieved are fused to obtain a fused global feature vector corresponding to the candidate image;

[0009] The similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images is calculated, and a retrieval result of the image to be retrieved is determined based on the similarity.

[0010] In a possible implementation, the search library includes a global feature vector index library composed of original global feature vectors corresponding to at least one base library image, and a local feature vector index library composed of original local feature vectors corresponding to the base library image;

[0011] Before matching multiple candidate images from a preset search library based on the local feature vector and the global feature vector corresponding to the image to be searched, the method further includes:

[0012] Performing global feature extraction and local feature extraction on the image to be retrieved respectively to obtain a global feature vector and a local feature vector corresponding to the image to be retrieved;

[0013] The method of matching multiple candidate images from a preset search library based on the local feature vector and the global feature vector corresponding to the image to be searched includes:

[0014] Compare the local feature vector corresponding to the image to be retrieved with the original local feature vector contained in the local feature vector index library to obtain a first candidate set consisting of at least one base library image whose local feature vector matches the image to be retrieved;

[0015] Compare the global feature vector corresponding to the image to be retrieved with the original global feature vector contained in the global feature vector index library to obtain a second candidate set consisting of at least one base library image whose global feature vector matches the image to be retrieved;

[0016] The first candidate set and the second candidate set are combined to obtain a plurality of candidate images.

[0017] In a possible implementation, the calculating the local feature overlap between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved includes:

[0018] Matching the local feature vector of the image to be retrieved with the local feature vectors of each of the candidate images to obtain a matching result;

[0019] According to the matching result, the degree of coincidence of the local features of the image to be retrieved and each of the candidate images is calculated.

[0020] In a possible implementation manner, the local feature vector of the image to be retrieved is used to indicate a feature vector of at least one local feature point in the image to be retrieved; the local feature vector of the base image is used to indicate a feature vector of at least one local feature point in the base image;

[0021] The matching of the local feature vector of the image to be retrieved with the local feature vectors of each of the candidate images to obtain a matching result includes:

[0022] The local feature points contained in the image to be retrieved are bidirectionally matched with the local feature points contained in each of the candidate images to determine the target local feature points in each of the candidate images that are successfully bidirectionally matched with the image to be retrieved, and the number of the target local feature points is used as the matching result.

[0023] In a possible implementation, the bidirectionally matching the local feature points contained in the image to be retrieved with the local feature points contained in each of the candidate images to obtain target local feature points corresponding to the image to be retrieved and each of the candidate images includes:

[0024] For each local feature point P contained in the image to be retrieved n , the local feature point P n Respectively with each of the candidate images C i Contains local feature points r it To match, from each candidate image C i Contains local feature points r it In the process, select the local feature points P n Matching at least one first candidate feature point;

[0025] The first candidate feature point that meets the first preset condition is used as the first target feature point r itPn ;

[0026] For the candidate image C i The local feature points r it , the local feature point r it The local feature points P contained in the image to be retrieved are n Matching is performed from the local feature points P contained in the image to be retrieved n In the middle, filter out the local feature points r it at least one second candidate feature point that matches;

[0027] The second candidate feature point that meets the second preset condition is used as the second target feature point P nrit ;

[0028] When any local feature point P n With any local feature point r it The corresponding second target feature point P nrit is the same local feature point, and the local feature point P n The corresponding first target feature point r itPn With the local feature point rit When the first target feature point r is the same local feature point, itPn As the image to be retrieved and the candidate image C i The corresponding target local feature points.

[0029] In a possible implementation manner, the first candidate feature point that meets the first preset condition is used as the first target feature point C itPn ,include:

[0030] Calculate the local feature point P n The distance between the first candidate feature points and each of the first candidate feature points;

[0031] If the distance corresponding to any of the first candidate feature points satisfies the following proportional relationship, it is considered that the first candidate feature point meets the first preset condition:

[0032] l p1 <λ×l p2

[0033] Among them, l p1 represents the first candidate feature point of p1, l p2 represents any first candidate feature point except the p1th first candidate feature point; λ represents a preset proportional coefficient;

[0034] The second candidate feature point that meets the second preset condition is used as the second target feature point include:

[0035] Calculate the local feature point r it The distance between the second candidate feature points and each of the second candidate feature points;

[0036] If the distance corresponding to any of the second candidate feature points satisfies the following proportional relationship, it is considered that the second candidate feature point meets the second preset condition:

[0037] l q1 <λ×l q2

[0038] Among them, l q1 represents the q1th first candidate feature point, l p2 represents any second candidate feature point except the q1th second candidate feature point; λ represents the preset proportional coefficient.

[0039] In a possible implementation, calculating the degree of overlap of local features between the image to be retrieved and each of the candidate images according to the matching result includes:

[0040] According to the number of target local feature points corresponding to each candidate image, the local feature overlap between the image to be retrieved and each candidate image is calculated, and the expression is:

[0041]

[0042] Among them, w i N represents the local feature overlap between the image to be retrieved and the i-th candidate image; i represents the number of target local feature points corresponding to the i-th candidate image; i∈I, I represents the total number of candidate images.

[0043] In a possible implementation, for each candidate image, based on the corresponding local feature overlap, fusing the global feature vector corresponding to the candidate image and the global feature vector of the image to be retrieved to obtain the fused global feature vector corresponding to the candidate image includes:

[0044] For each candidate image, semantically weighting the global feature vector corresponding to the image to be retrieved and the global feature vectors corresponding to each candidate image based on the corresponding local feature overlap;

[0045] The global feature vector corresponding to the image to be retrieved and the semantically weighted global feature vectors corresponding to each of the candidate images are superimposed to obtain a fused global feature vector corresponding to the candidate image.

[0046] In a possible implementation, for each candidate image, based on the corresponding local feature overlap, the global feature vector corresponding to the candidate image and the global feature vector of the image to be retrieved are fused to obtain the fused global feature vector corresponding to the candidate image, which is expressed as:

[0047] u ci =w i *v T +(1-w i )*v ci

[0048] Among them, u ci represents the fused global feature vector corresponding to the i-th candidate image; w i represents the local feature overlap corresponding to the i-th candidate image; v T represents the global feature vector corresponding to the image to be retrieved; v ci Represents the global feature vector corresponding to the i-th candidate image.

[0049] In a possible implementation, calculating the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images includes:

[0050] The similarity between the global feature vector of the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images is calculated, and the expression is:

[0051]

[0052] Among them, α i represents the similarity between the fused global feature vector corresponding to the i-th candidate image and the global feature vector of the image to be retrieved; v T represents the global feature vector corresponding to the image to be retrieved; ci represents the fused global feature vector corresponding to the i-th candidate image; |v T | represents the modulus length of the global feature vector corresponding to the image to be retrieved; |u ci | represents the modulus of the fused global feature vector corresponding to the i-th candidate image.

[0053] In a possible implementation, determining the search result of the image to be searched based on the similarity includes:

[0054] Sorting the candidate images in descending order of similarity;

[0055] The sorting result is used as the retrieval result of the image to be retrieved and displayed.

[0056] In a second aspect, an embodiment of the present application provides an image retrieval device, comprising:

[0057] A matching module, used to match multiple candidate images from a preset retrieval library based on the local feature vector and the global feature vector corresponding to the image to be retrieved;

[0058] A calculation module, used for calculating the local feature coincidence between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved;

[0059] A fusion module, for fusing, for each candidate image, a global feature vector corresponding to the candidate image and a global feature vector of the image to be retrieved based on the corresponding local feature coincidence, to obtain a fused global feature vector corresponding to the candidate image;

[0060] The sorting module is used to calculate the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images, and determine the retrieval result of the image to be retrieved based on the similarity.

[0061] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0065] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0066] The image retrieval method, apparatus, device, storage medium and program product provided in the embodiments of the present application perform an initial comparison between the image to be retrieved and a preset retrieval library to obtain multiple candidate images; calculate the local feature overlap between the image to be retrieved and each candidate image, thereby increasing the capture of interactive information between images and the use of prior knowledge; integrate the local feature overlap into the global features of the image to be retrieved and each candidate image, thereby realizing the fusion of local information and global information, making full use of local features and global semantic features, capturing more diverse image content information, and enhancing the image representation capability; finally, the candidate images are screened and sorted again by calculating the similarity, thereby greatly improving the accuracy of the retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of a scene of an image retrieval method in an embodiment;

[0069] Figure 2 is a schematic diagram of a flow chart of an image retrieval method in one embodiment;

[0070] Figure 3is another schematic diagram of a flow chart of an image retrieval method in one embodiment;

[0071] Figure 4 The figure is a schematic diagram of the structure of a computer device according to an embodiment.

[0072] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0073] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0074] The image retrieval method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0075] Among them, the user can upload the image to be retrieved to the server 104 through the terminal 102, so that the server 104 matches the local feature vector and the global feature vector corresponding to the image to be retrieved from the preset search library to obtain multiple candidate images; then, based on the local feature vector corresponding to each candidate image and the local feature vector corresponding to the image to be retrieved, the local feature coincidence between the image to be retrieved and each candidate image is calculated; and the global feature vector corresponding to each candidate image is semantically weighted using the local feature coincidence to obtain the weighted global feature vector corresponding to each candidate image; finally, the similarity between the global feature vector corresponding to the image to be retrieved and each weighted global feature vector is calculated, and the candidate images are sorted based on the similarity. In this way, the server 104 feeds back the sorting result as the image retrieval result to the terminal 102 for display.

[0076] In the above-mentioned image retrieval method, the present application first calculates the overlap of local feature points between images, and utilizes the local semantic information of the image, so that the system can focus on specific areas in the image, capture more subtle and key image elements, and improve the ability to recognize image details during the retrieval process; secondly, the local information is combined with the global semantic information to complete a comprehensive analysis of the image content, thereby capturing more diverse image content information, enhancing the ability to represent the image to be retrieved, and improving the accuracy of the retrieval.

[0077] The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0078] In one embodiment, an image retrieval method is provided. This embodiment uses the method applied to the server 104 as an example. Figure 2 As shown, the image retrieval method includes:

[0079] Step 202: Based on the local feature vector and the global feature vector corresponding to the image to be retrieved, multiple candidate images are matched from a preset retrieval library.

[0080] Specifically, the search library includes a large amount of image data, which can come from a variety of channels, such as images collected by web crawlers or images in professional databases. These images are stored in the server 104 after preprocessing such as format conversion and size adjustment. Furthermore, in order to achieve efficient image retrieval function, the base library images in the search library are also subjected to feature extraction based on image features, and the extracted features are stored for subsequent retrieval; wherein, image features can be underlying features such as color, texture, shape, etc., or more advanced semantic features. Furthermore, when retrieving an image to be retrieved, the image to be retrieved can be compared with the base library images in the search library to screen out multiple base library images whose similarity meets the preset requirements as candidate images.

[0081] Step 204: Calculate the local feature coincidence between the image to be retrieved and each candidate image based on the local feature vector corresponding to each candidate image and the local feature vector corresponding to the image to be retrieved.

[0082] Specifically, local feature vectors refer to features extracted from local areas of an image, including edges, corners, lines, curves, and areas with special attributes. Local feature vectors have strong judgment and are invariant to certain transformations such as rotation, translation, and scaling of the image.

[0083] The local feature overlap represents the similarity between the local features of the image to be retrieved and each candidate image. The embodiment of the present application fully utilizes the local semantic information of the image by calculating the local feature overlap between the image to be retrieved and each candidate image, so that the system can focus on specific areas in the image and capture more subtle and key image elements. This feature greatly improves the ability to recognize image details during the retrieval process.

[0084] Step 206 : For each candidate image, based on the corresponding local feature overlap, the global feature vector corresponding to the candidate image is fused with the global feature vector of the image to be retrieved to obtain a fused global feature vector corresponding to the candidate image.

[0085] Specifically, the global feature vector refers to the feature that can represent the entire image. It describes the overall attributes of the image or target, such as color and shape. Compared with local features, global features focus more on describing the entire image. Common global features include color features, texture features, and shape features.

[0086] Furthermore, in this embodiment, a pre-trained feature extraction model is used to complete the global feature extraction of the image to be retrieved or the base image in the retrieval library, capture the global semantic information of the candidate image, and obtain the global feature vector representation of the candidate image. Exemplarily, the feature extraction model is trained based on the Dinov2 model or the ResNet model.

[0087] Step 208: Calculate the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each candidate image, and determine the retrieval result of the image to be retrieved based on the similarity.

[0088] Specifically, by calculating the similarity between the global feature vector of the image to be retrieved and the fused global feature vectors corresponding to each candidate image, the candidate images are screened again to obtain a ranking result.

[0089] The above-mentioned image retrieval method performs an initial comparison between the image to be retrieved and a preset retrieval library to obtain multiple candidate images; calculates the local feature overlap between the image to be retrieved and each candidate image, thereby increasing the capture of interactive information between images and the use of prior knowledge; integrates the local feature overlap into the global features of the image to be retrieved and each candidate image, thereby realizing the fusion of local information and global information, making full use of local features and global semantic features, capturing more diverse image content information, and enhancing the image representation capability; finally, the candidate images are screened and sorted again by calculating the similarity, thereby greatly improving the accuracy of the retrieval.

[0090] See also Figure 3, Optionally, the retrieval library includes a global feature vector index library composed of original global feature vectors corresponding to at least one base library image, and a local feature vector index library composed of original local feature vectors corresponding to the base library images;

[0091] Before step 202, the following steps are included:

[0092] Step 201, performing global feature extraction and local feature extraction on the image to be retrieved, so as to obtain a global feature vector and a local feature vector corresponding to the image to be retrieved;

[0093] Step 202 includes:

[0094] Step 2021: compare the local feature vector corresponding to the image to be retrieved with the original local feature vector contained in the local feature vector index library to form a first candidate set consisting of at least one base library image whose local feature vector matches the image to be retrieved;

[0095] Step 2022: Compare the global feature vector corresponding to the image to be retrieved with the original global feature vector contained in the global feature vector index library to form a second candidate set consisting of at least one base library image whose global feature vector matches the image to be retrieved;

[0096] Step 2023: perform a union process on the first candidate set and the second candidate set to obtain multiple candidate images.

[0097] Specifically, the retrieval library can adopt multiple existing professional databases, and can also use web crawlers to collect the required images as base library images based on the existing professional databases.

[0098] The process of constructing the above-mentioned retrieval library may include: obtaining at least one base library image; performing global feature extraction and local feature extraction on the base library image respectively to obtain the original global feature vector and the original local feature vector corresponding to the base library image; using a preset data structure to store the original global feature vector and the original local feature vector respectively, and obtain the corresponding global feature vector index library and local feature vector index library.

[0099] The preset data structure may be a BallTree tree structure, an HNSW graph structure, etc., which are used to respectively construct a global feature vector index library corresponding to the original global feature vector and a local feature vector index library corresponding to the original local feature vector.

[0100] It should be noted that in this embodiment, a local feature extraction method can be used to extract local feature points and feature point vectors of the image to be retrieved and the base library picture, and then a local feature encoding method can be used to obtain a one-dimensional feature vector representation of the image to be retrieved and the base library picture. Exemplarily, the local feature extraction method of this embodiment includes SIFT (Scale-Invariant Feature Transform, scale-invariant feature transformation algorithm), SURF (Speeded Up Robust Features, accelerated robust feature algorithm), etc.; the local feature encoding method includes VLAD (Vector of Locally Aggregated Descriptors, local aggregation descriptor vector algorithm), FV (Fisher Vector, Fisher Vector algorithm), etc.

[0101] In this embodiment, when constructing the above-mentioned retrieval library and performing global feature extraction on the base library image, the base library image can be resized based on a second preset size. The adjustment process may include: based on the original size of the base library image and the second preset size, calculating the scaling ratio of the base library image in the horizontal and vertical directions, and then determining the adjustment position corresponding to each pixel point on the base library image based on the horizontal scaling ratio and the vertical scaling ratio, and using a bilinear interpolation method to determine the pixel value corresponding to the adjustment position, to obtain the adjusted base library image, and to form a global feature vector of the second preset size.

[0102] The process of extracting global features of the image to be retrieved refers to the process of extracting global features of the underlying image mentioned above, which will not be repeated here.

[0103] The global feature vector corresponding to the image to be retrieved is compared with the original global feature vector contained in the global feature vector index library to form a second candidate set consisting of at least one base library image, which may include: resizing the image to be retrieved based on the second preset size, and the adjustment process may include: calculating the scaling ratio of the image to be retrieved in the horizontal direction and the vertical direction based on the original size of the image to be retrieved and the second preset size, and then determining the adjustment position corresponding to each pixel point on the image to be retrieved based on the scaling ratio in the horizontal direction and the scaling ratio in the vertical direction, and using the bilinear interpolation method to determine the pixel value corresponding to the adjustment position, and obtaining the adjusted image to be retrieved, forming a global feature vector of the second preset size. Further, the global feature vector of the image to be retrieved and the global feature vector of each base library image of the same size are compared, and the global similarity between the image to be retrieved and each base library image is calculated, and a second candidate set is formed by a number of base library images with large global similarity, wherein the second candidate set may include two base library images, which can be understood as using the base library image with the largest global similarity and the base library image with the second largest global similarity to form the second candidate set.

[0104] Finally, the first candidate set and the second candidate set are merged, and repeated images in the base library are deduplicated to obtain the final candidate images.

[0105] Specifically, after extracting the local feature vector and the global feature vector of the image to be retrieved, a two-way vector recall is performed. Among them, the first way of recall is to perform vector retrieval on the local feature vector index library of the extracted local feature vector of the image to be retrieved, and obtain multiple base library images with high local similarity, and these base library images constitute the first candidate set. The other way is to perform vector retrieval on the global feature vector index library of the extracted global feature vector of the image to be retrieved, and obtain multiple base library images with high similarity, and these base library images constitute the second candidate set. The above-mentioned image retrieval method performs an initial comparison between the image to be retrieved and the local feature vector index library and the global feature vector index library, and increases the diversity of the candidate set through multi-way recall, thereby improving the hit rate of image retrieval.

[0106] Optionally, step 204 includes:

[0107] Matching the local feature vector of the image to be retrieved with the local feature vectors of each candidate image to obtain a matching result;

[0108] According to the matching results, the local feature overlap between the image to be retrieved and each candidate image is calculated.

[0109] It should be noted that the local feature vector of the image to be retrieved is used to indicate the feature vector of at least one local feature point in the image to be retrieved; the local feature vector of the base image is used to indicate the feature vector of at least one local feature point in the base image.

[0110] Exemplarily, the step of matching the local feature vector of the image to be retrieved with the local feature vector of each candidate image to obtain a matching result includes: bidirectionally matching the local feature points contained in the image to be retrieved with the local feature points contained in each candidate image to determine the target local feature points in each candidate image that successfully match the image to be retrieved bidirectionally, and taking the number of target local feature points as the matching result.

[0111] In step 201, when the image to be retrieved is subjected to local feature extraction, a local feature point dataset β can be obtained. T ={p1,p2,…p n …,p N}, where P n represents the nth local feature point in the image to be retrieved, n∈N, N represents the total number of local feature points in the image to be retrieved, and each local feature point vector p n are all multi-dimensional vectors. Candidate image c i After local feature extraction, the local feature point dataset β is obtained. i = {r i1 ,r i2 ,…,r it ,…,r iT}, where r it represents the tth local feature point in the ith candidate image, t∈T, T is the total number of local feature points in the ith candidate image, i∈I, I represents the total number of candidate images, and each local feature point vector r it are all multi-dimensional vectors.

[0112] Exemplarily, the step of bidirectionally matching the local feature points contained in the image to be retrieved with the local feature points contained in each candidate image to obtain the target local feature points corresponding to the image to be retrieved and each candidate image, and taking the number of the target local feature points as the matching result may include: extracting each local feature point P for the image to be retrieved n , using a matcher, the local feature point P n Respectively with each candidate image C i Contains local feature points r it Matching is performed, from each candidate image C i Contains local feature points r it and selecting at least one first candidate feature point; and using the first candidate feature point that meets the first preset condition as the first target feature point. Furthermore, for the candidate image C i The local feature points r it , using a matcher, the local feature point r it Respectively with the local feature points P contained in the image to be retrieved n Matching is performed from the local feature points P contained in the image to be retrieved n , selecting at least one second candidate feature point; and using the second candidate feature point that meets the second preset condition as the second target feature point When the local feature point P n With local feature point r it The corresponding second target feature point is the same local feature point, and the local feature point P n The corresponding first target feature point With local feature point r it For the same local feature point, the first target feature point As the image to be retrieved and the candidate image C i The corresponding target local feature points. The matchers used in this embodiment include Brute-Force matcher, FLANN matcher, etc.

[0113] The first candidate feature point that meets the first preset condition is used as the first target feature point. The process includes: calculating the local feature point P n The distance between each first candidate feature point; if the distance corresponding to any first candidate feature point satisfies the following proportional relationship, it is considered that the first candidate feature point meets the first preset condition:

[0114] l p1 <λ×l p2

[0115] Among them, l p1 represents the first candidate feature point of p1, l p2 represents any first candidate feature point except the p1th first candidate feature point; λ represents a preset proportional coefficient;

[0116] The second candidate feature point that meets the second preset condition is used as the second target feature point. Including: Calculate local feature points r it The distance between each second candidate feature point; if the distance corresponding to any second candidate feature point satisfies the following proportional relationship, it is considered that the second candidate feature point meets the second preset condition:

[0117] l q1 <λ×l q2

[0118] Among them, lq1 represents the q1th first candidate feature point, l p2 represents any second candidate feature point except the q1th second candidate feature point; λ represents a preset proportional coefficient.

[0119] Furthermore, the above calculation of the local feature overlap between the image to be retrieved and each candidate image based on the matching results includes:

[0120] According to the number of target local feature points corresponding to each candidate image, the local feature overlap between the image to be retrieved and each candidate image is calculated, and its expression is:

[0121]

[0122] Among them, w i N represents the local feature overlap between the image to be retrieved and the i-th candidate image; i Represents the number of target local feature points corresponding to the i-th candidate image; i∈I, I represents the total number of candidate images.

[0123] For example, local feature extraction is performed on the image to be retrieved to obtain local feature points P1 and P2; local feature extraction is performed on the candidate image c1 matched in the retrieval library to obtain the local feature point r corresponding to the candidate image c1. 11 ,r 12 ,r 13 .

[0124] Use the matcher to match the local feature point P1 with the local feature point r corresponding to the candidate image c1. 11 ,r 12 ,r 13 Matching is performed, and the local feature point P2 is matched with the local feature point r corresponding to the candidate image c1. 11 ,r 12 ,r 13 For example, the local feature point P1 is matched to the r 11 ,r 12 ,r 13 Three first candidate feature points, r corresponding to local feature point P2 11 ,r 12 Two candidate feature points.

[0125] Furthermore, for the local feature point P1 corresponding to r 11 ,r 12 ,r 13 The three first candidate feature points are firstly 11 This first candidate feature point calculates r 12 The product of the preset proportional coefficient, r 13The product of the preset proportional coefficient and the 11 Whether it is less than r at the same time 12 The product of the preset proportional coefficient, r 13 The product of the preset proportional coefficient, if so, r 11 As the first target feature point; if not, continue to target r 12 This first candidate feature point calculates r 11 The product of the preset proportional coefficient, r 13 The product of the preset proportional coefficient and the 12 Whether it is less than r at the same time 11 The product of the preset proportional coefficient, r 13 The product of the preset proportional coefficient, if so, r 12 As the first target feature point; if not, continue to target r 13 This first candidate feature point calculates r 11 The product of the preset proportional coefficient, r 12 The product of the preset proportional coefficient and the 23 Whether it is less than r at the same time 11 The product of the preset proportional coefficient, r 12 The product of the preset proportional coefficient, if so, r 13 As the first target feature point, otherwise, it is considered that there is no first target feature point in the candidate image c1. 13 At the same time, it is less than r 11 The product of the preset proportional coefficient, r 12 The product of the preset proportional coefficient is r 13 As the first target feature point corresponding to the local feature point P1. 12 As the first target feature point corresponding to the local feature point P2.

[0126] Accordingly, a matcher is used to match the local feature points r contained in the candidate image c1 11 Match with local feature points P1 and P2 respectively, and match the local feature point r 12 Match with local feature points P1 and P2 respectively, local feature point r 13 Match with local feature points P1 and P2 respectively, and select the local feature points r 11 The corresponding P1 and P2 are used as the second candidate feature points; and the local feature point r 12 The matched P1 and P2 are used as the second candidate feature points.

[0127] Similarly, match P1 as the local feature point r 11 The corresponding second target feature point, P1, is used as the local feature point r 12The corresponding second target feature point, P1, is used as the local feature point r 13 The corresponding second target feature point.

[0128] Finally, for the local feature point P1, it can be known that its corresponding first target feature point is r 13 , and the local feature point r 13 The corresponding second target feature point is also P1. At this time, the local feature point P1 and the local feature point r 13 If the two-way matching is successful, then the local feature point r 13 is the target local feature point of the local feature point P1.

[0129] For the local feature point P2, it can be seen that its corresponding first target feature point is r 12 , however, the local feature point r 12 The corresponding second target feature point is P1. At this time, the local feature point P2 and the local feature point r 12 Bidirectional matching fails, local feature point r 12 It cannot be used as the target local feature point of the local feature point P2.

[0130] At this time, it can be determined that the candidate image c1 contains only one target local feature point, and the matching result is 1.

[0131] The above-mentioned image retrieval method obtains the number of target local feature points that completely match the image to be retrieved and each candidate image by matching the local feature points of the image to be retrieved with each candidate image one by one, and then obtains the local feature overlap between the image to be retrieved and each candidate image based on the number of target local feature points. This scheme increases the capture of interactive information between images and the use of prior knowledge. By using the local semantic information of the image, the system can focus on specific areas in the image, capture more subtle and key image elements, and improve the ability to identify image details during the retrieval process.

[0132] Optionally, step 206 includes: for each candidate image, based on the corresponding local feature coincidence, semantically weighting the global feature vector corresponding to the image to be retrieved and the global feature vector corresponding to each candidate image;

[0133] The semantically weighted global feature vector corresponding to the image to be retrieved and the semantically weighted global feature vectors corresponding to each candidate image are superimposed to obtain a fused global feature vector corresponding to the candidate image.

[0134] Specifically, the expression of step 206 is:

[0135] u ci =w i *v T +(1-w i)*v ci

[0136] Among them, u ci represents the fused global feature vector corresponding to the i-th candidate image; w i represents the local feature overlap corresponding to the i-th candidate image; v T Represents the global feature vector corresponding to the image to be retrieved; v ci Represents the global feature vector corresponding to the i-th candidate image.

[0137] Specifically, in conventional technologies, there are many ways to integrate local feature information into global feature information, such as feature concatenation, feature summation, etc. In this embodiment, a semantic feature weight method is used, and the local feature overlap of each candidate image is used as a weight to re-weight the global feature information.

[0138] The above-mentioned image retrieval method uses the local feature overlap as the weight and re-weights the global feature information, thereby realizing the fusion of local information and global information, making full use of local features and global semantic features, capturing more diverse image content information, and enhancing the image representation capability.

[0139] Optionally, in step 208, the expression is:

[0140]

[0141] Among them, α i represents the similarity between the fused global feature vector corresponding to the i-th candidate image and the global feature vector of the image to be retrieved; v T Represents the global feature vector corresponding to the image to be retrieved; u ci represents the fused global feature vector corresponding to the i-th candidate image; |v T | represents the modulus length of the global feature vector corresponding to the image to be retrieved; |u ci | represents the modulus of the fused global feature vector corresponding to the i-th candidate image.

[0142] Specifically, in conventional technology, there are many ways to calculate the similarity between two vectors, such as cosine similarity, Euclidean distance, Pearson correlation coefficient, etc. In this embodiment, the cosine similarity method is adopted, and the vector similarity between the global feature vector of the image to be retrieved and the fused global feature vector corresponding to each candidate image is obtained by measuring the cosine value of the angle between the global feature vector of the image to be retrieved and the fused global feature vector corresponding to each candidate image.

[0143] Furthermore, in step 206, the similarity α iThe candidate images are sorted from high to low; finally, the sorting result is displayed as the retrieval result of the image to be retrieved.

[0144] The above-mentioned image retrieval method re-screens multiple candidate images obtained by the initial comparison by calculating the global feature vector of the image to be retrieved and the similarity between the fused global feature vector corresponding to each candidate image that has fused local feature information and global feature information, thereby greatly improving the accuracy of the retrieval.

[0145] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] The above-mentioned image retrieval method performs an initial comparison between the image to be retrieved and the preset retrieval library, increases the diversity of the candidate set through a multi-way recall method, and improves the hit rate of the image retrieval.

[0147] Secondly, by matching the local feature points of the image to be retrieved and each candidate image one by one, the number of target local feature points that completely match the two is obtained, and then the local feature overlap between the image to be retrieved and each candidate image is obtained based on the number of target local feature points. The capture of interactive information between images and the use of prior knowledge are increased. By using the local semantic information of the image, the system can focus on specific areas in the image, capture more subtle and key image elements, and improve the ability to identify image details during the retrieval process.

[0148] Thirdly, the local feature overlap is used as the weight to re-weight the global feature information, thus realizing the fusion of local information and global information, making full use of local features and global semantic features, capturing more diverse image content information, and enhancing the image representation capability.

[0149] Finally, by calculating the similarity between the global feature vector of the image to be retrieved and the fused global feature vector corresponding to each candidate image that fuses local feature information and global feature information, the multiple candidate images obtained by the initial comparison were screened again, greatly improving the accuracy of the retrieval. In addition, the screened images were sorted in descending order of similarity, and the sorting results were displayed to the user, improving the user experience.

[0150] Based on the same inventive concept, the embodiment of the present application also provides an image retrieval device for implementing the above-mentioned image retrieval method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned image retrieval method, so the specific limitations in one or more device embodiments provided below can refer to the limitations on the image retrieval method above, and will not be repeated here.

[0151] In one embodiment, an image retrieval device is provided, comprising: an image recall module, a sorting module, an image processing module and a display module.

[0152] A matching module, used to match multiple candidate images from a preset retrieval library based on the local feature vector and the global feature vector corresponding to the image to be retrieved;

[0153] A calculation module, used for calculating the local feature coincidence between the image to be retrieved and each candidate image based on the local feature vector corresponding to each candidate image and the local feature vector corresponding to the image to be retrieved;

[0154] A fusion module is used to fuse the global feature vector corresponding to each candidate image with the global feature vector of the image to be retrieved based on the corresponding local feature coincidence degree, so as to obtain a fused global feature vector corresponding to the candidate image;

[0155] The sorting module is used to calculate the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each candidate image, and determine the retrieval result of the image to be retrieved based on the similarity.

[0156] Optionally, the retrieval library includes a global feature vector index library composed of original global feature vectors corresponding to at least one base library image, and a local feature vector index library composed of original local feature vectors corresponding to the base library image;

[0157] The matching module is also configured to:

[0158] Performing global feature extraction and local feature extraction on the image to be retrieved respectively to obtain a global feature vector and a local feature vector corresponding to the image to be retrieved; and

[0159] Compare the local feature vector corresponding to the image to be retrieved with the original local feature vector contained in the local feature vector index library to obtain a first candidate set consisting of at least one base library image whose local feature vector matches the image to be retrieved;

[0160] Compare the global feature vector corresponding to the image to be retrieved with the original global feature vector contained in the global feature vector index library to obtain a second candidate set consisting of at least one base library image whose global feature vector matches the image to be retrieved;

[0161] The first candidate set and the second candidate set are combined to obtain multiple candidate images.

[0162] Optionally, the computing module is further configured to:

[0163] Matching the local feature vector of the image to be retrieved with the local feature vectors of each candidate image to obtain a matching result;

[0164] According to the matching results, the local feature overlap between the image to be retrieved and each candidate image is calculated.

[0165] Optionally, the local feature vector of the image to be retrieved is used to indicate the feature vector of at least one local feature point in the image to be retrieved; the local feature vector of the base image is used to indicate the feature vector of at least one local feature point in the base image;

[0166] The computing module is also configured to:

[0167] The local feature points contained in the image to be retrieved are bidirectionally matched with the local feature points contained in each candidate image to determine the target local feature points in each candidate image that are successfully bidirectionally matched with the image to be retrieved, and the number of target local feature points is taken as the matching result.

[0168] Optionally, the computing module is further configured to:

[0169] For each local feature point P contained in the image to be retrieved n , the local feature point P n Respectively with each candidate image C i Contains local feature points r it Matching is performed, from each candidate image C i Contains local feature points r it In the middle, filter out the local feature points P n Matching at least one first candidate feature point;

[0170] The first candidate feature point that meets the first preset condition is used as the first target feature point r itPn ;

[0171] For candidate image C iThe local feature points r it , the local feature point r it Respectively with the local feature points P contained in the image to be retrieved n Matching is performed from the local feature points P contained in the image to be retrieved n In the middle, filter out the local feature points r it at least one second candidate feature point that matches;

[0172] The second candidate feature point that meets the second preset condition is used as the second target feature point

[0173] When any local feature point P n With any local feature point r it The corresponding second target feature point is the same local feature point, and the local feature point P n The corresponding first target feature point With the local feature point r it For the same local feature point, the first target feature point As the image to be retrieved and the candidate image C i The corresponding target local feature points.

[0174] Optionally, the computing module is further configured to:

[0175] Calculate the local feature point P n The distance between each first candidate feature point;

[0176] If the distance corresponding to any first candidate feature point satisfies the following proportional relationship, it is considered that the first candidate feature point meets the first preset condition:

[0177] l p1 <λ×l p2

[0178] Among them, l p1 represents the first candidate feature point of p1, l p2 represents any first candidate feature point except the p1th first candidate feature point; λ represents a preset proportional coefficient; and

[0179] Calculate local feature points r it The distance between each second candidate feature point;

[0180] If the distance corresponding to any second candidate feature point satisfies the following proportional relationship, it is considered that the second candidate feature point meets the second preset condition:

[0181] l q1 <λ×l q2

[0182] Among them, l q1 represents the q1th first candidate feature point, l p2 represents any second candidate feature point except the q1th second candidate feature point; λ represents a preset proportional coefficient.

[0183] Optionally, the computing module is further configured to:

[0184] According to the number of target local feature points corresponding to each candidate image, the local feature overlap between the image to be retrieved and each candidate image is calculated, and its expression is:

[0185]

[0186] Among them, w i N represents the local feature overlap between the image to be retrieved and the i-th candidate image; i Represents the number of target local feature points corresponding to the i-th candidate image; i∈I, I represents the total number of candidate images.

[0187] Optionally, the fusion module is further configured to:

[0188] For each candidate image, semantic weighting is performed on the global feature vector corresponding to the image to be retrieved and the global feature vector corresponding to each candidate image based on the corresponding local feature overlap;

[0189] The global feature vector corresponding to the image to be retrieved and the semantically weighted global feature vectors corresponding to each candidate image are superimposed to obtain a fused global feature vector corresponding to the candidate image.

[0190] Optionally, the fusion module is further configured to:

[0191] u ci =w i *v T +(1-w i )*v ci

[0192] Among them, u ci represents the fused global feature vector corresponding to the i-th candidate image; w i represents the local feature overlap corresponding to the i-th candidate image; v T Represents the global feature vector corresponding to the image to be retrieved; v ci Represents the global feature vector corresponding to the i-th candidate image.

[0193] Optionally, the sorting module is further configured to:

[0194] Calculate the similarity between the global feature vector of the image to be retrieved and the fused global feature vectors corresponding to each candidate image. The expression is:

[0195]

[0196] Among them, α i represents the similarity between the fused global feature vector corresponding to the i-th candidate image and the global feature vector of the image to be retrieved; v T Represents the global feature vector corresponding to the image to be retrieved; u ci represents the fused global feature vector corresponding to the i-th candidate image; |v T | represents the modulus length of the global feature vector corresponding to the image to be retrieved; |u ci | represents the modulus of the fused global feature vector corresponding to the i-th candidate image.

[0197] Optionally, the sorting module is further configured to:

[0198] Sort the candidate images in descending order of similarity;

[0199] The ranking results are displayed as the retrieval results of the image to be retrieved.

[0200] Each module in the above device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0201] See also Figure 4 , Figure 4 The figure is a schematic diagram of the structure of the computer device provided by the present application. The computer device 40 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the computer device 40 also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus.

[0202] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0203] The specific implementation process of the processor 401 can refer to the above-mentioned image retrieval method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0204] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0205] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0206] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0207] The present application also provides a computer program product, including a computer program, which implements the above-mentioned image retrieval method when executed by a processor.

[0208] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned image retrieval method is implemented.

[0209] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0210] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0211] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0214] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0215] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0216] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image retrieval method, characterized in that: The method comprises: Based on the local feature vector and the global feature vector corresponding to the image to be retrieved, multiple candidate images are matched from a preset retrieval library; Calculating the local feature overlap between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved; For each candidate image, based on the corresponding local feature coincidence, the global feature vector corresponding to the candidate image and the global feature vector of the image to be retrieved are fused to obtain a fused global feature vector corresponding to the candidate image; The similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images is calculated, and a retrieval result of the image to be retrieved is determined based on the similarity.

2. The method according to claim 1, characterized in that: The retrieval library comprises a global feature vector index library composed of original global feature vectors corresponding to at least one base library image, and a local feature vector index library composed of original local feature vectors corresponding to the base library image; Before matching multiple candidate images from a preset search library based on the local feature vector and the global feature vector corresponding to the image to be searched, the method further includes: Performing global feature extraction and local feature extraction on the image to be retrieved respectively to obtain a global feature vector and a local feature vector corresponding to the image to be retrieved; The method of matching multiple candidate images from a preset search library based on the local feature vector and the global feature vector corresponding to the image to be searched includes: Compare the local feature vector corresponding to the image to be retrieved with the original local feature vector contained in the local feature vector index library to obtain a first candidate set consisting of at least one base library image whose local feature vector matches the image to be retrieved; Compare the global feature vector corresponding to the image to be retrieved with the original global feature vector contained in the global feature vector index library to obtain a second candidate set consisting of at least one base library image whose global feature vector matches the image to be retrieved; The first candidate set and the second candidate set are combined to obtain a plurality of candidate images.

3. The method according to claim 1, characterized in that The calculating the local feature coincidence between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved comprises: Matching the local feature vector of the image to be retrieved with the local feature vectors of each of the candidate images to obtain a matching result; According to the matching result, the degree of coincidence of the local features of the image to be retrieved and each of the candidate images is calculated.

4. The method according to claim 3, characterized in that: The local feature vector of the image to be retrieved is used to indicate the feature vector of at least one local feature point in the image to be retrieved; the local feature vector of the base image is used to indicate the feature vector of at least one local feature point in the base image; The matching of the local feature vector of the image to be retrieved with the local feature vectors of each of the candidate images to obtain a matching result includes: The local feature points contained in the image to be retrieved are bidirectionally matched with the local feature points contained in each of the candidate images to determine the target local feature points in each of the candidate images that are successfully bidirectionally matched with the image to be retrieved, and the number of the target local feature points is used as the matching result.

5. The method according to claim 4, characterized in that The step of bidirectionally matching the local feature points contained in the image to be retrieved with the local feature points contained in each of the candidate images to obtain target local feature points corresponding to the image to be retrieved and each of the candidate images includes: For each local feature point P contained in the image to be retrieved n , the local feature point P n Respectively with each of the candidate images C i Contains local feature points r it To match, from each candidate image C i Contains local feature points r it In the process, select the local feature points P n Matching at least one first candidate feature point; The first candidate feature point that meets the first preset condition is used as the first target feature point For the candidate image C i The local feature points r it , the local feature point r it The local feature points P contained in the image to be retrieved are n Matching is performed from the local feature points P contained in the image to be retrieved n In the middle, filter out the local feature points r it at least one second candidate feature point that matches; The second candidate feature point that meets the second preset condition is used as the second target feature point When any local feature point P n With any local feature point r it The corresponding second target feature point is the same local feature point, and the local feature point P n The corresponding first target feature point With the local feature point r it When the first target feature point is the same local feature point, As the image to be retrieved and the candidate image C i The corresponding target local feature points.

6. The method according to claim 5, characterized in that The first candidate feature point that meets the first preset condition is used as the first target feature point include: Calculate the local feature point P n The distance between the first candidate feature points and each of the first candidate feature points; If the distance corresponding to any of the first candidate feature points satisfies the following proportional relationship, it is considered that the first candidate feature point meets the first preset condition: l p1 <λ×l p2 Among them, l p1 represents the first candidate feature point of p1, l p2 represents any first candidate feature point except the p1th first candidate feature point; λ represents a preset proportional coefficient; The second candidate feature point that meets the second preset condition is used as the second target feature point include: Calculate the local feature point r it The distance between the second candidate feature points and each of the second candidate feature points; If the distance corresponding to any of the second candidate feature points satisfies the following proportional relationship, it is considered that the second candidate feature point meets the second preset condition: l q1 <λ×l q2 Among them, l q1 represents the q1th first candidate feature point, l p2 represents any second candidate feature point except the q1th second candidate feature point; λ represents the preset proportional coefficient.

7. The method according to claim 4, characterized in that The calculating, based on the matching result, the degree of coincidence of the local features of the image to be retrieved and each of the candidate images comprises: According to the number of target local feature points corresponding to each candidate image, the local feature overlap between the image to be retrieved and each candidate image is calculated, and the expression is: Among them, w i N represents the local feature overlap between the image to be retrieved and the i-th candidate image; i represents the number of target local feature points corresponding to the i-th candidate image; i∈I, I represents the total number of candidate images.

8. The method according to any one of claims 1 to 7, characterized in that The step of fusing, for each candidate image, a global feature vector corresponding to the candidate image and a global feature vector of the image to be retrieved based on the corresponding local feature overlap to obtain a fused global feature vector corresponding to the candidate image includes: For each candidate image, semantically weighting the global feature vector corresponding to the image to be retrieved and the global feature vectors corresponding to each candidate image based on the corresponding local feature overlap; The global feature vector corresponding to the image to be retrieved and the semantically weighted global feature vectors corresponding to each of the candidate images are superimposed to obtain a fused global feature vector corresponding to the candidate image.

9. The method according to any one of claims 1 to 7, characterized in that: For each candidate image, based on the corresponding local feature overlap, the global feature vector corresponding to the candidate image and the global feature vector of the image to be retrieved are fused to obtain the fused global feature vector corresponding to the candidate image, which is expressed as: u ci =w i *v T +(1-w i )*v ci Among them, u ci represents the fused global feature vector corresponding to the i-th candidate image; w i represents the local feature overlap corresponding to the i-th candidate image; v T represents the global feature vector corresponding to the image to be retrieved; v ci Represents the global feature vector corresponding to the i-th candidate image.

10. The method according to claim 1, characterized in that The calculating the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images includes: The similarity between the global feature vector of the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images is calculated, and the expression is: Among them, α i represents the similarity between the fused global feature vector corresponding to the i-th candidate image and the global feature vector of the image to be retrieved; v T represents the global feature vector corresponding to the image to be retrieved; ci represents the fused global feature vector corresponding to the i-th candidate image; |v T | represents the modulus length of the global feature vector corresponding to the image to be retrieved; |u ci | represents the modulus of the fused global feature vector corresponding to the i-th candidate image.

11. The method according to claim 1, characterized in that: The step of determining the retrieval result of the image to be retrieved based on the similarity comprises: Sorting the candidate images in descending order of similarity; The sorting result is used as the retrieval result of the image to be retrieved and displayed.

12. An image retrieval device, characterized in that: include: A matching module, used to match multiple candidate images from a preset retrieval library based on the local feature vector and the global feature vector corresponding to the image to be retrieved; A calculation module, used for calculating the local feature coincidence between the image to be retrieved and each of the candidate images based on the local feature vectors corresponding to each of the candidate images and the local feature vectors corresponding to the image to be retrieved; A fusion module, for fusing, for each candidate image, a global feature vector corresponding to the candidate image and a global feature vector of the image to be retrieved based on the corresponding local feature coincidence, to obtain a fused global feature vector corresponding to the candidate image; The sorting module is used to calculate the similarity between the global feature vector corresponding to the image to be retrieved and the fused global feature vectors corresponding to each of the candidate images, and determine the retrieval result of the image to be retrieved based on the similarity.

13. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

Images