A precise matching method for recalling images from large-scale information databases

Abstract

The present invention discloses a precise matching method for recalled images from a large-scale information database. For an input retrieval image x, k recalled images are first obtained from the image feature database db. A first-order differential calculation is then performed on the recalled image similarity distances, and the recalled images are segmented. The matching data is then recombined based on a set threshold to obtain matching image sets tgt0, tgt1, and tgt2. After filtering for non-matches and possible matches, and performing precise matching, the results of sets tgt0 and tgt1' are combined to obtain tgt_final, thereby obtaining the final matching result. Compared to the prior art, the method employed by the present invention can more accurately match filtered images recalled from a large-scale information database, effectively meeting the requirements for high-precision retrieval.

Description

Technical Field

[0001] The present invention relates to an image matching method, and in particular to an accurate matching method for recalling images from a large-scale information database. Background Art

[0002] In the existing technology, accurate image retrieval, especially accurate retrieval of large-scale images, generally has two steps in structure: recall and precise matching. Recall refers to finding the k most similar images from a large-scale image library, generally called top k; precise matching refers to accurately filtering and matching these top k images to match the desired results.

[0003] Existing content-based image retrieval methods typically obtain retrieval results through similarity measurement algorithms. Image features are typically trained on deep learning networks to generate image feature models. To improve retrieval efficiency, quantitative indexes are typically generated through deep hash mapping or kmeans clustering. However, this approach still suffers from poor retrieval accuracy for massive image volumes and cannot meet high-precision retrieval requirements. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for more accurately matching filtered images after recalling a large-scale information library, in response to the shortcomings of the existing technology.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0006] A method for accurately matching recalled images in a large-scale information database comprises the following steps: step S1, constructing an image feature database db based on image features of existing images; step S2, inputting a search image x, finding multiple image features similar to the search image x from the image feature database db, and obtaining k recalled images; step S3, performing first-order differential calculation on the similarity distance of the recalled images; step S4, performing segmentation processing on the recalled images; step S5, recombining matching data based on a set threshold to obtain matching image sets tgt0, tgt1, and tgt2; step S6, performing non-matching and possible matching filtering; step S7, accurately matching: matching images in set tgt1 using a preset accurate matching algorithm to obtain a new target image set tgt1'; step S8, merging the results of sets tgt0 and tgt1' to obtain tgt_final, thereby obtaining a final matching result.

[0007] Preferably, in step S1, a preset deep learning model is used to convert the existing image into a vector with consistent length, and the vector is used as the image feature.

[0008] Preferably, in the image feature database db, the image format corresponding to each image feature is jpg, webp or bmp format.

[0009] Preferably, in step S2, the process of recalling images includes: step S20, using the deep learning model in step S1 to convert the retrieved image x into image features, and then normalizing it to obtain feature xq; step S21, using a preset retrieval algorithm to search for image features similar to feature xq from the image feature library db, sorting the search results in order of similarity, and then obtaining the most similar k images and the vector distance between each image and feature xq, wherein the k vector distances are set to ds = [d(1), d(2),…, d(k)], and ds is used as the similarity between the k images and xq, and the images that are ranked closer to the front have higher similarity with xq.

[0010] In the method for accurately matching recalled images from a large-scale information library disclosed in the present invention, for an input retrieval image x, k recalled images are first obtained from the image feature library db. The first-order differential of the recalled image similarity distance is then calculated, and the recalled images are segmented. The matching data is then recombined based on a set threshold to obtain matching image sets tgt0, tgt1, and tgt2. After filtering for non-matches and possible matches, and performing accurate matching, the results of sets tgt0 and tgt1' are combined to obtain tgt_final, thereby obtaining the final matching result. Compared to the prior art, the method adopted by the present invention can more accurately match filtered images recalled from a large-scale information library, better meeting the requirements of high-precision retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of the accurate matching method for recalling images from a large-scale information database of the present invention. DETAILED DESCRIPTION

[0012] The present invention will be described in more detail below with reference to the accompanying drawings and embodiments.

[0013] This invention discloses a method for accurately matching recalled images from a large-scale information base. Figure 1 , which includes the following steps:

[0014] Step S1, constructing an image feature database db based on image features of existing images;

[0015] Step S2: input a search image x, find multiple image features similar to the search image x from the image feature database db, and obtain k recall images;

[0016] Step S3, perform first-order differential calculation on the similarity distance of the recalled images:

[0017] Define a recall function y = f(x,n), where n = 1, 2, 3…, k, then there is a corresponding similarity distance matrix ds(x) = [f(x,1), f(x,2),…, f(x,k)];

[0018] Assume d(i) = f(x,i+1)-f(x,i), then we can calculate:

[0019] d(1)=f(x,2)-f(x,1);

[0020] d(2)=f(x,3)-f(x,2),...;

[0021] d(k-1)=f(x,k)-f(x,k-1);

[0022] Where d is the first-order differential of f, and then (k-1) differential values ​​are obtained. In the k recalled images, the size of the d(i) value is used to reflect the similarity of the images;

[0023] According to the above differential values, we can get the set d:{d(1),d(2),...,d(k-1)};

[0024] Let the maximum value of the element in d be dmax, and the corresponding image number is q, then: dmax = f(x,q+1)-f(x,q), which means that the distance between the image group to the left of q and the image group to the right of q is the largest;

[0025] Step S4, segment the recalled image:

[0026] Using the differential calculation results obtained in step S3, the recalled images are clustered according to their similarity, and the distance space ds(x) of the image features is divided into four subspaces, corresponding to ds0, ds1, ds2 and ds3 respectively, and three segmentation metrics dmax0, dmax1 and dmax2 are obtained at the same time;

[0027] Step S5, recombining the matching data based on the set threshold to obtain matching image sets tgt0, tgt1 and tgt2;

[0028] Step S6: Filter non-matching and possible matching:

[0029] Put the first 5 values ​​in set tgt2 into set tgt1. If there are less than 5 values ​​in set tgt2, put all of them into set tgt1.

[0030] Step S7, accurate matching:

[0031] Match the images in the set tgt1 using a preset exact matching algorithm to obtain a new target image set tgt1';

[0032] Step S8: Merge the results of the set tgt0 and the set tgt1' to obtain tgt_final, thereby obtaining the final matching result.

[0033] In the above method, for an input search image x, k recalled images are first obtained from the image feature database db. The first-order differential of the recalled image similarity distance is then calculated, and the recalled images are segmented. The matching data is then recombined based on a set threshold to obtain matching image sets tgt0, tgt1, and tgt2. After filtering for non-matches and possible matches, and performing exact matching, the results of sets tgt0 and tgt1' are combined to obtain tgt_final, which then yields the final matching result. Compared to existing technologies, the above method employed by the present invention can achieve more accurate matching of filtered images recalled from large-scale information repositories, better meeting the requirements of high-precision retrieval.

[0034] In step S1 of this embodiment, a preset deep learning model is used to convert the existing image into a vector of consistent length, and the vector is used as the image feature. Furthermore, in the image feature database db, the image format corresponding to each image feature is jpg, webp or bmp format.

[0035] Regarding the specific implementation means of recalling images, in step S2 of this embodiment, the process of recalling images includes:

[0036] Step S20, using the deep learning model in step S1 to convert the search image x into image features, and then perform normalization processing to obtain features xq;

[0037] In step S21, a preset retrieval algorithm is used to search for image features similar to feature xq from the image feature library db, and the search results are sorted in order of similarity to obtain the k most similar images and the vector distance between each image and feature xq, wherein the k vector distances are set to ds = [d(1), d(2), ..., d(k)], and ds is used as the similarity between the k images and xq, and the images that are ranked higher have higher similarity with xq.

[0038] As a preferred embodiment, in step S4, the process of clustering the recalled images according to their similarity includes:

[0039] Step S40, finding the maximum value dmax0 of the elements in the set d, the maximum value dmax0 divides d into two parts: ds0 and ds_tmp0, where ds0 = [d(1), d(2), ..., d(q)], ds_tmp0 = [d(q+1), d(q+2), ..., d(k-1)];

[0040] Step S41, taking the maximum value dmax1 of the elements in ds_tmp0, corresponding to sequence number q1, and dividing ds_tmp0 into two segments: ds1 and ds_tmp1, where ds1 = [d(q+1), d(q+2), ..., d(q1)] and ds_tmp1 = [d(q1+1), d(q1+2), ..., d(k-1)];

[0041] Step S42, taking the maximum value dmax2 of the elements in ds_tmp1, corresponding to sequence number q2, and dividing ds_tmp1 into two segments: ds2 and ds3, where ds2 = [d(q2+1), d(q2+2), ..., d(q2)] and ds3 = [d(q2+1), d(q2+2), ..., d(k-1)];

[0042] The set ds is divided into four parts using the maximum value dmax0, the maximum value dmax1 and the maximum value dmax2, and the four parts correspond to ds0, ds1, ds2 and ds3.

[0043] In order to filter out the results that are sure to match, in step S5 of this embodiment, three thresholds U1, U2 and U3 are defined, where:

[0044] U1 = 0.35, used to confirm the smoothness of ds and confirm the matching graph;

[0045] U2=0.1 is used to confirm the situation of distinguishing the matching graph from other graphs;

[0046] U3=0.04, used in the following cases:

[0047] The first case is that the values ​​of the elements in the ds set are small and smooth, indicating that the recalled images are similar to each other. In this case, if dmax0 is less than or equal to U3, all values ​​less than or equal to U1 in ds are taken to obtain the matching image set tgt0, tgt1 and tgt2 are taken as empty sets, and then jump to step S6;

[0048] The second case is that the value of ds is divided into two parts: dmax0 is greater than or equal to U2, and dmax1 and dmax2 are both less than or equal to U3. Since ds0 is the most similar subset, the elements of the ds0 set are directly put into the set tgt0, the set tgt1 is set to be an empty set, the set tgt2 is set to be the union of ds1, ds2 and ds3, and then jump to step S6;

[0049] In the third case, both dmax0 and dmax1 are greater than or equal to U2, and dmax2 is less than U3. In this case, directly set tgt0 to ds0 + ds1, tgt1 is empty, tgt2 is the union of ds2 and ds3, and then jump to step S6;

[0050] In the fourth case, dmax0 is greater than or equal to U2, dmax1 is greater than or equal to U3 but less than U2, and dmax2 is less than U3. In this case, the set tgt0 is set to ds0, tgt1 is set to ds1, and tgt2 is set to the union of ds2 and ds3, and then jump to step S6;

[0051] In the fifth case, dmax0 is greater than or equal to U2, and dmax1 and dmax2 are greater than or equal to U3. In this case, set tgt0 is taken as ds0, tgt1 is taken as the combination of ds1 and ds2, tgt2 is taken as ds3, and then jump to step S6.

[0052] In order to facilitate understanding of the invention, the above invention contents are sorted out below and the following embodiments are provided to explain the technical solution of the present invention more clearly.

[0053] Example 1

[0054] The technical solutions of this embodiment include:

[0055] Step 1: Build an image feature library;

[0056] Image feature library: Companies generally use independently developed and established deep learning models (here referred to as M_model) to convert various images into vectors of consistent length, called image features, to facilitate image retrieval. The collection of all image features constitutes the image feature library (db). Note that each feature has a corresponding image, which may be in formats such as jpg, webp, or bmp.

[0057] The similarity between two images can be calculated using their respective image features. Since image features correspond to a high-dimensional mathematical vector, the distance between two high-dimensional mathematical vectors can be calculated using methods such as Euclidean distance and discrete cosine distance. This distance value can quantify the similarity between the two images.

[0058] Step 2: Image recall;

[0059] Suppose there is a retrieval image x. Now we need to find images that are highly similar to it from the image feature database db, which are called recall images. We can first use the following steps:

[0060] 1. Use M_model in step 1 to convert the search image x into features and normalize it to obtain xq.

[0061] 2. Use a retrieval algorithm to search for image features similar to xq from the image feature database db. Sort the search results by similarity, and you'll get the k most similar images and their vector distances to xq. Let these k vector distances be ds = [d(1), d(2), …, d(k)], which can be thought of as the similarity between the k images and xq. Images ranked higher in the ranking have higher similarity to xq.

[0062] Step 3: Calculate the first-order differential of the recalled image;

[0063] Define a recall function: y = f(x,n), where n = 1, 2, 3, …, k. For example, k = 100. The function value is the feature distance between image x and the nth image. Then f(x,1) represents the result most similar to image x, f(x,2) the second most similar, and so on. The corresponding similarity distance matrix: ds(x) = [f(x,1), f(x,2), …, f(x,k)], is essentially an array sorted by distance.

[0064] Assume d(i) = f(x,i+1)-f(x,i), then we can calculate:

[0065] d(1)=f(x,2)-f(x,1), d(2)=f(x,3)-f(x,2),..., d(k-1)=f(x,k)-f(x,k-1);

[0066] From the above results, we can see that d is actually the first-order differential of f, and a total of (k-1) differential values ​​are obtained. Among the k most similar images recalled, the size of the d(i) value reflects the change in image similarity.

[0067] The above differential values ​​are represented by a set d, which can be described as: d:{d(1),d(2),...,d(k-1)}.

[0068] Suppose the maximum value of the elements in d is dmax, and the corresponding image number is q, then: dmax = f(x,q+1)-f(x,q), which means that the distance between the image group to the left of q and the image group to the right of q is the largest.

[0069] Step 4: Segmentation processing of the recall image;

[0070] Using the differential calculation results of step 3, the recalled images can be clustered according to their similarity:

[0071] 1. Find the maximum value dmax0 of the elements in d. This value divides d into two parts, set as ds0 and ds_tmp0. ds0 = [d(1), d(2), ..., d(q)], ds_tmp0 = [d(q+1), d(q+2), ..., d(k-1)].

[0072] 2. Take the maximum value dmax1 of the elements in ds_tmp0, which corresponds to index q1. This value divides ds_tmp0 into two segments, ds1 and ds_tmp1. ds1 = [d(q+1), d(q+2), ..., d(q1)], ds_tmp1 = [d(q1+1), d(q1+2), ..., d(k-1)].

[0073] 3. Take the maximum value dmax2 of the elements in ds_tmp1, corresponding to index q2. This value divides ds_tmp1 into two segments, ds2 and ds3. ds2 = [d(q2+1), d(q2+2), ..., d(q2)], ds3 = [d(q2+1), d(q2+2), ..., d(k-1)].

[0074] These three steps divide the distance space ds(x) of image features into four subspaces, corresponding to ds0, ds1, ds2, and ds3, and obtain three segmentation metrics dmax0, dmax1, and dmax2. That is, dmax0, dmax1, and dmax2 divide the ds set into four parts, corresponding to ds0, ds1, ds2, and ds3.

[0075] Step 5: Recombining matching data based on threshold;

[0076] This step is to filter out the results that are definitely matched. Define three thresholds: U1 = 0.35 for confirming the matching graph in the case of ds smoothing, U2 = 0.1 for distinguishing the matching graph from other graphs, and U3 = 0.04 for the following situations:

[0077] The first case is when the values ​​of the elements in the ds set are small and smooth, indicating that the recalled images are similar. In this case, if dmax0 is less than or equal to U3, simply find the matching image set tgt0 for all values ​​in ds that are less than or equal to U1, set tgt1 and tgt2 to empty sets, and then skip to step 6.

[0078] The second case is when the values ​​of ds are divided into two very different subsets. That is, dmax0 is greater than or equal to U2, while dmax1 and dmax2 are both less than or equal to U3. Because ds0 is the most similar subset, simply put the elements of ds0 into set tgt0, set tgt1 as an empty set, and set tgt2 as the union of ds1, ds2, and ds3. Then skip to step 6.

[0079] The third case also shows polarization of the ds value, but it is relatively smooth. In this case, dmax0 and dmax1 are both greater than or equal to U2, while dmax2 is less than U3. In this case, simply set tgt0 to ds0 + ds1, tgt1 to be empty, tgt2 to be the union of ds2 and ds3, and then skip to step 6.

[0080] In the fourth case, dmax0 is greater than or equal to U2, dmax1 is greater than or equal to U3 but less than U2, and dmax2 is less than U3. In this case, set tgt0 to be ds0, tgt1 to be ds1, and tgt2 to be the union of ds2 and ds3. Then skip to step 6.

[0081] In the fifth case, dmax0 is greater than or equal to U2, and dmax1 and dmax2 are greater than or equal to U3. In this case, set tgt0 is ds0, tgt1 is the union of ds1 and ds2, and tgt2 is ds3. Then skip to step 6.

[0082] Step 6: Non-matching and possible matching filtering;

[0083] In the previous step, we obtained three sets: tgt0, tgt1, and tgt2. These three sets actually represent the matching results. Step 5 actually relies on a threshold to determine this relationship, which may still result in missed detections in some cases. Furthermore, the sorted ds in step 2 may actually place non-matching images before matching images, which can easily lead to missed detections. To address this missed detection, you can do the following:

[0084] Put the first 5 values ​​in tgt2 into tgt1. If there are less than 5 values ​​in tgt2, put all of them into tgt1.

[0085] Step 7: Accurate matching;

[0086] The images in tgt1 are matched using a more accurate algorithm, such as a corner matching algorithm such as SURF, to obtain a new target image set tgt1'.

[0087] Step 8: Merge the results;

[0088] Combine the results of tgt0 and tgt1 to get tgt_final, which is the final matching result.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements or improvements made within the technical scope of the present invention should be included in the scope of protection of the present invention.

Claims

1. A precise matching method for recalling images from a large-scale information database, characterized in that: The following steps are included: Step S1, constructing an image feature database db based on image features of existing images; Step S2: input a search image x, find multiple image features similar to the search image x from the image feature database db, and obtain k recall images; Step S3, perform first-order differential calculation on the similarity distance of the recalled images: Define a recall function y = f(x,n), where n = 1, 2, 3…, k, then there is a corresponding similarity distance matrix ds(x) = [f(x,1), f(x,2),…, f(x,k)]; Assume d(i) = f(x,i+1)-f(x,i), then we can calculate: d(1)=f(x,2)-f(x,1); d(2)=f(x,3)-f(x,2),...; d(k-1)=f(x,k)-f(x,k-1); Where d is the first-order differential of f, and then (k-1) differential values ​​are obtained. In the k recalled images, the size of the d(i) value is used to reflect the similarity of the images; According to the above differential values, we can get the set d:{d(1),d(2),...,d(k-1)}; Let the maximum value of the element in d be dmax, and the corresponding image number is q, then: dmax = f(x,q+1)-f(x,q), which means that the distance between the image group to the left of q and the image group to the right of q is the largest; Step S4, segment the recalled image: Using the differential calculation results obtained in step S3, the recalled images are clustered according to their similarity, and the distance space ds(x) of the image features is divided into four subspaces, corresponding to ds0, ds1, ds2 and ds3 respectively, and three segmentation metrics dmax0, dmax1 and dmax2 are obtained at the same time; Step S5, recombining the matching data based on the set threshold to obtain matching image sets tgt0, tgt1 and tgt2; Step S6: Filter non-matching and possible matching: Put the first 5 values ​​in set tgt2 into set tgt1. If there are less than 5 values ​​in set tgt2, put all of them into set tgt1. Step S7, accurate matching: Match the images in the set tgt1 using a preset exact matching algorithm to obtain a new target image set tgt1'; Step S8: Merge the results of the set tgt0 and the set tgt1' to obtain tgt_final, thereby obtaining the final matching result.

2. The accurate matching method for recalling images from a large-scale information database according to claim 1, characterized in that: In step S1, a preset deep learning model is used to convert the existing image into a vector with the same length, and the vector is used as the image feature.

3. The accurate matching method for recalling images from a large-scale information database as claimed in claim 2, characterized in that: In the image feature database db, the image format corresponding to each image feature is jpg, webp or bmp format.

4. The accurate matching method for recalling images from a large-scale information database according to claim 2, characterized in that: In step S2, the process of recalling images includes: Step S20, using the deep learning model in step S1 to convert the search image x into image features, and then perform normalization processing to obtain features xq; In step S21, a preset retrieval algorithm is used to search for image features similar to feature xq from the image feature library db, and the search results are sorted in order of similarity to obtain the k most similar images and the vector distance between each image and feature xq, wherein the k vector distances are set to ds = [d(1), d(2), ..., d(k)], and ds is used as the similarity between the k images and xq, and the images that are ranked higher have higher similarity with xq.

5. The accurate matching method for recalling images from a large-scale information database as claimed in claim 4, characterized in that: In step S4, the process of clustering the recalled images according to their similarity includes: Step S40, finding the maximum value dmax0 of the elements in the set d, the maximum value dmax0 divides d into two parts: ds0 and ds_tmp0, where ds0 = [d(1), d(2), ..., d(q)], ds_tmp0 = [d(q+1), d(q+2), ..., d(k-1)]; Step S41, taking the maximum value dmax1 of the elements in ds_tmp0, corresponding to sequence number q1, and dividing ds_tmp0 into two segments: ds1 and ds_tmp1, where ds1 = [d(q+1), d(q+2), ..., d(q1)] and ds_tmp1 = [d(q1+1), d(q1+2), ..., d(k-1)]; Step S42, taking the maximum value dmax2 of the elements in ds_tmp1, corresponding to sequence number q2, and dividing ds_tmp1 into two segments: ds2 and ds3, where ds2 = [d(q2+1), d(q2+2), ..., d(q2)] and ds3 = [d(q2+1), d(q2+2), ..., d(k-1)]; The set ds is divided into four parts using the maximum value dmax0, the maximum value dmax1 and the maximum value dmax2, and the four parts correspond to ds0, ds1, ds2 and ds3.

6. The accurate matching method for recalling images from a large-scale information database according to claim 5, characterized in that: In step S5, three thresholds U1, U2 and U3 are defined, where: U1 = 0.35, used to confirm the smoothness of ds and confirm the matching graph; U2 = 0.1, used to confirm the distinction between the matching graph and other graphs; U3=0.04, used in the following cases: The first case is that the values ​​of the elements in the ds set are small and smooth, indicating that the recalled images are similar to each other. In this case, if dmax0 is less than or equal to U3, all values ​​less than or equal to U1 in ds are taken to obtain the matching image set tgt0, tgt1 and tgt2 are taken as empty sets, and then jump to step S6; The second case is that the value of ds is divided into two parts: dmax0 is greater than or equal to U2, and dmax1 and dmax2 are both less than or equal to U3. Since ds0 is the most similar subset, the elements of the ds0 set are directly put into the set tgt0, the set tgt1 is set to be an empty set, the set tgt2 is set to be the union of ds1, ds2 and ds3, and then jump to step S6; In the third case, both dmax0 and dmax1 are greater than or equal to U2, and dmax2 is less than U3. In this case, directly set tgt0 to ds0 + ds1, tgt1 is empty, tgt2 is the union of ds2 and ds3, and then jump to step S6; In the fourth case, dmax0 is greater than or equal to U2, dmax1 is greater than or equal to U3 but less than U2, and dmax2 is less than U3. In this case, the set tgt0 is set to ds0, tgt1 is set to ds1, and tgt2 is set to the union of ds2 and ds3, and then jump to step S6; In the fifth case, dmax0 is greater than or equal to U2, and dmax1 and dmax2 are greater than or equal to U3. In this case, set tgt0 is taken as ds0, tgt1 is taken as the combination of ds1 and ds2, tgt2 is taken as ds3, and then jump to step S6.

Images