Image search precision optimization method and system based on combination of rough arrangement and fine arrangement

Abstract

The invention relates to the technical field of image processing, in particular to an image search precision optimization method and system based on the combination of rough arrangement and fine arrangement, and the method comprises the steps: a rough arrangement model stage: calculating the rough arrangement similarity of a query image and all images in a database, and sorting and screening out a preset number of candidate images according to the rough arrangement similarity; a model fine arrangement stage: performing multi-scale reconstruction on the query image to generate reconstructed images with different resolutions; extracting feature vectors of the reconstructed images through a fine arrangement model, performing expansion operation on the feature vectors, and calculating fine arrangement similarity between the query image and the candidate image; and a similarity weighting combination stage: performing normalization processing on the rough ranking similarity and the fine ranking similarity, dynamically adjusting the weight coefficients of the rough ranking similarity and the fine ranking similarity according to the difference of the rough ranking similarity, generating a final similarity through weighted average, and finally sequencing and outputting a retrieval result. And by dynamically combining the distance of rough arrangement and the distance of fine arrangement, the image retrieval precision is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically to an image search accuracy optimization method and system based on the combination of rough ranking and fine ranking. Background Art

[0002] In traditional image retrieval systems, the rough ranking model and the fine ranking model are often used as two independent stages. The rough ranking model is used to quickly screen candidate images, and the fine ranking model is used to more accurately rank these candidate images. However, in the prior art, the similarity calculation of the rough ranking and the fine ranking models is usually independent, lacking an effective combination of the similarities of the two, and unable to flexibly adjust the weights between the two, resulting in inaccurate image retrieval results or complex calculation processes in some cases.

[0003] In addition, existing fine ranking models often process images with a single scale, making it difficult to fully utilize the multi-scale feature information of images, which limits the accuracy of retrieval results. In order to improve the effect of image retrieval, how to fully combine multi-scale features and reasonably adjust the similarity weights between the rough ranking and the fine ranking models has become an urgent technical problem to be solved. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide an image search accuracy optimization method and system based on the combination of rough ranking and fine ranking, introducing multi-scale information, and further improving the accuracy of image retrieval by dynamically combining the distances of rough ranking and fine ranking.

[0005] To achieve the above object, the present invention provides the following technical solution: An image search accuracy optimization method based on the combination of rough ranking and fine ranking, the steps of which are as follows: (1) Rough ranking model stage: Calculate the rough ranking similarity between the query image and all images in the database, sort and screen a preset number (such as 1000 - 5000 images) of candidate images according to the rough ranking similarity. The rough ranking model uses a lightweight network (such as MobileNet) to quickly extract global features and perform preliminary ranking through cosine similarity or Euclidean distance; (2) Fine ranking model stage: Perform multi-scale reconstruction on the query image to generate reconstructed images with different resolutions; Extract the feature vectors of each reconstructed image through the fine ranking model, and perform an expansion operation on the feature vectors to calculate the fine ranking similarity between the query image and the candidate images; Multi-scale reconstruction: Reconstruct the query image N times with different scaling factors (such as 0.5×, 1×, 2×) to generate multi-resolution images.

[0006] Feature expansion: Through convolution kernel operations (such as 3×3), the feature dimension is converted from 1, wi, hi, c1, wi, hi, c to C×k×k, hi×wiC×k×k, hi×wi to extract local detailed features.

[0007] Similarity calculation: Perform matrix multiplication on the expanded query features and candidate features, filter the maximum similarity value (threshold ≥ 0.55), and generate a refined ranking similarity score after normalization; (3) Similarity weighted combination stage: Normalize the rough ranking similarity and the refined ranking similarity, dynamically adjust the weight coefficients of the two according to the difference in the rough ranking similarity, generate the final similarity through weighted average, and sort the candidate images according to the final similarity to output the retrieval result; Normalize the rough ranking and refined ranking similarities, and dynamically adjust the weights according to the similarity difference between the top two candidates in the rough ranking: If the difference < 0.1, assign the refined ranking weight a = 0.9 and the rough ranking weight b = 0.1; Otherwise, a = 0.5 and b = 0.5.

[0008] Generate the final similarity after weighting, and sort and output the retrieval result.

[0009] In some of the embodiments, the specific manner of the multi-scale reconstruction is: Reconstruct the query image according to N different scaling factors to obtain N reconstructed images with different resolutions, where N is an integer greater than 1.

[0010] In some of the embodiments, the feature vector expansion operation includes: Performing convolution operations on the feature vectors of the query image and the candidate image respectively, Converting the dimension of the original feature matrix of the query image from [1, wi, hi, c] to [C*k*k, hi*wi], where C is the number of feature channels, k is the convolution kernel size, and wi and hi are the width and height of any reconstructed query image i respectively; Converting the dimension of the original feature matrix of the candidate image from [1, wj, hj c] to [C*k*k, hj*wj], where C is the number of feature channels, k is the convolution kernel size, and hj and wj represent the height and width of any candidate image j respectively.

[0011] In some of the embodiments, the calculation of the refined ranking similarity includes: Performing matrix multiplication operations on the expanded query image features and candidate image features to generate a similarity matrix; Extracting the maximum similarity value for each pair of images, and filtering valid matches through a threshold, and summarizing and normalizing to obtain the refined ranking similarity scores at each reconstructed resolution.

[0012] In some of these embodiments, the condition for dynamically adjusting the weight coefficient is as follows: if the similarity difference between the top two candidate images in the rough ranking is less than a preset threshold, a higher weight is assigned to the fine ranking similarity; otherwise, the weights of the fine ranking and the rough ranking similarities are equal.

[0013] In some of these embodiments, the preset threshold is 0.1, and the adjustment range of the weight coefficient is as follows: the fine ranking weight a ∈ [0.5, 0.9], and the rough ranking weight b = 1 - a.

[0014] To achieve the above object, the present invention also provides the following technical solution: an image search accuracy optimization system based on the combination of rough ranking and fine ranking, which is obtained according to the image search accuracy optimization method described above. The image search accuracy optimization system includes: A rough ranking module, which is used to calculate the rough ranking similarity and screen candidate images; A fine ranking module, which is used for multi-scale reconstruction of the query image, feature expansion, and calculation of the fine ranking similarity; A weighted fusion module, which is used to dynamically adjust the weight coefficients of the rough ranking and the fine ranking similarities and generate the final retrieval result.

[0015] In some of these embodiments, the fine ranking module includes a multi-scale reconstruction unit, a feature expansion unit, and a similarity calculation unit, wherein the feature expansion unit realizes the dimension conversion of the feature matrix through convolution kernel operations.

[0016] In some of these embodiments, the weighted fusion module adaptively adjusts the weight according to the difference in the rough ranking similarity, and outputs the final similarity through normalization and weighted averaging.

[0017] To achieve the above object, the present invention provides the following technical solution: a computer-readable storage medium storing a computer program, and when the program is executed, the image search accuracy optimization method described above is realized.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: by weighted averaging the fine ranking similarity and the rough ranking similarity, the accuracy and efficiency of image retrieval are improved. It is used to improve the accuracy of image search in the textile industry, especially in the retrieval of model images, fabric images, and pattern images; Multi-scale feature fusion: by reconstructing at different resolutions, the global and local features of the image are fully captured; Efficient feature expansion: convolution operations reduce the computational complexity and improve the efficiency of similarity matching; Dynamic weight mechanism: adaptively adjusts the weight according to the rough ranking result, taking into account both accuracy and robustness; Wide applicability: can be extended to fields such as video retrieval and object recognition.

[0019] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable, and the present application will be described in detail and understood through the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is the system architecture diagram of the present invention; Figure 2 is the refined ranking module diagram of the present invention; Figure 3 is the flowchart of feature expansion and similarity calculation of the present invention; Figure 4 is the logic diagram of dynamic weight adjustment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] Please refer to Figures 1-3 , the present invention provides a technical solution: an image search accuracy optimization method based on the combination of rough ranking and refined ranking, and its steps are as follows: (1) Rough ranking model stage: Calculate the rough ranking similarity between the query image and all images in the database, and sort and screen out a preset number of candidate images according to the rough ranking similarity; (2) Refined ranking model stage: Perform multi-scale reconstruction on the query image to generate reconstructed images with different resolutions; Extract the feature vectors of each reconstructed image through the refined ranking model, and perform expansion operations on the feature vectors to calculate the refined ranking similarity between the query image and the candidate images; (3) Similarity weighted combination stage: Normalize the rough ranking similarity and the refined ranking similarity, dynamically adjust the weight coefficients of the two according to the difference in the rough ranking similarity, generate the final similarity through weighted average, and sort the candidate images according to the final similarity to output the retrieval result.

[0023] The specific method of the multi-scale reconstruction is: Reconstruct the query image according to N different scaling factors to obtain N reconstructed images with different resolutions, where N is an integer greater than 1.

[0024] The feature vector expansion operation includes: performing convolution operations on the feature vectors of the query image and the candidate images respectively, Convert the dimension of the original feature matrix of the query image from [1, wi, hi, c] to [C*k*k, hi*wi], where C is the number of feature channels, k is the convolution kernel size, and wi and hi are the width and height of any reconstructed query image i respectively; Convert the dimension of the original feature matrix of the candidate image from [1, wj, hj c] to [C*k*k, hj*wj], where C is the number of feature channels, k is the convolution kernel size, and hj and wj represent the height and width of any candidate image j respectively.

[0025] The calculation of the fine-ranking similarity includes: performing matrix multiplication on the expanded query image features and candidate image features to generate a similarity matrix; extracting the maximum similarity value for each pair of images, and screening valid matches through a threshold, and summarizing and normalizing to obtain the fine-ranking similarity scores at each reconstructed resolution.

[0026] The condition for dynamically adjusting the weight coefficient is: if the similarity difference between the top two candidate images in the rough-ranking similarity is less than the preset threshold, then assign a higher weight to the fine-ranking similarity; otherwise, the weights of the fine-ranking and rough-ranking similarities are equal.

[0027] The preset threshold is 0.1, and the adjustment range of the weight coefficient is: the fine-ranking weight a ∈ [0.5, 0.9], and the rough-ranking weight b = 1 - a.

[0028] An image search accuracy optimization system based on the combination of rough-ranking and fine-ranking, which is obtained according to the described image search accuracy optimization method. The described image search accuracy optimization system includes: A rough-ranking module for calculating the rough-ranking similarity and screening candidate images; A fine-ranking module for multi-scale reconstruction of the query image, feature expansion, and fine-ranking similarity calculation; A weighted fusion module for dynamically adjusting the weight coefficients of the rough-ranking and fine-ranking similarities and generating the final retrieval result.

[0029] The described fine-ranking module includes a multi-scale reconstruction unit, a feature expansion unit, and a similarity calculation unit, where the feature expansion unit realizes the dimension conversion of the feature matrix through convolution kernel operations.

[0030] The described weighted fusion module adaptively adjusts the weight according to the difference in the rough-ranking similarity, and outputs the final similarity through normalization and weighted averaging.

[0031] A computer-readable storage medium stores a computer program, and when the program is executed, it implements the described image search accuracy optimization method.

[0032] Through the technical solution of this application, its specific implementation process is: 1. Coarse Ranking Model Stage The task of the coarse ranking stage is to screen out several candidate images that are most similar to the query image by calculating the distances between the feature vectors of the query image and each image in the database. The specific implementation is as follows: 1) Obtain the feature vector of the query image, and use the coarse ranking model to calculate the feature distances between the query image and all images in the database, that is, the coarse ranking similarity.

[0033] 2) Sort according to the distances and select the top 2000 images as candidate images.

[0034] 2. Fine Ranking Model Stage The fine ranking stage performs further processing on the candidate images based on the coarse ranking. The query image is reconstructed in this stage and passes through the fine ranking model to obtain multiple fine ranking feature vectors, and the fine ranking similarity with the candidate images in the database is calculated. Among them, the reconstruction and the calculation of the fine ranking similarity are part of the innovations of the present invention.

[0035] 1) Reconstruct candidate images First, the query image is reconstructed according to N different scaling factors to obtain N reconstructed images with different resolutions. The purpose of this step is to capture the multi-scale feature information of the image through images with different resolutions. All reconstructed images will undergo unified preprocessing to adjust the pixel value range of the images and convert them into a format acceptable to the model for subsequent feature extraction and similarity calculation.

[0036] 2) Process each reconstructed image through the fine ranking model to calculate N feature vectors of the reconstructed image. The dimension of each feature vector is [1, wi, hi, c], where wi and hi are the width and height of the feature map under the reconstructed image with different resolutions. C is the number of feature channels of the fine ranking model.

[0037] 3) Obtain the fine ranking similarity To calculate the fine ranking similarity between the query image and the database image, we adopt a method based on the feature unfold of the fine ranking model to obtain the fine ranking features, and generate the feature representation after unfolding through convolution operations. A series of similarity calculations are performed based on this feature. The specific implementation process is as follows: ① Feature unfold operation First, for each input image and its corresponding feature vector, we "unfold" its features through a convolution operation. This operation processes the input features based on a convolutional kernel (K), converting the original feature matrix from its original shape [1, wi, hi, c] to an unfolded query with a feature dimension of [C*k*k, hi*wi], where C is the number of feature channels, k is the size of the convolutional kernel, and wi and hi are the width and height of any reconstructed query image i, respectively. It can be understood that there are C*k*k small image features.

[0038] Similarly, the refined ranking feature vectors in the database are also unfolded in a similar manner. For the candidate image pairs in the database, the shape of the unfolded feature vector target is [hj*wj, C*k*k], where hj and wj represent the height and width of any candidate image j, respectively, C is the number of feature channels, and k is the size of the convolutional kernel.

[0039] ② Calculate similarity After the feature unfolding, we calculate the similarity between the unfolded feature vector query of the query image and multiple unfolded feature vectors target of the database images. By performing a matrix multiplication operation on the unfolded features, a similarity matrix is obtained. At this time, through the torch.matmul operation in python, the vectors are multiplied to calculate their relationship (or similarity), forming a relationship matrix.

[0040] ③ Maximum similarity screening For each pair of candidate images and query images, we extract the maximum similarity value from the similarity matrix and perform screening based on a set threshold. Specifically, first, the torch.max function is used to find the maximum similarity value of hj*wj in target corresponding to each query image hi*wi. Then, we set all similarity values less than 0.55 to 0, and the remaining similarity values to 1. This step ensures that only those image pairs with relatively high similarities are considered valid matches. The threshold can be adaptively adjusted based on statistical analysis to optimize the matching results. In the experiment, we used 0.55 as the default threshold, but this method can be applied to different settings in the range of 0.4 - 0.7.

[0041] ④ Similarity score aggregation After filtering out the valid matching pairs, we sum up the similarities between each reconstructed query image and all candidate images, and normalize them to obtain the final similarity score. This similarity score represents the degree of matching between the query image at a single reconstructed resolution and the candidate images in the database. Specifically, by summing up the similarity values of all valid matches and dividing by the number of matches, we obtain the comprehensive similarity between the query image at a single reconstructed resolution and the images in the database, i.e., similarity.

[0042] ⑤ Generate a similarity list Based on N different scaling factors for reconstruction, repeat the above steps to obtain N similarities, i.e., similarity.

[0043] 3 Combine the fine-ranking and coarse-ranking similarities The innovation of the present invention also lies in flexibly weighted-averaging the fine-ranking similarity and the coarse-ranking similarity to improve the retrieval accuracy. According to different scenarios, use weight coefficients a and b to adjust the contributions of the fine-ranking and the coarse-ranking, i.e.: Final similarity = a * fine-ranking similarity + b * coarse-ranking similarity The specific implementation process is as follows: 1) Perform normalization processing Summarize the N similarities, i.e., similarity, obtained in the above steps into a similarity list, and use the normalization method to process it to ensure that all similarity values are within a unified range. The normalized similarity values help to eliminate the differences between different scales, enabling fair comparison of similarities at each scale.

[0044] 2) Weighted calculation of similarity To determine how to adjust the weighted ratio between the coarse-ranking similarity and the fine-ranking similarity, the present invention introduces a conditional judgment based on the difference in the coarse-ranking similarity. Specifically, for the coarse-ranking similarities arranged from high to low, if the difference between the first two similarities, i.e., the coarse-ranking similarities of the two most similar images, is less than 0.1, it is considered that the contribution of the fine-ranking similarity to the final result should be greater, and a higher weight is given to the fine-ranking similarity, setting a = 0.9 and b = 0.1; otherwise, a lower weight is given to the fine-ranking similarity, setting the weights a = 0.5 and b = 0.5. That is, the final similarity = a * fine-ranking similarity + b * coarse-ranking similarity.

[0045] 3) Combine, sort, and return the final result Sort the final similarity values obtained by combining the weighted fine-ranking similarity and the coarse-ranking similarity in descending order. Thus, an ordered list of image IDs and similarity values is obtained. Finally, the system returns based on these results to help the user obtain the most relevant images.

[0046] Embodiment and data verification When retrieving 2,000 test images using a dataset of 500,000 images, for model image retrieval: Coarse ranking stage: Input the model image, extract features through MobileNet, and screen the top 2,000 candidate images.

[0047] Fine ranking stage: Reconstruct the query image by scaling it 3 times (0.5×, 1×, 2×), extract features, and expand them into a C×9,hi×wiC×9,hi×wi dimension.

[0048] Similarity calculation: Screen the maximum similarity value (threshold 0.55), and generate similarity scores for each scale after normalization.

[0049] Weight adjustment: If the difference between the top two in the coarse ranking is <0.1, a = 0.9, b = 0.1; otherwise, evenly distribute the weights.

[0050] Result output: After weighted ranking, the Top1 accuracy rate is improved from 20.93% to 81.3%.

[0051] The retrieval results of the test images for model images and fabric images are shown in Table 1 as follows: Table 1 Test Image Retrieval Data Table

[0052] The actual website business application data is as follows. It is called about 600,000 times per day on average, the manual review workload is reduced, and the efficiency of designers in selecting styles is improved.

[0053] Original image transformation types: Shrinking (resolution reduced to 30%), cropping the local area (retaining 40% of the area) Color matching change (HSV hue offset ±15%), continuous printing (repeating pattern splicing)

[0054] The above embodiments only represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several variations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

[0055] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirits of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image search accuracy optimization method based on the combination of rough ranking and fine ranking, characterized in that: The steps are as follows: (1) Coarse ranking model stage: Calculate the coarse ranking similarity between the query image and all images in the database, and screen out a preset number of candidate images according to the coarse ranking similarity for sorting. (2) Fine ranking model stage: Perform multi-scale reconstruction on the query image to generate reconstructed images with different resolutions; extract the feature vectors of each reconstructed image through the fine ranking model, and perform an unfolding operation on the feature vectors to calculate the fine ranking similarity between the query image and the candidate images. (3) Similarity weighted combination stage: Normalize the coarse ranking similarity and the fine ranking similarity, dynamically adjust the weight coefficients of the two according to the difference in the coarse ranking similarity, generate the final similarity through weighted average, and sort the candidate images according to the final similarity to output the retrieval result.

2. The method and system for optimizing the image search accuracy based on the combination of rough ranking and fine ranking according to claim 1, characterized in that: The specific method of the multi-scale reconstruction is: Reconstruct the query image according to N different scaling factors to obtain N reconstructed images with different resolutions, where N is an integer greater than 1.

3. The method and system for optimizing the image search accuracy based on the combination of rough ranking and fine ranking according to claim 2, characterized in that: The unfolding operation of the feature vector includes: performing convolution operations on the feature vectors of the query image and the candidate images respectively. Convert the dimension of the original feature matrix of the query image from [1, wi, hi, c] to [C*k*k, hi*wi], where C is the number of feature channels, k is the size of the convolution kernel, and wi and hi are the width and height of any reconstructed query image i respectively. Convert the dimension of the original feature matrix of the candidate image from [1, wj, hj c] to [C*k*k, hj*wj], where C is the number of feature channels, k is the size of the convolution kernel, and hj and wj represent the height and width of any candidate image j respectively.

4. The method and system for optimizing the image search accuracy based on the combination of rough ranking and fine ranking according to claim 3, wherein: The calculation of the fine ranking similarity includes: performing matrix multiplication on the unfolded query image features and candidate image features to generate a similarity matrix; extracting the maximum similarity value of each pair of images, and screening out valid matches through a threshold, and summarizing and normalizing to obtain the fine ranking similarity scores at each reconstructed resolution.

5. The method and system for optimizing the accuracy of image search based on the combination of rough ranking and fine ranking according to claim 4, characterized in that: The condition for dynamically adjusting the weight coefficient is: If the similarity difference between the top two candidate images in the coarse ranking similarity is less than the preset threshold, then assign a higher weight to the fine ranking similarity; otherwise, the weights of the fine ranking and the coarse ranking similarity are equal.

6. The method and system for optimizing the image search accuracy based on the combination of rough ranking and fine ranking according to claim 5, characterized in that: The preset threshold is 0.1, and the adjustment range of the weight coefficient is: the fine ranking weight a ∈ [0.5, 0.9], and the coarse ranking weight b = 1 - a.

7. An image search accuracy optimization system based on the combination of rough ranking and fine ranking, characterized in that: An image search accuracy optimization system is obtained according to the image search accuracy optimization method described in any one of claims 1-6. The image search accuracy optimization system includes: A coarse ranking module for calculating the coarse ranking similarity and screening candidate images. A fine ranking module for multi-scale reconstruction of the query image, feature unfolding, and calculation of the fine ranking similarity. A weighted fusion module for dynamically adjusting the weight coefficients of the coarse ranking and the fine ranking similarity to generate the final retrieval result.

8. The image search accuracy optimization system based on the combination of rough ranking and fine ranking according to claim 7, wherein: The fine ranking module includes a multi-scale reconstruction unit, a feature unfolding unit, and a similarity calculation unit, where the feature unfolding unit realizes the dimension conversion of the feature matrix through convolution kernel operations.

9. The method and system for optimizing the image search accuracy based on the combination of rough ranking and fine ranking according to claim 7 or 8, characterized in that: The weighted fusion module adaptively adjusts the weight according to the difference in the coarse ranking similarity, and outputs the final similarity through normalization and weighted average.

10. A computer-readable storage medium storing a computer program, characterized in that, When the described program is executed, it implements the method for optimizing image search accuracy as described in any one of claims 1-6.

Images