An image retrieval method based on multi-direction global features

Abstract

The application discloses an image retrieval method based on multi-direction global features. First, the integrated deep feature map of an image is extracted by using a deep residual network (ResNet) model and an iterative multi-layer fusion model. Second, the spatial and frequency clues of the integrated deep feature map are captured by using a Gabor filter to obtain a multi-direction feature map. Finally, a matching feature vector is obtained by using an L2 normalization and whitening method. The final retrieval result is obtained by using the cosine similarity between the matching feature vector of a current image to be retrieved and the matching feature vector of each sample image in a retrieval sample database, so that the image retrieval is realized. The method integrates deep feature maps of different levels to obtain an integrated deep feature map with distinguishing characteristics. The integrated deep feature map learned is combined with direction clues by simulating the direction selection mechanism of a human being, so that an image representation with higher discrimination is obtained, and the image retrieval performance is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, and specifically to an image retrieval method based on multi-directional global features. Background Technology

[0002] With the rapid development of mobile internet technology and the widespread adoption of smart mobile devices, image data has experienced explosive growth. Against this backdrop, efficient image retrieval algorithms have become a hot research topic. In recent years, deep learning technology has greatly promoted the development of the image retrieval field. In particular, the extraction of deep image features using convolutional neural network models and attention models has achieved significant progress in computer vision tasks. In early image retrieval, pre-trained deep learning models were typically used to extract image feature maps. These pre-trained models were usually generalized on large-scale image datasets. To achieve better results in specific image retrieval tasks, fine-tuning the model is a common strategy. Fine-tuning involves making minor parameter adjustments to the pre-trained model, retaining the general knowledge learned by the pre-trained model while adapting to new datasets, thus improving the model's generalization ability. Research shows that deep features extracted at different levels of the network model have significantly different image description capabilities; however, current network models do not fully leverage the complementary advantages of features at different levels, and compatibility issues exist between features at different levels. Against this backdrop, how to integrate features at different levels and effectively process these integrated features for image similarity retrieval is a current challenge and difficulty in image retrieval research. Summary of the Invention

[0003] The present invention aims to address the problem that current network models do not leverage the complementary advantages of features at different levels, and that there are compatibility issues between features at different levels. It provides an image retrieval method based on multi-directional global features, which can effectively integrate multi-level features and capture information from different directions.

[0004] To solve the above problems, the present invention is achieved through the following technical solution:

[0005] An image retrieval method based on multi-directional global features includes the following steps:

[0006] Step 1: Feed each sample image in the retrieved sample database into the pre-trained integrated deep feature extraction model to extract the integrated deep feature map of each sample image.

[0007] Step 1.1: Input each sample image into the pre-trained deep residual network model that integrates deep feature extraction models to extract the deep feature map and shallow feature map of each sample image;

[0008] Step 1.2: Input the deep feature map and shallow depth map of each sample image into the iterative multi-layer fusion model of the pre-trained integrated deep feature extraction model to obtain the integrated deep feature map of each sample image.

[0009] Step 2: Perform Gabor filtering on the integrated depth feature map of each sample image in the retrieved sample database to obtain the filtered integrated depth feature map of each sample image.

[0010] Step 3: Extract the maximum orientation cue, average orientation cue, and Gabor energy of the filtered integrated deep feature map of each sample image in the retrieval sample database;

[0011] Step 4: Input the Gabor energy of each sample image in the retrieved sample database into the Sigmoid function to obtain the harmony factor;

[0012] Step 5: Integrate the maximum directional cue and the average directional cue based on the harmony factor to obtain the harmonious Gabor response of each sample image in the retrieval sample database.

[0013] Step 6: Pool the harmonic Gabor response of each sample image in the retrieved sample database to obtain the multi-directional feature vector of each sample image.

[0014] Step 7: Perform L2 normalization, whitening, and L2 normalization again on the multi-directional feature vector of each sample image in the retrieved sample database to obtain the matching feature vector of each sample image.

[0015] Step 8: Perform the same processing as steps 1 to 7 on the current image to be retrieved to obtain the matching feature vector of the current image to be retrieved;

[0016] Step 9: Calculate the cosine similarity between the matching feature vector of the current image to be retrieved and the matching feature vector of each sample image in the retrieval sample database, and return the final retrieval results in descending order of cosine similarity.

[0017] The specific process of step 1.2 above is as follows:

[0018] Step 1) Perform layer normalization on the deep feature map U and shallow feature map V of each sample image to obtain the layer-normalized deep feature map. Shallow feature maps with normalized layers

[0019] Step 2) Normalize the deep feature map Convolution processing is performed to obtain the deep feature map D; simultaneously, the shallow feature map is normalized. Different convolution processes are performed to obtain the first shallow depth feature map S1 and the second shallow depth feature map S2.

[0020] Step 3) For the deep feature map D of the t-th iteration (t) Similarity calculation is performed between the first shallow depth feature map S1 and the multi-layer similarity map of the t-th iteration. Where D (0) =D;

[0021] Step 4) For the multi-level similarity graph of the t-th iteration After normalization, the normalized multi-layer similarity graph I of the t-th iteration is obtained. (t) ;

[0022] Step 5) Compare the second shallow depth feature map S2 with the normalized multi-layer similarity map I of the t-th iteration. (t) Perform weighted aggregation to obtain the aggregated depth feature map G in the t-th iteration. (t) ;

[0023] Step 6) Generate the deep feature map D of the t-th iteration. (t) The aggregated deep feature map G in the t-th iteration (t) Adding them together, we obtain the deep feature map D of the (t+1)th iteration. (t+1) ;

[0024] Step 7) Repeat steps 3) to 6) a predetermined number of times to obtain the final depth feature map.

[0025] Step 8) Generate the final iterative depth feature map Convolution processing is performed to obtain the reshaped depth feature map.

[0026] Step 9) Reconstruct the depth feature map Adding the shallow feature map V to the integrated deep feature map O for each sample image yields the integrated deep feature map O.

[0027] In step 5 above, the harmonious Gabor response R harm (x,y) is:

[0028]

[0029] In the formula, R represents the harmony factor. max (x,y) represents the maximum directional cue, R avg (x,y) represents the average directional cues, and ⊙ represents the Hadamard product.

[0030] Compared with the prior art, the present invention has the following features:

[0031] 1. Based on the visual application of image depth features, an integrated deep feature extraction model based on the deep residual network (ResNet) model and the iterative multilayer fusion model is proposed, which can obtain more discriminative integrated deep feature maps and improve retrieval accuracy.

[0032] 2. By using Gabor filtering to capture directional cues from the integrated deep feature map, and by simulating the human direction selection mechanism, the captured directional cues are integrated to obtain an image representation that balances the maximum and average directional responses. This effectively amplifies the most important directional cues in each learned integrated deep feature map, thereby improving the image's discriminability and further enhancing retrieval accuracy. Attached Figure Description

[0033] Figure 1 This is a flowchart of an image retrieval method based on multi-directional global features.

[0034] Figure 2 It is a flowchart of iterative multi-layer feature map integration. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0036] An image retrieval method based on multi-directional global features, such as Figure 1 As shown, the specific steps are as follows:

[0037] Step 1: Feed each sample image from the retrieved sample database into the pre-trained integrated deep feature extraction model to extract the integrated deep feature map of each sample image.

[0038] The integrated deep feature extraction model proposed in this invention consists of a deep residual network (ResNet) model, an iterative multi-layer fusion model, a generalized average pooling layer, and an L2 normalization layer. This invention pre-trains the integrated deep feature extraction model using a training dataset to obtain a pre-trained integrated deep feature extraction model. The training process is as follows:

[0039] Step 1) Input each training image into the ResNet model and extract the deep feature map of each training image. and shallow feature map Where C, H, and W represent the number of channels, height, and width, respectively.

[0040] Residual Networks (ResNet) are a type of deep convolutional neural network that addresses the vanishing gradient problem in deep networks by introducing residual connections. In ResNet, each convolutional layer does not directly map the input to the output; instead, it adds the input to the output, achieved through "shortcut" connections that span several layers.

[0041] Step 2) Perform layer normalization on the deep feature map U and the shallow feature map V to obtain the layer-normalized deep feature map. Shallow feature maps with normalized layers

[0042] Step 3) Normalize the deep feature map Perform 1×1 convolution to obtain the deep feature map D, and simultaneously normalize the shallow feature map. Different 1×1 convolution processes are performed to obtain the first shallow depth feature map S1 and the second shallow depth feature map S2, so as to map the deep feature map and the shallow feature map to a new feature space and reduce the number of model parameters.

[0043]

[0044] In the formula, Conv(·) represents the convolution operation; φ1, φ2 and φ3 represent learnable parameters. D represents the deep feature map, S1 represents the first shallow feature map, and S2 represents the second shallow feature map. Thus, the spatial dimensions of D, S1, and S2 are reshaped as follows: Three depth feature maps were obtained.

[0045] Step 4) For the deep feature map D of the t-th iteration (t) Similarity calculation is performed between the first shallow depth feature map S1 and the multi-layer similarity map of the t-th iteration.

[0046]

[0047] In the formula, t represents the iteration number, t∈0,1,2,3,…,D (0) =D, T This indicates transpose, and C represents the number of channels.

[0048] Step 5) For the multi-level similarity graph of the t-th iteration After normalization, the normalized multi-layer similarity graph I of the t-th iteration is obtained. t ).

[0049]

[0050] Step 6) Compare the second shallow depth feature map S2 with the normalized multi-layer similarity map I of the t-th iteration. (t) Perform weighted aggregation to obtain the aggregated depth feature map G(t-th iteration). t ).

[0051] G (t) =I (t) S2 T

[0052] Step 7) Generate the deep feature map D of the t-th iteration. (t) The aggregated deep feature map G in the t-th iteration (t) Adding them together, we obtain the deep feature map D of the (t+1)th iteration. (t+1) .

[0053]

[0054] Step 8) After repeating steps 4) to 7) (the number of iterations is preset), the spatial dimensions of the deep feature map are reshaped. The final iteration of the deep feature map is obtained.

[0055] Step 9) Generate the final iterative depth feature map Perform 1×1 convolution to match the number of channels to V in the shallow feature map, resulting in a reconstructed deep feature map.

[0056]

[0057] In the formula, φ4 represents the learnable parameter.

[0058] Step 10) Reconstruct the depth feature map Adding the shallow feature map V to the integrated deep feature map ensures that the model output integrated deep feature map is in the same feature space as the input deep feature map, resulting in the integrated deep feature map O.

[0059]

[0060] In the formula,

[0061] Step 11) Perform generalized average pooling on the integrated deep feature map O of each training image to obtain the integrated feature vector f of each training image.

[0062]

[0063] In the formula, p represents the learnable parameter of generalized average pooling, initially p = 3. H and W represent the height and width, respectively.

[0064] Step 12) Perform L2 normalization on the integrated feature vector f of the training image to obtain the normalized integrated feature vector f′.

[0065] Step 13) Calculate the contrastive loss for the L2-normalized integrated feature vector f′, and train the integrated deep feature extraction model based on the contrastive loss until the model converges, obtaining the pre-trained integrated deep feature extraction model. The loss function is as follows:

[0066]

[0067] In the formula, f′ α and f′ β These are the normalized integrated feature vectors of training images α and β, respectively. Y(i,j) is the matching label of training images α and β. If α and β match, then Y(i,j) = 1; otherwise, Y(i,j) = 0. The threshold m = 0.85. Max(·) represents the maximum value function.

[0068] This invention utilizes the pre-trained integrated deep feature extraction model obtained above to extract the integrated deep feature map of each sample image in the retrieval sample database. The specific process is as follows:

[0069] Step 1.1: Input each sample image into the pre-trained deep residual network model of the integrated deep feature extraction model to extract the deep feature map U and the shallow feature map V of each sample image.

[0070] Step 1.2: Input the deep feature map U and shallow feature map V of each sample image into the iterative multi-layer fusion model of the pre-trained integrated deep feature extraction model to obtain the integrated deep feature map O of each sample image.

[0071] Step 1) Perform layer normalization on the deep feature map U and the shallow feature map V to obtain the layer-normalized deep feature map. Shallow feature maps with normalized layers

[0072] Step 2) Normalize the deep feature map Convolution processing is performed to obtain the deep feature map D; simultaneously, the shallow feature map is normalized. Different convolution processes are performed to obtain the first shallow depth feature map S1 and the second shallow depth feature map S2.

[0073] Step 3) For the deep feature map D of the t-th iteration (t) Similarity calculation is performed between the first shallow depth feature map S1 and the multi-layer similarity map of the t-th iteration. Where D (0)=D;

[0074] Step 4) For the multi-level similarity graph of the t-th iteration After normalization, the normalized multi-layer similarity graph I of the t-th iteration is obtained. (t) ;

[0075] Step 5) Compare the second shallow depth feature map S2 with the normalized multi-layer similarity map I of the t-th iteration. (t) Perform weighted aggregation to obtain the aggregated depth feature map G in the t-th iteration. (t) ;

[0076] Step 6) Generate the deep feature map D of the t-th iteration. (t) The aggregated deep feature map G in the t-th iteration (t) Adding them together, we obtain the deep feature map D of the (t+1)th iteration. (t+1) ;

[0077] Step 7) Repeat steps 3) to 6) a predetermined number of times to obtain the final depth feature map.

[0078] Step 8) Generate the final iterative depth feature map Convolution processing is performed to obtain the reshaped depth feature map.

[0079] Step 9) Reconstruct the depth feature map Adding it to the shallow feature map V yields the integrated deep feature map O.

[0080] Step 2: Perform Gabor filtering on the integrated depth feature map of each sample image in the retrieved sample database to capture the integrated depth feature map in the direction θ. j Using directional clues, the filtered integrated deep feature map R(x,y,θ) of each sample image is obtained. j ).

[0081]

[0082] In the formula, x'=xcosθ j +ysinθ j y′=-xsinθ j +ycosθ j x = 1, 2, 3, ..., W represents the width of the feature map, y = 1, 2, 3, ..., H represents the height of the feature map; θ j This represents the j-th direction, where j∈{1,2,3,4}. γ is the aspect ratio. σ is the standard deviation of the Gaussian factor, σ = 2π; It's a phase shift. λ is the wavelength, and λ = 3. This represents the convolution operation.

[0083] Step 3: Extract the maximum orientation cue, average orientation cue, and Gabor energy of the filtered integrated deep feature map of each sample image in the retrieval sample database.

[0084] Maximum directional clue R max (x,y) is:

[0085] R max (x,y)=Max{R(x,y,θ j )}

[0086] Average Directional Clue R avg (x,y) is:

[0087] R avg (x,y)=Sum{R(x,y,θ j )} / N θ

[0088] Gabor Energy R eng (x,y) is:

[0089]

[0090] In the formula, Sum(·) represents the summation function, and θ j =(j-1)π / N θ N θ N represents the total number of directions. θ =4.

[0091] Step 4: Calculate the Gabor energy R for each sample image in the retrieved sample database. eng (x, y) is fed into the Sigmoid function, which maps the Gabor energy to between 0 and 1, yielding the harmony factor.

[0092]

[0093] Step 5: For each sample image in the retrieved sample database, based on the harmony factor... For the maximum directional cue R max (x,y) and average directional cues R avg Integrating (x,y) yields the harmonious Gabor response R. harm (x,y).

[0094] Harmonious Gabor Response R harm (x,y) is:

[0095]

[0096] In the formula, ⊙ represents the Hadamard product.

[0097] Step 6: Calculate the harmonious Gabor response R for each sample image in the retrieved sample database. harm Pooling is performed on (x,y) to obtain the multi-directional feature vector F for each sample image.

[0098]

[0099] In the formula, p represents the pooling parameter for regulating feature map aggregation, and p = 3.

[0100] Step 7: Perform L2 normalization, whitening, and L2 normalization again on the multi-directional feature vectors of each sample image in the retrieved sample database to obtain the matching feature vector of each sample image.

[0101] Step 8: Perform the same processing as steps 1 to 7 on the current image to be retrieved to obtain the matching feature vector of the current image to be retrieved.

[0102] Step 9: Calculate the cosine similarity between the matching feature vector of the current image to be retrieved and the matching feature vector of each sample image in the retrieval sample database, and return the final retrieval results in descending order of cosine similarity.

[0103] This invention proposes an image retrieval method based on multi-directional global features. First, it extracts integrated deep feature maps from images using a deep residual network (ResNet) model and an iterative multi-layer fusion model. Second, it captures spatial and frequency cues from the integrated deep feature maps using Gabor filtering to obtain multi-directional feature vectors. Finally, it obtains matching feature vectors using L2 normalization, whitening, and further L2 normalization. The final retrieval result is obtained by calculating the cosine similarity between the matching feature vector of the current image to be retrieved and the matching feature vector of each sample image in the retrieval sample database, thus achieving image retrieval. This method integrates feature maps of different depths to obtain discriminative integrated deep feature maps. By simulating the human orientation selection mechanism, it combines the learned integrated deep feature maps with directional cues to obtain more discriminative image representations, significantly improving image retrieval performance.

[0104] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A method for image retrieval based on multi-directional global features, characterized in that, The steps include the following: Step 1: Feed each sample image in the retrieved sample database into the pre-trained integrated deep feature extraction model to extract the integrated deep feature map of each sample image. Step 1.1: Input each sample image into the pre-trained deep residual network model that integrates deep feature extraction models to extract the deep feature map and shallow feature map of each sample image; Step 1.2: Input the deep feature map and shallow feature map of each sample image into the iterative multi-layer fusion model of the pre-trained integrated deep feature extraction model to obtain the integrated deep feature map of each sample image. Step 1) performing layer normalization processing on deep feature maps and shallow feature maps of each sample image to obtain layer-normalized deep feature maps and layer-normalized shallow feature maps ;​​​ Step 2) Normalize the deep feature map Convolution processing is performed to obtain deep feature maps. Simultaneously, the shallow feature maps of the layer are normalized. Different convolution processes are performed to obtain the first shallow layer depth feature map. Second shallow depth feature map ; Step 3) For the first Iterative deep feature maps Compared with the first shallow depth feature map Perform similarity calculation to obtain the first... Iterative multi-level similarity graph ,in ; Step 4) For the first Iterative multi-level similarity graph Perform normalization to obtain the normalized first... Iterative multi-level similarity graph ; Step 5) Process the second shallow depth feature map With normalized first Iterative multi-level similarity graph Perform weighted aggregation to obtain the first... Iterative aggregation of deep feature maps ; Step 6) The first Iterative deep feature maps With the Iterative aggregation of deep feature maps Add them together to get the first one. Iterative deep feature maps ; Step 7) Repeat steps 3) to 6) a predetermined number of times to obtain the final depth feature map. ; Step 8) Generate the final iterative depth feature map Convolution processing is performed to obtain the reshaped depth feature map. ; Step 9) Reshape the depth feature map With shallow feature map By adding them together, we obtain the integrated deep feature map for each sample image. ; Step 2: Perform Gabor filtering on the integrated depth feature map of each sample image in the retrieved sample database to obtain the filtered integrated depth feature map of each sample image. Step 3: Extract the maximum orientation cue, average orientation cue, and Gabor energy of the filtered integrated deep feature map of each sample image in the retrieval sample database; Step 4: Input the Gabor energy of each sample image in the retrieved sample database into the Sigmoid function to obtain the harmony factor; Step 5: Integrate the maximum directional cue and the average directional cue based on the harmony factor to obtain the harmonious Gabor response of each sample image in the retrieval sample database. Harmonious Gabor Response for: In the formula, Represents the harmony factor. Indicates the maximum directional clue. Indicates average directional cues. It represents the Hadamardi (or Hadama) stack; Step 6: Pool the harmonic Gabor response of each sample image in the retrieved sample database to obtain the multi-directional feature vector of each sample image. Step 7: Perform L2 normalization, whitening, and L2 normalization again on the multi-directional feature vector of each sample image in the retrieved sample database to obtain the matching feature vector of each sample image. Step 8: Perform the same processing as steps 1 to 7 on the current image to be retrieved to obtain the matching feature vector of the current image to be retrieved; Step 9: Calculate the cosine similarity between the matching feature vector of the current image to be retrieved and the matching feature vector of each sample image in the retrieval sample database, and return the final retrieval results in descending order of cosine similarity.

Images