Image Retrieval Method and Medium

Abstract

The present invention discloses an image retrieval method and medium, including the following steps: obtaining a to-be-retrieved image, and performing feature extraction on the to-be-retrieved image to obtain a depth feature map corresponding to the to-be-retrieved image; performing weighted processing on each channel of the depth feature map to generate a weighted feature map; extracting local features and global features of the weighted feature map, and performing feature fusion based on the local features and global features to generate a fused feature; calculating a to-be-retrieved hash code corresponding to the fused feature based on a hash-guided metric loss method; calculating the similarity between the to-be-retrieved image and database images based on the to-be-retrieved hash code and the database image hash codes, and determining a target image according to the similarity. It can effectively prevent the loss of important semantic information during the binarization process, thereby improving the accuracy of image retrieval.

Description

Technical Field

[0001] This application relates to the technical field of image retrieval, and particularly relates to an image retrieval method and medium. Background Art

[0002] In the related art, in the field of supervised hashing image retrieval, the feature representation of an image and the quantization step are usually processed separately; this processing method mainly relies on first extracting the feature representation of the image, and then binarizing these features to generate hash codes. However, this separated processing method may cause some important semantic information to be lost during the generation of binary codes, thereby affecting the accuracy of retrieval. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems in the related art. To this end, an object of the present invention is to propose an image retrieval method that can effectively prevent the loss of important semantic information during the binarization process, thereby improving the accuracy of image retrieval.

[0004] In a first aspect, an embodiment of the present invention proposes an image retrieval method, including the following steps: obtaining a to-be-retrieved image, and performing feature extraction on the to-be-retrieved image to obtain a depth feature map corresponding to the to-be-retrieved image; performing weighted processing on each channel of the depth feature map to generate a weighted feature map; extracting local features and global features of the weighted feature map, and performing feature fusion based on the local features and global features to generate a fused feature; calculating a to-be-retrieved hash code corresponding to the fused feature based on a hash-guided metric loss method; calculating the similarity between the to-be-retrieved image and a database image hash code based on the to-be-retrieved hash code and the database image hash code, and determining a target image according to the similarity.

[0005] Further, performing weighted processing on each channel of the depth feature map to generate a weighted feature map includes: performing global average pooling on each channel of the depth feature map to generate a global spatial compression representation; calculating a weight vector based on the global spatial compression representation according to an activation function; performing an element-wise multiplication operation on the depth feature map according to the weight vector to generate a weighted feature map.

[0006] Further, the global spatial compression representation is calculated by the following formula:

[0007] ;

[0008] where, represents the global spatial compression representation, represents the height of the feature map, represents the width of the feature map, represents the spatial position in the feature map The eigenvalue of the th channel, that is, spatial compression of the feature map along the channel dimension;

[0009] ;

[0010] Among them, represents the activation function, represents the activation function, represents the element value of the feature map, which comes from the output feature map of the previous layer in the image feature extraction network and is used to enhance the non-linear expression ability of the feature;

[0011] ;

[0012] ;

[0013] ;

[0014] Among them, represents generating an intermediate representation containing global information. After this process, the feature map is compressed and transformed into information more suitable for downstream tasks; then, Z will enter the second fully connected layer to further generate the channel weight vector W, represents the activation function, and represents the bias vector, and represents the weight matrix, represents the weight vector, represents the weighted feature map, represents the component of the weight vector for each channel.

[0015] Furthermore, local features and global features of the weighted feature map are extracted, and feature fusion is performed based on the local features and global features to generate a fused feature, including: processing the weighted feature map using a convolutional layer to generate local features and global features; reshaping the local features into a first matrix and reshaping the global features into a second matrix; transposing the second matrix, performing matrix multiplication of the transposed second matrix and the first matrix, and using a softmax layer to calculate the spatial attention map; performing matrix multiplication of the spatial attention map and the second matrix to obtain a new feature map; combining the new feature map and the global features based on element-wise addition to generate a fused feature.

[0016] Furthermore, the spatial attention map is calculated by the following formula:

[0017] ;

[0018] Among them, represents the correlation between the th position in the local feature and the th position in the global feature, represents the number of pixels, represents the first matrix, represents the second matrix;

[0019] The fusion feature is calculated by the following formula:

[0020] ;

[0021] Among them, represents the fusion feature, represents the weight coefficient dynamically adjusted during the learning process. In the initial stage of the model, it is usually initialized to 0, but during the training process, it will be gradually updated according to the gradient to adjust the fusion method of the final feature map; represents the feature value at the th position in the local feature map, represents the feature value at the th position in the global feature map.

[0022] Furthermore, calculating the hash code to be retrieved corresponding to the fusion feature based on the hash-guided metric loss method includes: performing global average pooling operation on the fusion feature to obtain the corresponding global feature vector; inputting the global feature vector into a fully connected layer to map the global feature vector to a dimension consistent with the length of the hash code to obtain a mapped feature; and converting the mapped feature into a binary hash code based on the sign function.

[0023] Furthermore, the global feature vector is calculated by the following formula:

[0024] ;

[0025] Among them, represents the global feature vector, represents the height of the feature map, represents the width of the feature map, represents the fusion feature, and are the spatial indices in the height and width directions of the feature map respectively, is the channel index;

[0026] ;

[0027] Among them, represents the mapped feature, represents the weights of the fully connected layer, represents the bias parameter of the fully connected layer;

[0028] ;

[0029] where, represents the binary hash code.

[0030] Furthermore, in the process of generating the hash code, the hash-guided metric loss method is used to limit the learning range of the metric term. Specifically, it includes: calculating the metric loss and correcting the error component in the metric loss based on a preset threshold to alleviate the conflict between the metric loss and the quantization loss; combining the metric loss and the quantization loss to reduce the information loss in the binarization process; using the hash-guided metric loss function to optimize the model parameters.

[0031] In some embodiments, the metric loss is calculated by the following formula:

[0032] ;

[0033] where, represents the metric loss function, represents the set of proxy vectors, represents the input embedding vector, represents the positive proxy vector, represents the boundary value, represents the scaling factor;

[0034] The error component in the metric loss is corrected by the following formula:

[0035] ;

[0036] where, represents the improved hash-guided metric loss function, represents the set of all negative sample pairs, represents the hash code of the proxy anchor, represents the hash code of the current sample, represents the hash code of the proxy anchor of the negative sample, represents and the cosine similarity between, represents the margin hyperparameter, represents the preset threshold;

[0037] ;

[0038] where, represents the minimum distance of the best global solution, Indicates the length of the hash code;

[0039] In the process of combining the metric loss and the quantization loss, the learning objective is expressed by the following formula:

[0040] ;

[0041] ;

[0042] where, is set to 1, represents the quantization loss, represents the predicted value of the i-th sample, represents the true value of the i-th sample, represents the squared L2 norm of the predicted value and the target value, and its calculation formula is: , is the dimension of the hash code, and are respectively the th sample's predicted value and target value in the th dimension.

[0043] In a second aspect, an embodiment of the present invention proposes a computer-readable storage medium, on which an image retrieval program is stored. When the image retrieval program is executed by a processor, the above-mentioned image retrieval method is implemented.

[0044] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention.

[0045] According to the image retrieval method of an embodiment of the present invention, first, an image to be retrieved is obtained, and feature extraction is performed on the image to be retrieved to obtain a depth feature map corresponding to the image to be retrieved; then, each channel of the depth feature map is weighted to generate a weighted feature map; then, local features and global features of the weighted feature map are extracted, and feature fusion is performed based on the local features and global features to generate a fused feature; then, a hash code to be retrieved corresponding to the fused feature is calculated based on a hash-guided metric loss method; then, the similarity between the image to be retrieved and the database image hash code is calculated based on the hash code to be retrieved and the database image hash code, and a target image is determined according to the similarity. Thus, it effectively prevents the loss of important semantic information during the binarization process, and further improves the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic flowchart of the image retrieval method according to an embodiment of the present invention;

[0047] Figure 2Schematic diagram of the ResNet-50 model architecture according to an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of the channel weighting process according to an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the multi-scale context collaboration process according to an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the retrieval effect of the ImageNet dataset according to an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of the retrieval effect of the CIFAR-10 dataset according to an embodiment of the present invention. Detailed implementation manners

[0052] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0053] The image retrieval method according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0054] Please refer to Figure 1 , Figure 1 is a flowchart of the image retrieval method according to an embodiment of the present invention. As shown in Figure 1 , the image retrieval method includes the following steps:

[0055] S101, obtaining the image to be retrieved, and performing feature extraction on the image to be retrieved to obtain the depth feature map corresponding to the image to be retrieved.

[0056] As an example, as shown in Figure 2 , first, in terms of the feature extraction model architecture, a pre-trained ResNet-50 model is used as the backbone network. This model has been extensively trained on the ImageNet dataset and has strong generalization ability. Through residual learning in the deep network, the problem of gradient disappearance during the training of the deep neural network is reduced, ensuring the efficiency of model optimization and the robustness of features. In Figure 2 , represents the output feature map of the third residual block (Res3) of ResNet-50, where 1024 represents the number of feature channels and 14×14 represents the spatial dimension of the feature map. It represents the feature map after being processed by the Res4 block, which has 2048 channels and a spatial dimension of 7×7. These parameter designs enable the model to maintain sufficient feature expression ability at deeper network levels while controlling the computational complexity.

[0057] Next, a multi-branch structure is adopted, including a global branch and a local branch. After the third residual block (Res3) of ResNet-50, a dual-branch structure of local and global is introduced to process and refine the multi-dimensional features of the image. Among them, the global branch maintains the basic structure of ResNet-50, removes all pooling layers after the Res4 block, and retains the complete spatial information of the feature map, thus promoting the quality of global context features. In addition, in the global branch, the Feature Pyramid Pooling (FPP) technology is added to perform multi-scale pooling operations on the feature map, integrate the feature pooling results within different windows into a feature vector, ensure the diversity and scale invariance of global features, and assist the network in making more accurate global semantic predictions. In the output of the global branch, F g [512,7,7] represents the output feature map after feature pyramid pooling. The number of channels is reduced to 512, and the 7×7 spatial dimension is maintained. This dimensionality reduction design effectively reduces the number of model parameters while retaining key feature information.

[0058] The local branch is dedicated to capturing and enhancing fine-grained local features. The Atrous Spatial Pyramid Pooling (ASPP) module is adopted. This module contains multi-scale atrous convolutional layers and can handle the problem of target size changes in images. The parallel dilated convolutions of the ASPP module can capture spatial contexts of different scales at once, strengthening the model's understanding of local information. After fusing the feature maps with different dilation rates, convolutional processing yields more representative and discriminative local attributes. In the local branch, Concat represents the feature concatenation operation, which concatenates the feature maps generated by convolutional layers with different dilation rates in the ASPP module along the channel dimension to form a richer feature representation. SelfAtt represents the self-attention mechanism, which is used to calculate the correlation between positions within the feature map and enhance the model's attention to important local features. The final output of the local branch F l[512, 7, 7] has the same dimension as the global branch output, facilitating subsequent feature fusion. In addition, the local branch also includes a self-attention module (Self-Attention Module) to deeply explore the correlations of each local feature point, enhancing the model's understanding and discrimination ability of image details. The multi-scale context collaboration module fuses and collaborates on the features of the global and local branches. Finally, through a fully connected layer, a final descriptor with a dimension of [512×1] is generated. This descriptor has strong discrimination ability and robustness and can be used for efficient image retrieval tasks. This dual extraction and fusion strategy of global and local features enables the model to simultaneously focus on the overall semantics and detailed features of the image, thus achieving better performance in image retrieval tasks. Especially for images with complex backgrounds or subtle differences, this structural design shows obvious advantages.

[0059] S102, perform weighted processing on each channel of the depth feature map to generate a weighted feature map.

[0060] In some embodiments, performing weighted processing on each channel of the depth feature map to generate a weighted feature map includes: performing global average pooling on each channel of the depth feature map to generate a global spatial compression representation; calculating a weight vector based on an activation function according to the global spatial compression representation; performing an element-wise multiplication operation on the depth feature map according to the weight vector to generate a weighted feature map.

[0061] In some embodiments, the global spatial compression representation is calculated by the following formula:

[0062] ;

[0063] where, represents the global spatial compression representation, represents the height of the feature map, represents the width of the feature map, represents the spatial position in the feature map at the feature value of the

[0064] th channel, that is, performing spatial compression on the feature map along the channel dimension;

[0065] where, represents the activation function, represents the activation function, represents the element value of the feature map, which is from the output feature map of the previous layer in the image feature extraction network and is used to enhance the non-linear expression ability of the feature;

[0066] ;

[0067] ;

[0068] ;

[0069] Among them, means generating an intermediate representation containing global information. After this process, the feature map is compressed and transformed into information more suitable for downstream tasks; then, Z will enter the second fully connected layer to further generate the channel weight vector W. means activation function, and means bias vector, and means weight matrix, means weight vector, means weighted feature map, means the component for each channel in the weight vector.

[0070] As an example, as Figure 3 shown, channel weighting aims to dynamically adjust the weights of each channel in the feature map, thereby strengthening the channel features that contribute significantly to the image retrieval task and suppressing irrelevant or interfering features. First, taking the deep feature map as the input, global average pooling is performed on the feature map of each channel to generate a global spatial compression representation, and the global spatial compression representation is calculated by the following formula:

[0071] ;

[0072] Among them, means global spatial compression representation, means the height of the feature map, means the width of the feature map;

[0073] In terms of the weight generation mechanism, use and activation function to construct the weight generation mechanism, function, as a smooth non-linear activation function, can reduce the problem of gradient disappearance and optimize the training of deep networks. The function is expressed by the following formula:

[0074] ;

[0075] Then, perform two-layer fully connected operations on the generated global spatial compression representation and use activation function:

[0076] ;

[0077] ;

[0078] Among them, means generating an intermediate representation containing global information. After this process, the feature map is compressed and transformed into information more suitable for downstream tasks; then, Z will enter the second fully connected layer to further generate the channel weight vector W. means activation function, and means bias vector, and means weight matrix, means weight vector.

[0079] Then, perform an element-wise multiplication operation on the generated weight vector and the input depth feature map:

[0080] ;

[0081] Among them, means weighted feature map, means the component of the weight vector for each channel. It represents the weight on channel and is used to multiply element-wise with the corresponding channel features to highlight the features contributing to the image retrieval task. The dot product operation ensures that the global information in the weight vector can directly affect each element of the corresponding feature map, rather than just a single dimension. In this way, the channel weighting module not only amplifies or suppresses the features, but also enhances the semantic consistency and information reliability of the feature map.

[0082] Next, output: the weighted feature map. Finally, output the feature map after channel weighting, which has stronger representational power and decision-making accuracy, and will be used for subsequent feature fusion and retrieval processes.

[0083] Through the above steps, the channel weighting module can dynamically adjust the weights of each channel, highlight the features beneficial to the current image retrieval task, and thus enhance the overall representational ability of the network. This mechanism realizes the weighted adjustment of the feature map through the combination of global average pooling, Swish, and Sigmoid activation functions, optimizing the extraction of image features and subsequent retrieval effects.

[0084] S103, extract the local and global features of the weighted feature map, and perform feature fusion based on the local and global features to generate fused features.

[0085] In some embodiments, local features and global features of a weighted feature map are extracted, and feature fusion is performed based on the local features and the global features to generate a fused feature, including: processing the weighted feature map using a convolutional layer to generate local features and global features; reshaping the local features into a first matrix and reshaping the global features into a second matrix; transposing the second matrix, performing matrix multiplication on the transposed second matrix and the first matrix, and using a softmax layer to calculate a spatial attention map; performing matrix multiplication on the spatial attention map and the second matrix to obtain a new feature map; and combining the new feature map and the global features based on element-wise addition to generate a fused feature.

[0086] In some embodiments, the spatial attention map is calculated by the following formula:

[0087] ;

[0088] where, represents the correlation between the -th position in the local features and the -th position in the global features, represents the number of pixels, represents the first matrix, represents the second matrix;

[0089] The fused feature is calculated by the following formula:

[0090] ;

[0091] where, represents the fused feature, represents a weight coefficient that is dynamically adjusted during the learning process. In the initial stage of the model, it is usually initialized to 0, but during the training process, it will be gradually updated according to the gradient to adjust the fusion method of the final feature map. represents the feature value at the -th position in the local feature map, represents the feature value at the -th position in the global feature map.

[0092] It should be noted that in deep feature learning, the multi-scale fusion of image features is crucial for capturing rich semantic information. However, in the process of feature fusion, dealing with the semantic differences between features has always been a challenge that cannot be ignored. Therefore, in order to effectively narrow the semantic gap between feature maps of different scales, an innovative Multi-scale Contextual Collaboration Module (MSCCM) is proposed, which calculates the correlation between pixels in different feature maps through matrix multiplication. Subsequently, it uses this correlation as the weight vector of the high-level feature map. The greater the similarity between the feature representations of two pixels, the higher their correlation.

[0093] As an example, as Figure 4 shown, first, the input feature map is processed using a convolutional layer, aiming to perform channel compression to reduce the computational load while generating feature map A (local feature) and B (global feature). Then, the local feature map A and the global feature map B are respectively reshaped into matrices L and G, where N = H × W represents the number of pixels. L and G respectively represent the reshaped local and global features.

[0094] Then, in order to measure the spatial attention, G is transposed and matrix-multiplied with L, and a softmax layer is applied to calculate the spatial attention map P. The spatial relationship matrix map P maps the spatial correlation between pixels:

[0095] ;

[0096] where, represents the correlation between the th position in the local feature and the th position in the global feature, represents the number of pixels, represents the first matrix, represents the second matrix;

[0097] Next, matrix multiplication is performed by multiplying G and the spatial attention map P to obtain a new feature map T. Each vector in T contains the integrated information of the correlation between the features at each position in the entire image and the current position.

[0098] Then, B and T are combined through element-wise addition to obtain the final output:

[0099] ;

[0100] where, represents the fused feature, It represents the weight coefficient that is dynamically adjusted during the learning process. In the initial stage of the model, it is usually initialized to 0, but during the training process, it will gradually adjust the fusion method of the final feature map according to the gradient update. It represents the feature value at the th position in the local feature map, and represents the feature value at the th position in the global feature map.

[0101] It should be noted that α starts as 0 during initialization. However, it will gradually adjust the weight distribution as learning progresses.

[0102] Each position in the final feature is the weighted sum of the features at all positions in the global feature. Since the final feature is generated from the top-level feature, high-level semantic information is well preserved in the final output, thus greatly enhancing the representation ability of the deep network descriptor. For the image retrieval task in complex scenarios, it will undoubtedly have important practical value.

[0103] S104. Calculate the hash code to be retrieved corresponding to the fusion feature based on the hash-guided metric loss method.

[0104] In some embodiments, calculating the hash code to be retrieved corresponding to the fusion feature based on the hash-guided metric loss method includes: performing a global average pooling operation on the fusion feature to obtain the corresponding global feature vector; inputting the global feature vector into a fully connected layer to map the global feature vector to a dimension consistent with the length of the hash code to obtain a mapped feature; and converting the mapped feature into a binary hash code based on the sign function.

[0105] In some embodiments, the global feature vector is calculated by the following formula:

[0106] ;

[0107] where, represents the global feature vector, represents the height of the feature map, represents the width of the feature map, represents the fusion feature, and are the spatial indices in the height and width directions of the feature map respectively, is the channel index;

[0108] I ;

[0109] where, represents the mapped feature, represents the weight of the fully connected layer, represents the bias parameter of the fully connected layer;

[0110] ;

[0111] in, Represents a binary hash code.

[0112] In some embodiments, in the process of generating hash codes, a hash-guided metric loss method is used to limit the learning range of the metric items, specifically, including: calculating the metric loss and correcting the error component in the metric loss based on a preset threshold to alleviate the conflict between the metric loss and the quantization loss; combining the metric loss and the quantization loss to reduce the information loss in the binarization process; and optimizing the model parameters using the hash-guided metric loss function.

[0113] In some embodiments, the metric loss is calculated by the following formula:

[0114] ;

[0115] in, represents the metric loss function, represents the set of proxy vectors, represents the embedding vector representing the input, represents the positive proxy vector, Indicates the boundary value, represents the scaling factor;

[0116] The error component in the metric loss is corrected by the following formula:

[0117] ;

[0118] in, represents the improved hash-guided metric loss function, represents the set of all negative sample pairs, A hash code representing the proxy anchor point. Represents the hash code of the current sample, The proxy anchor hash code representing the negative sample, express and The cosine similarity between represents the margin hyperparameter, Indicates the preset threshold;

[0119] ;

[0120] in, represents the minimum distance to the best global solution, Indicates the length of the hash code;

[0121] In the process of set metric loss and quantization loss, the learning objective is expressed by the following formula:

[0122] ;

[0123] ;

[0124] where, is set to 1, represents the quantization loss, represents the predicted value of the i-th sample, represents the true value of the i-th sample, represents the squared L2 norm of the predicted value and the target value, and its calculation formula is: , is the dimension of the hash code, and are respectively the predicted value and the target value of the -th sample in the -th dimension.

[0125] It should be noted that in the image retrieval task, the design of the metric loss function is crucial. The traditional Proxy-Anchor method uses the cosine similarity estimated in a small batch of data to design the metric loss term. Therefore, the Hash-Guided Metric Loss (HGM-Loss) is proposed to improve the linear calculation part of the cosine similarity in the proxy-anchor metric loss, so as to limit the scope of metric term learning and thus improve the accuracy and robustness of feature extraction.

[0126] As an example, the proxy-anchor calculates the metric loss through the following formula:

[0127] ;

[0128] where, represents the metric loss function, represents the proxy vector set, represents the input embedding vector, represents the positive proxy vector. Each class has a proxy vector, which is continuously updated during training to maximize the distance between different classes while minimizing the distance within the same class, represents the boundary value, represents the scaling factor.

[0129] It should be noted that is the main metric loss function, which is used to ensure that the hash code H of the sample and its corresponding proxy anchor H pIt has a good similarity relationship in the hash space. Specifically, it achieves this goal by maximizing the similarity between similar samples and minimizing the similarity between different samples.

[0130] Next, a preset threshold is introduced , by introducing a specific threshold to correct the part of the calculation error component in the proxy anchor metric loss to alleviate the conflict between the metric loss and the quantization loss. The improved hash-guided metric loss function is expressed by the following formula:

[0131] ;

[0132] where, represents the improved hash-guided metric loss function, represents the set of all negative sample pairs, represents the hash code of the proxy anchor, represents the hash code of the current sample, represents the hash code of the proxy anchor of the negative sample, represents and the cosine similarity between, represents the margin hyperparameter, represents the preset threshold;

[0133] For the first term of the formula, calculate the similarity between positive sample pairs and control the influence of the similarity through , if the similarity is lower than , it does not contribute to the loss; otherwise, the loss increases according to the similarity.

[0134] The second term, calculate the similarity between negative sample pairs and control the influence of the similarity through , contrary to the first term, here it is hoped that the similarity of negative sample pairs is as low as possible, so a negative sign is introduced.

[0135] It should be noted that the setting of the preset threshold depends on the length of the binary code and the number of categories, and it represents the minimum distance of the optimal global solution , the purpose of the HGM loss is to push the distance between different categories to the appropriate threshold determined based on the minimum distance , the preset threshold is calculated by the following formula:

[0136] ;

[0137] where, ​​Represents the length of the hash code;

[0138] In the process of combining the metric loss and the quantization loss, the learning objective is expressed by the following formula:

[0139] ;

[0140] ;

[0141] Where, Is set to 1, Represents the quantization loss, Represents the predicted value of the i-th sample, Represents the true value of the i-th sample.

[0142] Use the hash-guided metric loss function to optimize the model parameters, including the following steps:

[0143] 1) Initialize parameters: At the beginning of training, all parameters of the network (including proxy vectors and other model weights) are randomly initialized.

[0144] 2) Forward propagation: The input image passes through the feature extraction module of the network to generate a feature representation, and then is binarized by the hash coding module.

[0145] 3) Use the hash-guided metric loss function And the quantization loss function To calculate the total loss.

[0146] 4) Backward propagation: Update the model parameters by calculating the gradient of the loss function, so that the representation ability of the feature extraction and hash coding modules is gradually optimized.

[0147] 5) Parameter update: Use an optimization algorithm (SGD) to update the model parameters, and iterate the training until the model converges.

[0148] S105, Calculate the similarity between the image to be retrieved and the database images based on the hash code of the image to be retrieved and the hash codes of the database images, and determine the target image according to the similarity.

[0149] As an example, this step is mainly completed by comparing the hash codes of the image to be retrieved and the images in the database and calculating the similarity between the two. Specifically, it includes: First, input the feature encoding: Input the image to be retrieved into the trained deep network, extract its feature representation, and generate the corresponding binary hash code through the hash coding module. Let the hash code of the image to be retrieved be . Each image in the database has also extracted features and generated hash codes through the same network. Let the hash code of the i-th image in the database be . Next, calculate the similarity of the hash codes. Use the Hamming distance to calculate the hash code of the image to be retrieved and the hash code of each image in the database to calculate the similarity. The Hamming distance is defined as the number of different positions between two binary codes of the same length, that is:

[0150] ;

[0151] where, is the length of the hash code; and respectively represent and at the bit binary value.

[0152] Then, perform similarity sorting: Calculate the Hamming distance between the image to be retrieved and all images in the database, and arrange these distances in ascending order. The smaller the distance, the higher the similarity. The sorting result obtains an image list sorted in descending order of similarity, that is, arranged in the order from the most similar to the least similar to the image to be retrieved.

[0153] Finally, return the retrieval result: According to the sorting result, select the top N most similar images as the retrieval result and return them to the user. These images are the images in the database with the highest similarity to the image to be retrieved.

[0154] In addition, to better illustrate the retrieval effect of the image retrieval method proposed by the present invention, take Figure 5 and Figure 6 as examples. Figure 5 is the retrieval effect of the image retrieval method proposed by the embodiment of the present invention on the ImageNet dataset; while Figure 6 is the retrieval effect of the image retrieval method proposed by the embodiment of the present invention on the CIFAR-10 dataset. In Figure 5 and Figure 6 , the left column Query is the input image, and the right five columns are the top 5 most similar images retrieved.

[0155] In summary, according to the image retrieval method of the embodiments of the present invention, first, a to-be-retrieved image is obtained, and feature extraction is performed on the to-be-retrieved image to obtain a depth feature map corresponding to the to-be-retrieved image; then, each channel of the depth feature map is weighted to generate a weighted feature map; then, local features and global features of the weighted feature map are extracted, and feature fusion is performed based on the local features and global features to generate a fusion feature; then, a to-be-retrieved hash code corresponding to the fusion feature is calculated based on a hash-guided metric loss method; then, the similarity between the to-be-retrieved image and the database image is calculated based on the to-be-retrieved hash code and the database image hash code, and a target image is determined according to the similarity. Thus, it is effectively prevented from losing important semantic information during the binarization process, and further the accuracy of image retrieval is improved.

[0156] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, on which an image retrieval program is stored. When the image retrieval program is executed by a processor, the above-mentioned image retrieval method is implemented.

[0157] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus or device and execute the instructions), or used in combination with these instruction execution systems, apparatus or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport a program for use by or in combination with an instruction execution system, apparatus or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting or otherwise processing it as appropriate, and then storing it in a computer memory.

[0158] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0159] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0160] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0161] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0162] In the present invention, unless otherwise clearly specified or limited, the terms "installed", "connected", "coupled", "fixed", etc. shall be construed broadly. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal connection of two components or the interaction relationship between two components, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0163] In the present invention, unless otherwise clearly specified or limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may mean that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "beneath" and "underneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0164] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image retrieval method, characterized in that: The following steps are involved: Acquire an image to be retrieved, and perform feature extraction on the image to be retrieved to obtain a depth feature map corresponding to the image to be retrieved; Performing weighted processing on each channel of the depth feature map to generate a weighted feature map; Extracting local features and global features of the weighted feature map, and performing feature fusion based on the local features and the global features to generate fused features; Calculating the to-be-retrieved hash code corresponding to the fusion feature based on a hash-guided metric loss method; Calculating the similarity between the image to be retrieved and the database image based on the hash code to be retrieved and the hash code of the database image, and determining the target image according to the similarity; The weighted processing is performed on each channel of the depth feature map to generate a weighted feature map, including: Performing global average pooling on each channel of the depth feature map to generate a global spatial compressed representation; Calculating a weight vector according to the global spatial compressed representation based on an activation function; Performing a principal element multiplication operation on the depth feature map according to the weight vector to generate a weighted feature map; The global space compression representation is calculated using the following formula: ； in, represents the global space compressed representation, represents the height of the feature map, represents the width of the feature map, Represents the spatial position in the feature map Place The characteristic value of each channel; ； in, express activation function, express activation function, Represents the element value of the feature map; ； ； ； in, It represents the generation of an intermediate representation containing global information. After this process, the feature map is compressed and converted into information that is more suitable for downstream tasks. Then, Z enters the second fully connected layer to further generate the channel weight vector W. express activation function, and represents the bias vector, and represents the weight matrix, represents the weight vector, represents the weighted feature map, Represents the component in the weight vector for each channel.

2. The image retrieval method according to claim 1, wherein: Extracting local features and global features from the weighted feature map, and performing feature fusion based on the local features and the global features to generate fused features, including: Processing the weighted feature map using a convolutional layer to generate local features and global features; Reshaping the local features into a first matrix, and reshaping the global features into a second matrix; Transposing the second matrix, performing matrix multiplication on the transposed second matrix and the first matrix, and calculating a spatial attention map using a softmax layer; Performing matrix multiplication on the spatial attention map and the second matrix to obtain a new feature map; The new feature map and the global feature are combined based on element-by-element addition to generate a fused feature.

3. The image retrieval method according to claim 2, wherein: The spatial attention map is calculated by the following formula: ； in, Represents the local feature Position and global feature The correlation between the positions, Indicates the number of pixels, represents the first matrix, represents the second matrix; The fusion feature is calculated by the following formula: ； in, represents the fusion feature, Represents the weight coefficient that is dynamically adjusted during the learning process; Represents the local feature map The eigenvalues at the positions, Indicates the first The eigenvalues at each position.

4. The image retrieval method according to claim 1, wherein: Calculating the to-be-retrieved hash code corresponding to the fusion feature using a hash-guided metric loss method includes: Performing a global average pooling operation on the fused features to obtain a corresponding global feature vector; Inputting the global feature vector into a fully connected layer to map the global feature vector to a dimension consistent with the hash code length to obtain a mapping feature; The mapping feature is converted into a binary hash code based on a sign function.

5. The image retrieval method according to claim 4, wherein: The global eigenvector is calculated by the following formula: ； in, represents the global eigenvector, represents the height of the feature map, represents the width of the feature map, represents the fusion feature, and are the spatial indexes of the feature map in height and width directions, is the channel index; ； in, represents the mapping feature, represents the weight of the fully connected layer, Represents the bias parameters of the fully connected layer; ； in, Represents a binary hash code.

6. The image retrieval method according to claim 1, wherein: In the process of generating the hash code, a hash-guided metric loss method is used to limit the learning range of the metric item, specifically including: Calculate the metric loss and correct the error component in the metric loss based on a preset threshold to alleviate the conflict between the metric loss and the quantization loss; Combining metric loss and quantization loss to reduce information loss in the binarization process; Model parameters are optimized using a hash-guided metric loss function.

7. The image retrieval method according to claim 6, wherein: The metric loss is calculated by the following formula: ； in, represents the metric loss function, represents the set of proxy vectors, represents the embedding vector of the input, represents the positive proxy vector, Indicates the boundary value, represents the scaling factor; The error component in the metric loss is corrected by the following formula: ； in, represents the improved hash-guided metric loss function, represents the set of all negative sample pairs, A hash code representing the proxy anchor point. Represents the hash code of the current sample, The proxy anchor hash code representing the negative sample, express and The cosine similarity between represents the margin hyperparameter, Indicates the preset threshold; ； in, represents the minimum distance to the best global solution, Indicates the length of the hash code; In the process of combining metric loss and quantization loss, the learning objective is expressed by the following formula: ； ； in, Set to 1, represents the quantization loss, represents the predicted value of the i-th sample, represents the true value of the i-th sample, It represents the square L2 norm of the predicted value and the target value, and its calculation formula is: , is the dimension of the hash code, and Respectively The sample in The predicted and target values of the dimension.

8. A computer-readable storage medium, characterized in that An image retrieval program is stored thereon, and when the image retrieval program is executed by a processor, the image retrieval method according to any one of claims 1 to 7 is implemented.

Images