An image retrieval method based on key local information

Abstract

The embodiment of the application discloses a kind of based on key local information's image retrieval method, device, system, it is related to image processing technical field, can realize using key local content retrieval to obtain the complete image of its main body, to improve retrieval accuracy.The application includes: original image library image processing;Query image feature extraction;Similarity retrieval process.Specifically: adjust original image size, extract image library original image convolution feature, cluster convolution feature map, generate key content area according to clustering feature map, carry out feature extraction to original image and key content area and obtain feature vector, adjust query image size, extract the feature vector of query image, calculate the similarity of query image and original image library image and its key content area, according to result sorting and return corresponding original image.The application is applicable to the image retrieval task that query image does not have integrity but contains enough key information to retrieve original image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image retrieval method based on key local information. Background Technology

[0002] Image information processing and retrieval are crucial for information reasoning and decision-making. However, in practical applications, incomplete image information and incomplete retrieval content are common problems. Although these issues can usually be somewhat alleviated with advancements in reconnaissance, image acquisition, and imaging technologies, a system capable of addressing incomplete image information is particularly critical when limited information is available.

[0003] Most existing image retrieval technologies extract global features from the entire image to obtain feature vectors that represent image information, and then perform similarity retrieval. This limits the completeness of the query image for users. When the input query image is only a small part of the target image, and the corresponding region is not found in the image library, it often fails to retrieve the original target. A few image retrieval technologies that address this problem employ relatively simple image region segmentation methods, making it difficult to focus on important feature regions; or they use supervised training of neural network models to segment key feature regions. However, this approach requires manual annotation of the dataset and introduces human factors, making the segmentation of key content regions subjective. Furthermore, because such models are trained according to certain specific categories, the trained network is difficult to use for cross-category region segmentation. This makes it difficult to achieve image retrieval tasks where the query image is incomplete but contains enough key information to retrieve the original image. Summary of the Invention

[0004] The embodiments of the present invention provide an image retrieval method based on key local information, which can retrieve the complete image of the subject by using key local content, thereby improving retrieval accuracy.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, the method provided by embodiments of the present invention includes:

[0007] S1. Preprocess the original image to obtain the key content features of the corresponding key content regions;

[0008] S2. Receive the image to be queried uploaded by the user and extract the image features of the image to be queried;

[0009] S3. Using the global features and key content features of each original image in the data library, perform similarity retrieval on the image to be queried.

[0010] In conjunction with the first aspect, in a first possible implementation of the first aspect, in S1, the preprocessing includes:

[0011] S11. Extract the convolutional features from the original image and form a convolutional feature map;

[0012] S12. Cluster the obtained convolutional feature maps to obtain clustered feature maps;

[0013] S13. Generate key content regions based on the clustering feature map;

[0014] S14. Perform feature extraction on the original image and the key content region respectively to obtain the feature vectors of the original image and the key content region respectively.

[0015] In conjunction with the first possible implementation of the first aspect, in the second possible implementation, in S11, the extraction of convolutional features from the original image and the formation of a convolutional feature map includes:

[0016] Adjust the size of the shortest side of the original image to a preset value;

[0017] Feature maps are extracted using a deep residual network, wherein a feature correction module is incorporated into the deep residual network;

[0018] The attention weights of a given intermediate feature map are obtained through the feature correction module. The obtained attention weights are multiplied by the given intermediate feature map to obtain a feature map with attention. The given intermediate feature map is output by the residual module in the deep residual network.

[0019] In conjunction with the second possible implementation of the first aspect, in the third possible implementation, the feature correction module includes: a spatial self-attention module M. self Channel Attention Module M channel Spatial attention module M spatial ;

[0020]

[0021]

[0022]

[0023] Where F represents the given intermediate feature map, F' represents the spatial self-attention correction feature, F” represents the channel attention correction feature, F”' represents the spatial attention correction feature, and the operator This indicates element-wise multiplication. The step of obtaining attention weights from a given intermediate feature map via the feature correction module includes:

[0024] In the spatial self-attention module, the given intermediate feature map is reduced in dimensionality by a 1×1 convolution operation to obtain a 1×H×W feature tensor. Then, the feature values ​​are transformed to the [0,1] interval by the Sigmoid activation function, and the spatial self-attention weights M are obtained by the SoftMax function. self Spatial self-attention modified features are obtained by element-wise multiplying spatial self-attention weights and input features, where H represents the height of the image and W represents the width of the image.

[0025] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation, the step of obtaining attention weights from a given intermediate feature map through the feature correction module further includes:

[0026] The spatial self-attention correction features are processed by the channel attention module using spatial max pooling and average pooling respectively, resulting in two feature tensors with dimensions of C×1×1, where C represents the number of channels.

[0027] The two obtained feature tensors of dimension C×1×1 are then passed through C shared 1×1 networks to map them into two new feature tensors of dimension C×1×1. The two new feature tensors are then summed element-wise and passed through the SoftMax activation function to output a channel attention weight M of dimension C×1×1. channel ;

[0028] The channel attention correction features are obtained by multiplying the channel attention weights and the spatial self-attention correction features of the input element by element.

[0029] In conjunction with the third or fourth possible implementation of the first aspect, in the fifth possible implementation, the step of obtaining attention weights from the given intermediate feature map through the feature correction module further includes: processing the channel attention correction features by the spatial attention module according to channel max pooling and average pooling respectively to obtain two feature tensors with dimensions of 1×H×W, and then merging the two feature tensors with dimensions of 1×H×W to form a feature tensor with dimensions of 2×H×W, where H represents the height dimension of the image and W represents the width dimension of the image;

[0030] The 2×H×W feature tensor is transformed into 1×H×W through a convolutional layer and used as the spatial attention weight. Then, the spatial attention weight is multiplied element-wise with the input channel attention correction feature to obtain the spatial attention correction feature.

[0031] In conjunction with the first possible implementation of the first aspect, in the sixth possible implementation of the first aspect, in S12, the obtained convolutional feature maps are clustered to obtain clustered feature maps, including:

[0032] Remove feature maps with insignificant activation from the obtained convolutional feature maps;

[0033] The remaining convolutional feature maps are classified according to the position of the largest response. Then, the convolutional feature maps of the same class are summed after retaining only the largest response value to obtain the clustering feature map. The retention of the largest response value includes: for the i-th feature map, if its largest response value is located at (H... i W i The value of ) is retained, while the values ​​of the other positions are set to zero.

[0034] In conjunction with the sixth possible implementation of the first aspect, in the seventh possible implementation of the first aspect, the step of classifying the remaining convolutional feature maps according to the position with the largest response using a clustering tool includes: obtaining the position with the largest response value in each remaining convolutional feature map and clustering the position vectors;

[0035] In S13, key content regions are generated based on the clustering feature map, including:

[0036] The size of the clustering feature map is adjusted to match the size of the original image, generating a rectangular bounding box. The response value within the coverage area of ​​the rectangular bounding box is higher than a response threshold. The size and position data of the rectangular bounding box are recorded as key content regions of the clustering feature map. In conjunction with the seventh possible implementation of the first aspect, in the eighth possible implementation of the first aspect, in S3, the global features and key content features of each original image in the data library are used to perform a similarity search on the image to be queried, including:

[0037] Calculate the cosine similarity between the feature vector of the image to be queried and the global features and key content features of each original image in the data library;

[0038] The calculated cosine similarities are sorted, and the degree to which the cosine similarity value is close to the value 1 represents the similarity between the two feature vectors.

[0039] In conjunction with the first aspect, in the ninth possible implementation of the first aspect, the extraction of image features of the query image described in S2 includes:

[0040] Adjust the height and width of the image to be queried to the same preset values ​​as those in the preprocessing in S1;

[0041] The convolutional features are obtained by using a deep residual network with the feature correction module, and the obtained convolutional features are unfolded into a high-dimensional column vector as the feature vector of the query image.

[0042] The image retrieval method based on key content provided in this invention adjusts the size of the original image, extracts the convolutional features of the original images in the image library, clusters the convolutional feature maps, generates key content regions based on the clustered feature maps, extracts features from the original image and the key content regions to obtain feature vectors, adjusts the size of the query image, extracts the feature vector of the query image, calculates the similarity between the query image and the original image in the image library and their key content regions, sorts the results, and returns the corresponding original image. This method achieves the retrieval of the complete image of the main body using key local content, thereby improving retrieval accuracy. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 An image retrieval process is provided as a specific example of an embodiment of the present invention;

[0045] Figure 2 This is a structural diagram of the feature correction module provided in an embodiment of the present invention;

[0046] Figure 3 shows an example of the effect of the clustering feature map provided in the embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the system architecture provided in an embodiment of the present invention;

[0048] Figure 5 This is an application method of the feature correction module provided in the embodiments of the present invention;

[0049] Figure 6 This is a schematic diagram of the method flow provided in an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Embodiments of the present invention will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of the present invention means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0051] The main design objective of this invention is to achieve image retrieval tasks where the query image is incomplete but contains enough key information to retrieve the original image. During feature extraction, an attention mechanism is used to extract effective features from the deep convolutional module, which can improve retrieval accuracy and speed while reducing computational power. The main design idea is to provide an image retrieval method based on key content, such as... Figure 6 As shown, it includes:

[0052] S1. Preprocess the original image to obtain the key content features of the corresponding key content regions.

[0053] S2. Receive the image to be queried uploaded by the user and extract the image features of the image to be queried.

[0054] S3. Using the global features and key content features of each original image in the data library, perform similarity retrieval on the image to be queried.

[0055] In this embodiment, in S1, the preprocessing includes:

[0056] S11. Extract the convolutional features of the original image and form a convolutional feature map. Adjust the shortest side size of the original image to a preset value. In a preferred solution, this preset value is 224 pixels. If this condition is not met, it will be automatically adjusted. In practical applications, the image is not stretched, that is, the height-width ratio of the image remains unchanged before and after size adjustment.

[0057] In this embodiment, a deep residual network (ResNet50) using an attention mechanism module based on the fusion of the channel domain and the spatial domain (abbreviated as the feature correction module) is used to extract the last convolutional feature map. This model is pre-trained on the ImageNet dataset in advance. The output of its last layer is a feature map with a channel number of C and H×W, where H represents the height size and W represents the width size. Specifically, the way the feature correction module is applied to the deep residual network is as Figure 5 shown, where the output of the residual module in each layer of ResNet50 is used as a given intermediate feature map and input into the feature correction module.

[0058] S12. Cluster the obtained convolutional feature map to obtain a clustered feature map. First, perform a preliminary screening on the convolutional feature maps of C channels to remove the feature maps with insignificant activation, and obtain C' feature maps. If the maximum value of a feature map is less than a certain preset threshold, then this feature map is a feature map with insignificant activation. In this embodiment, the preset threshold is taken as 0.1.

[0059] For the remaining C' feature maps, obtain the position (H i , W i ) with the largest response value, where H represents the height size, W represents the width size, and i represents the feature map serial number, taking an integer, 0 < i ≤ C'. Cluster the position vectors. In this embodiment, the k-means clustering algorithm is used to divide them into k categories according to the positions. Experiments show that the effect is optimal when k takes 4.

[0060] Add up the convolutional feature maps of the same class after only retaining the maximum response value to obtain a clustered feature map. The retention of the maximum response value means that for the i-th feature map, the value at the position (H i , W i ) with the largest response value is retained, and the values at the other positions are set to zero. Thus, the feature maps with insignificant activation are removed, and the feature maps are divided into several categories according to the positions with the largest response using the clustering method. The maximum response positions of the feature maps of each category are superimposed to obtain a clustered feature map, filtering the background noise of the clustered feature map. The clustering method includes, but is not limited to, the k-means clustering algorithm. When the k-means algorithm is selected, the k value is best taken as 4.

[0061] S13. Generate key content regions based on the clustering feature map. Adjust the size of the clustering feature map to match the size of the original image, generating a rectangular bounding box. The response value within the coverage area of ​​the rectangular bounding box is higher than a response threshold. Record the size and position data of the rectangular bounding box as the key content regions of the clustering feature map. In a preferred embodiment, the selected threshold is 20% of the maximum value in the clustering feature map.

[0062] Specifically, interpolation can be used to adjust the size of each cluster feature map to the original image size, generate a rectangular box covering the response area above a certain threshold, and obtain its size and position data, i.e., the detected key content area. (1.4) The feature extraction of each area to obtain feature vectors includes cropping the image key content area slices of the corresponding size at the corresponding position of the original image based on the position and size information generated based on each cluster feature map, adjusting the cropped image to a size of 224×224, using a pre-trained deep residual network ResNet50 with an attention mechanism to extract features, and unfolding the obtained convolutional features into a high-dimensional column vector. The extracted global features and key content features of the same image are associated with the original image ID and stored in the database.

[0063] S14. Extract features from the original image and the key content region respectively to obtain feature vectors for each. Adjust the height and width of the original image and the key content region to the same preset value. In this embodiment, the preset height and width are 224 pixels. Convolutional features are obtained through a deep residual network with an attention mechanism (implemented through a feature correction module), and these convolutional features are unfolded into high-dimensional column vectors. The convolutional features obtained using the deep residual network with the feature correction module are unfolded into high-dimensional column vectors and used as feature vectors for the original image and the key content region. Associate the extracted global features and key content features of the same image with the original image ID and store them in the database.

[0064] Further, in S11, extracting the convolutional features of the original image and forming a convolutional feature map includes: adjusting the shortest side size of the original image to a preset value. A feature map is extracted using a deep residual network. The convolutional feature map includes the feature maps obtained from the first to the last convolution, with the feature maps obtained during intermediate convolution stages referred to as "intermediate feature maps." An attention mechanism module (which can be called a feature correction module) based on channel and spatial domain fusion is added to the deep residual network. Through the feature correction module, attention weights for a given intermediate feature map are obtained, and the obtained attention weights are multiplied by the given intermediate feature map to obtain an attention-enabled feature map. The given intermediate feature map includes the feature maps output by the residual modules in each layer of the deep residual network.

[0065] Specifically, the feature correction module includes: a spatial self-attention module M. self Channel Attention Module M channel Spatial attention module M spatial .in:

[0066]

[0067]

[0068]

[0069] Where F represents the given intermediate feature map, F' represents the spatial self-attention correction feature, F” represents the channel attention correction feature, F”' represents the spatial attention correction feature, and the operator This indicates element-wise multiplication. The step of obtaining attention weights from a given intermediate feature map via the feature correction module includes: in the spatial self-attention module, reducing the dimensionality of the given intermediate feature map through a 1×1 convolution operation to obtain a 1×H×W feature tensor, then transforming the feature values ​​to the [0,1] interval using the Sigmoid activation function, and finally obtaining the spatial self-attention weights M using the SoftMax function. self Spatial self-attention modified features are obtained by element-wise multiplying spatial self-attention weights and input features, where H represents the height of the image and W represents the width of the image. For example, it can be implemented as the formula: M self (F) = SoftMax(∫ 1@1×1 (F)), where ∫ 1@1×1 (F) represents a two-dimensional convolution operation on F, where the size of the convolution kernel is 1×1 and the number of convolution kernels is 1. SoftMax() represents the SoftMax function.

[0070] Furthermore, the step of obtaining attention weights from a given intermediate feature map through the feature correction module further includes: processing the spatial self-attention corrected features using spatial max pooling and average pooling respectively through the channel attention module to obtain two feature tensors of dimension C×1×1, where C represents the number of channels. Then, the two obtained feature tensors of dimension C×1×1 are passed through C shared 1×1 networks to map them into two new feature tensors of dimension C×1×1. The two new feature tensors are then summed element-wise and passed through a SoftMax activation function to output a channel attention weight M of dimension C×1×1. channel The channel attention correction features are obtained by element-wise multiplying the channel attention weights and the input spatial self-attention correction features. For example, this can be implemented as the formula:

[0071] M channel (F) = SoftMax(∫ C@1×1 (AvgPool(F))+∫ C@1×1 (MaxPool(F))), where ∫ C@1×1 (F) represents a two-dimensional convolution operation on F, where the kernel size is 1×1 and the number of kernels is C. AvgPool(F) represents average pooling, and MaxPool(F) represents max pooling.

[0072] In this embodiment, the channel attention module performs both max pooling and average pooling on the feature map spatially, resulting in two c×1×1 weight vectors. These two weight vectors are then passed through the same MLP network, mapped to the weights of each channel. The sum of these two vectors is then passed through a sigmoid activation function to output a single c×1×1 channel weight. This approach effectively emphasizes or suppresses information from intermediate feature layers, aiding information propagation within the network and enhancing the ability to extract key information features from complex backgrounds.

[0073] Furthermore, the step of obtaining attention weights from a given intermediate feature map through the feature correction module further includes: processing the channel attention correction features using both channel max pooling and average pooling methods via the spatial attention module to obtain two feature tensors of dimension 1×H×W; then merging these two 1×H×W feature tensors to form a 2×H×W feature tensor, where H represents the image height and W represents the image width. A convolutional layer is then used to transform the 2×H×W feature tensor to 1×H×W, which is then used as the spatial attention weights. Finally, the spatial attention weights are multiplied element-wise with the input channel attention correction features to obtain the spatial attention correction features. For example, this can be implemented as the formula:

[0074] M spatial (F) = SoftMax(∫ 1@1×1 ([AvgPool(F);MaxPool(F)])), where ∫ 1@1×1 (F) represents a two-dimensional convolution operation on F, where the size of the convolution kernel is 1×1 and the number of convolution kernels is 1.

[0075] In this embodiment, in S12, the obtained convolutional feature maps are clustered to obtain clustered feature maps, including: removing feature maps with insignificant activation from the obtained convolutional feature maps. "Insignificant activation" can be understood as: if the maximum value of a feature map is less than a certain preset threshold, then this feature map is considered to have insignificant activation. The remaining convolutional feature maps are classified according to the position of the largest response using a clustering tool. Then, the convolutional feature maps of the same class are summed, retaining only the maximum response value, to obtain the clustered feature map. The retention of the maximum response value includes: for the i-th feature map, if its maximum response value is located at (H... i W i The value of ) is retained, while the values ​​of the other positions are set to zero.

[0076] Specifically, the step of classifying the remaining convolutional feature maps according to the location of the largest response using a clustering tool includes: obtaining the location with the largest response value in each remaining convolutional feature map and clustering the location vectors. For example, the effect of clustering feature maps shown in Figure 3 is that the clustering method is used to divide the feature maps into several classes according to the location of the largest response, where the location with the largest response value is obtained and the location vector is clustered. The step of generating key content regions based on the clustered feature maps includes generating clustered feature maps, adjusting the size of each clustered feature map to the original image size, generating a rectangular box covering the response area above a certain threshold, and obtaining its size and position data, i.e., the detected key content regions. Its beneficial effect is that the high response positions of the clustered feature maps can reflect the key parts of the target in the original image.

[0077] In S13, generating key content regions based on the clustering feature map includes: adjusting the size of the clustering feature map to match the size of the original image, generating a rectangular bounding box, wherein the response value within the coverage area of ​​the rectangular bounding box is higher than a response threshold. The size and position data of the rectangular bounding box are recorded as key content regions of the clustering feature map.

[0078] Interpolation can be used to adjust the size of each cluster feature map to the original image size, generating a rectangular box covering the response area above a certain threshold and obtaining its size and position data, i.e., the detected key content area. (1.4) The feature extraction of each area to obtain feature vectors includes cropping the image key content area slices of the corresponding size at the corresponding position of the original image based on the position and size information generated based on each cluster feature map, adjusting the cropped image to a size of 224×224, using a pre-trained deep residual network ResNet50 with an attention mechanism to extract features, and unfolding the obtained convolutional features into a high-dimensional column vector. The extracted global features and key content features of the same image are associated with the original image ID and stored in the database.

[0079] Furthermore, the process includes S14, where feature extraction of the original image and key content regions includes adjusting the height and width of the original image and key content regions to the same preset value. Convolutional features are obtained using the deep residual network with the added feature correction module, and these convolutional features are unfolded into high-dimensional column vectors, which serve as feature vectors for the original image and key content regions.

[0080] In this embodiment, in step S3, the global features and key content features of each original image in the data image library are used to perform similarity retrieval on the image to be queried. This includes calculating the cosine similarity between the feature vector of the image to be queried and the global features and key content features of each original image in the data image library. The calculated cosine similarities are sorted, and the degree to which the cosine similarity value is close to the value 1 represents the similarity between the two feature vectors. The first specified number of results are taken, and the original image is found based on its association with the original image ID, and returned as the final result.

[0081] In this embodiment, in S2, a query image uploaded by the user can be received, and image features of the query image can be extracted. Specifically, this includes: adjusting the height and width of the query image to the same preset values ​​as in the preprocessing described in S1. The height and width of the query image are adjusted to the same preset values. In this embodiment, the selected preset values ​​for height and width are 224 pixels. A deep residual network with the feature correction module is used to obtain convolutional features, and the obtained convolutional features are unfolded into a high-dimensional column vector, which serves as the feature vector of the query image. For example, for the query image, the image size is first adjusted, and a deep residual network with an attention mechanism is used to obtain convolutional features, which are then unfolded into a high-dimensional column vector. Therefore, the feature extraction of the query image includes: the user inputs a query image, which is first adjusted to a size of 224×224. Features are extracted using a pre-trained deep residual network ResNet50 with an attention mechanism, and the obtained convolutional features are unfolded into a high-dimensional column vector.

[0082] The subsequent similarity retrieval process mainly includes: calculating the cosine distance between the feature vector of the query image and the global features and key content features of each original image in the image database; sorting the obtained cosine distances in reverse order; taking the first specified number of results; finding the original image based on its association with the original image ID; and returning it as the final result. The similarity retrieval process includes:

[0083] Calculate the cosine distance between the feature vector of the query image and the global features and key content features of each original image in the database, i.e., the magnitude of the difference between the two vectors. Sort the obtained cosine distances in reverse order; the closer the cosine distance is to 1, the more similar the two images are. Optionally, when the original image library is large, an index built using the Faiss similarity vector retrieval library can be used to find local optima and improve retrieval speed. Take the first specified number of results, find the original image based on its association with the original image ID, and return it as the final result.

[0084] For example, in practical applications, this embodiment can be implemented as follows: Figure 1 The specific process is as follows:

[0085] SS1. Adjust the original image size. The minimum size of the shortest side of the input image in this invention is 224 pixels. If this condition is not met, the image will be automatically adjusted. In this step, the image is not stretched; that is, the aspect ratio of the original image remains unchanged.

[0086] SS2. Extract convolutional feature maps from the original images in the image library. The feature maps refer to all feature maps output by the last layer of the neural network. This example uses a deep residual network, ResNet50, with an attention mechanism to extract the feature maps from the last convolution. This model was pre-trained on the ImageNet dataset. Its last layer output is a feature map with c channels and an H×W dimension.

[0087] SS3. Clustering of Convolutional Feature Maps. First, perform initial screening on the c-channel feature maps, removing those with insignificant activation, resulting in c feature maps. For each feature map, obtain the location (H) with the largest response value. i W i We then cluster the data. In this example, we use the k-means clustering algorithm to divide the data into k categories based on their positions. Experiments show that a value of k = 4 yields the best results.

[0088] SS4. Map all maximum value locations in each cluster category onto a blank H×W feature map to obtain the cluster feature map for that category. Use interpolation to resize the k cluster feature maps to the original image size. The high response locations in these feature maps can reflect the key parts of the target in the original image. For each cluster feature map, generate a bounding box covering the response area above a certain threshold and obtain its size and location data, i.e., the detected key content area. In this example, the certain threshold is 20%.

[0089] SS5. Extract features from key content regions. Based on the location and size information generated from each cluster feature map, crop the corresponding size of key content regions from the original image, adjust their size to 224×224, and use a pre-trained ResNet50 deep residual network with an attention mechanism to extract features. Expand the obtained convolutional features into high-dimensional column vectors. Associate the extracted global features and key content features of the same image with the original image ID and store them in the database.

[0090] SS6. When a user inputs a query image, its size is first adjusted to 224×224.

[0091] SS7 uses a pre-trained deep residual network ResNet50 with an attention mechanism to extract features, and then unfolds the resulting convolutional features into a high-dimensional column vector.

[0092] SS8. Calculate the cosine distance between the feature vector of the query image and the global features and key content features of each original image in the database, i.e., the magnitude of the difference between the two vectors.

[0093] SS9. Sort the obtained cosine distances in reverse order. The closer the cosine distance is to 1, the more similar the two images are.

[0094] SS10: Take the first specified number of results, find the original image based on its association with the original image ID, and return the final result.

[0095] In this embodiment, convolutional features of the original images in the image library are extracted, the convolutional feature maps are clustered, key content regions are generated based on the clustered feature maps, feature vectors are extracted from the original images and key content regions, the size of the query image is adjusted, the feature vector of the query image is extracted, the similarity between the query image and the original images in the image library and their key content regions is calculated, the results are sorted, and the corresponding original images are returned. This method is particularly suitable for image retrieval tasks where the query image is incomplete but contains enough key information to retrieve the original image.

[0096] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for image retrieval based on key local information, characterized in that, The method comprises the following steps: S1, preprocessing the original image to obtain key content features corresponding to the key content region; S2, receiving a user-uploaded image to be queried and extracting image features of the image to be queried; S3, using global features and key content features of each original image in a data gallery to perform similarity retrieval on the image to be queried; In S1, the preprocessing comprises: S11, extracting convolutional features of the original image and forming a convolutional feature map; S12, clustering the obtained convolutional feature map to obtain a clustered feature map; S13, generating a key content region according to the clustered feature map; and S14, respectively extracting features of the original image and the key content region to obtain feature vectors of the original image and the key content region respectively; In S11, the extraction of the convolutional features of the original image and the formation of the convolutional feature map comprise: adjusting the size of the shortest side of the original image to a preset value; extracting a feature map through a deep residual network, wherein a feature correction module is added to the deep residual network; obtaining attention weights of a given intermediate feature map through the feature correction module, multiplying the obtained attention weights with the given intermediate feature map to obtain a feature map with attention, wherein the given intermediate feature map is output by a residual module in the deep residual network; The feature correction module comprises a spatial self-attention module M self , a channel attention module M channel , and a spatial attention module M spatial . wherein, denotes a given intermediate feature map, denotes a spatial self-attention corrected feature, denotes a channel attention corrected feature, denotes a spatial attention corrected feature, and the operator denotes element-wise multiplication. The attention weight is obtained from the given intermediate feature map through a feature correction module, including: in the spatial self-attention module, the given intermediate feature map is subjected to a 1*1 convolution operation to reduce the dimension to obtain a 1*H*W feature tensor, then the feature value is converted to the [0, 1] interval through a Sigmoid excitation function, and then a spatial self-attention weight is obtained by using a SoftMax function The spatial self-attention correction feature is obtained by element-wise multiplication of the spatial self-attention weight and the input feature, wherein H represents the height dimension of the picture, and W represents the width dimension of the picture.

2. The method of claim 1, wherein, The obtaining of the attention weights from the given intermediate feature map through the feature correction module further comprises: The spatial self-attention correction feature is respectively processed according to spatial maximum pooling and average pooling through the channel attention module to obtain two feature tensors with dimensions of Cx1x1, wherein C represents the number of channels. After that, the two obtained feature tensors with the dimensions of Cx1x1 pass through the shared C 1x1 networks, are mapped into two new feature tensors with the dimensions of Cx1x1, and the obtained two new feature tensors are element-wise added and then output to a channel attention weight M with the dimension of Cx1x1 through a SoftMax activation function channel ; The channel attention weights and the input spatial self-attention correction feature are multiplied element by element to obtain a channel attention correction feature.

3. The method according to claim 1 or 2, characterized in that, The obtaining of the attention weights from the given intermediate feature map through the feature correction module further comprises: the channel attention correction feature is respectively processed according to channel maximum pooling and average pooling through the spatial attention module to obtain two feature tensors with dimensions of 1xHxW, and then the two obtained feature tensors with dimensions of 1xHxW are merged to form a feature tensor with dimensions of 2xHxW; The feature tensor with dimensions of 2xHxW is converted to a feature tensor with dimensions of 1xHxW through a convolutional layer, and the feature tensor with dimensions of 1xHxW is taken as a spatial attention weight, and then the spatial attention weight and the input channel attention correction feature are multiplied element by element to obtain a spatial attention correction feature.

4. The method of claim 1, wherein, In S12, the clustering of the obtained convolutional feature map to obtain a clustered feature map comprises: In the obtained convolutional feature map, the feature maps with insignificant activations are removed; The remaining convolution feature maps are classified according to the position of the maximum response, and then the convolution feature maps of the same class are only retained after the maximum response value is added to obtain a clustering feature map, wherein the maximum response value is retained, including: for the i-th feature map, if the value of the position (H i , W i ) of the maximum response is retained, while the values of the remaining positions are set to zero.

5. The method of claim 4, wherein, The classification of the remaining convolutional feature maps according to the positions with the maximum responses comprises: obtaining the positions with the maximum responses in each remaining convolutional feature map, and clustering the position vectors; In S13, the generation of the key content region according to the clustered feature map comprises: Adjusting the cluster feature map size to the size of the original image, generating a rectangular frame, the response value in the coverage of the rectangular frame being higher than a response threshold; recording the size and position data of the rectangular frame as the key content area of the cluster feature map.

6. The method of claim 5, wherein, In S3, the similarity of the to-be-queried image is searched by using the global feature and the key content feature of each original image in the data library, including: The cosine similarity of the feature vector of the to-be-queried image and the global feature and the key content feature of each original image in the data library is calculated. The calculated cosine similarity is sorted, and the closeness of the value of the cosine similarity to the numerical value 1 represents the similarity between the two feature vectors.

7. The method of claim 1 wherein, The image feature of the to-be-queried image in S2 includes: Adjusting the height and width of the to-be-queried image to the same preset value in the preprocessing in S1; Using the deep residual network with the feature correction module to obtain the convolution feature, and expanding the obtained convolution feature into a high-dimensional column vector as the feature vector of the to-be-queried image.

Images