Image retrieval method based on attention enhancement and autoencoder fusion

Abstract

This invention presents an image retrieval method based on the fusion of attention enhancement and autoencoding. It uses an improved ResNet50 network to extract global and local feature maps, obtains global and local feature descriptors based on these maps, calculates image similarity, and derives the target image through similarity comparison. This image retrieval method improves upon traditional residual blocks by employing automaton encoding, effectively unifying local and global features into a single network. It uses an attention mechanism to extract regions of greater interest, avoiding algorithmic overhead and resulting in faster retrieval speed and higher accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, specifically relating to an image retrieval method based on attention enhancement and autoencoding fusion. Background Technology

[0002] Currently, heterogeneous data such as images, videos, audio, and text are growing at an astonishing rate every day. How to conveniently, quickly, and accurately search and retrieve the images users need or are interested in from these vast image databases, given the sheer volume of images containing rich visual information, is a pressing problem. Previously, most images were indexed using manual annotation, which suffered from low efficiency and subjectivity. TBIR (Text-Based Image Retrieval) requires image uploaders to provide necessary annotations, the system stores and indexes the images according to the annotations, and then the search engine uses retrieval technology to find images similar to the keywords provided by the user. However, because this method completely ignores the image content itself and relies excessively on user-provided or collected keywords, the accuracy of image retrieval results cannot be guaranteed. Therefore, CBIR (Content-Based Image Retrieval) was proposed and has received widespread attention. This retrieval method extracts low-level features from an image and uses the similarity between these features to obtain the final result. However, this traditional method only extracts low-level features, resulting in a "semantic gap." While convolutional neural networks extract high-level features with semantic characteristics, the traditional residual block arrangement is flawed in deep convolutional ResNets. The input feature maps are not normalized beforehand, which prevents the BN layer from playing a significant role. Furthermore, the extracted features are only single features and cannot identify the regions of interest in the image, resulting in low accuracy of the retrieval results. Summary of the Invention

[0003] The purpose of this invention is to provide an image retrieval method based on attention enhancement and autoencoding fusion, which solves the problem that existing technologies cannot determine the region of interest in an image, resulting in low accuracy of retrieval results.

[0004] The technical solution adopted in this invention is: an image retrieval method based on attention enhancement and autoencoding fusion, which uses an improved ResNet50 network to extract global feature maps and local feature maps, obtains global feature descriptors and local feature descriptors based on global feature maps and local feature maps, calculates image similarity, and obtains the target image through similarity comparison.

[0005] The invention is further characterized by:

[0006] The image retrieval method based on attention enhancement and autoencoding fusion is implemented according to the following steps:

[0007] Step 1: Improve the traditional ResNet50 model, train the residual network and form the network backbone, and obtain the global and local branch networks through the network backbone;

[0008] Step 2: Given an image, obtain two feature maps through the two branch networks in Step 1, namely the global feature map and the local feature map. Extract the deep activation feature D from the global feature map and the shallow activation feature S from the local feature map.

[0009] Step 3: Aggregate the deep activation features D extracted in Step 2 into a global feature and perform global feature learning to obtain a global feature descriptor. Based on the shallow activation features S extracted in Step 2, perform local feature learning through an attention mechanism to obtain a local feature descriptor.

[0010] Step 4: Calculate the similarity between the query image and the images in the Google Landmark dataset based on the local and global feature descriptors; rank and select the images in the Google Landmark dataset based on the similarity to obtain the target image.

[0011] Step 1 is implemented in the following steps:

[0012] Step 1.1: Improve the residual blocks of ResNet50. The residual blocks are arranged in the following order: batch normalization layer, then convolutional layer, and finally ReLU activation function layer.

[0013] Step 1.2: Train the ResNet residual network on the Google Landmark dataset. The number of iterations is 100k, the learning rate is 1e-3, and the weight decay is 0.0005. After training, two branches, global and local, are obtained.

[0014] Step 2 is implemented in the following steps:

[0015] Given an image, we obtain global and local feature maps using the hierarchical representation of convolutional layers. From the output of conv4, we obtain the shallow activation features S in the local feature maps, denoted as: S R^(〖H_S×W_S×C〗_S ) , obtains the deep activation feature D in the global feature map from the output of conv5, denoted as: D R^(〖H_D×W_D×C〗_D ), where H, W, and C represent the height, width, and number of channels in each case; the deep activation feature D has 2048 channels and a feature dimension of 2048, while the shallow activation feature S has 1024 channels and a feature dimension of 128.

[0016] Step 3 involves obtaining the global feature descriptor.

[0017] Step 3.1a: Use a fully connected layer F linear mapping layer to integrate the feature dimensions and complete the extraction of global features. The formula for global feature g is shown in (1):

[0018] (1)

[0019] Where F is the linear mapping matrix, b is the bias, and p is the norm of GeM Pooling. The element is represented by the hyperparameter p=3 in GeM Pooling;

[0020] Step 3.2a: The global feature learning adopts normalized softmax and cross-entropy loss, and reduces intra-class differences by introducing ArcFace margin. The ArcFace margin is calculated as shown in formula (2):

[0021] (2)

[0022] Where u is the cosine similarity, m is the arc margin, we set the arc margin m = 0.1, and c is a binary value;

[0023] The cross-entropy loss calculated using softmax normalization is shown in Equation (3):

[0024] (3)

[0025] The learnable scale parameter is initialized as: γ = = 45.25; Is class The L2 normalized classifier weights, where y is the one-hot ground truth label, and is 1 at class k;

[0026] Complete global feature learning to obtain a global descriptor. .

[0027] The specific steps for obtaining the local feature descriptor in step 3 are as follows:

[0028] Step 3.1b: Use an autoencoder AE structure to represent local features, that is, add a 1x1 convolution as encoder T to reduce the number of channels in the original feature map to obtain a low-dimensional local feature representation; and to facilitate training, a 1x1 convolution will be added as decoder to reconstruct the original feature map using low-dimensional local features; the autoencoder consists of two layers of 1x1 convolutions without stride, and the activation function is ReLU in the first layer and Softplus in the second layer;

[0029] The loss function formula for an auto encoder is shown in (4):

[0030] (4)

[0031] S is the input. It was generated after deconvolution;

[0032] Step 3.2b: Use an attention network to assign weights to the low-dimensional local features extracted in step 3.1b. The selection of local features mainly relies on a small attention module to select the most distinctive regions. Here, the attention heatmap is obtained by a small convolutional network. The output y of the attention mechanism is the weighted sum of the weights of the convolutional features extracted by the network. The output of the network is shown in formula (5).

[0033] (5)

[0034] Scoring function It is based on local features Training yielded, among which It is a function Parameters, scoring function The parameters in are passed through Training is performed using backpropagation, where the gradient is shown in equation (6):

[0035] (6)

[0036] To prevent the scoring function from learning negative weights, restrictions are imposed. The scoring function is designed using a two-layer CNN with softplus activation at the top, and uses a 1x1 convolutional filter;

[0037] Step 3.3b: Use attention weights to integrate local features and supervise the generation of the attention map, as shown in formula (7):

[0038] (7)

[0039] Step 3.4b: Local features complete the basic classification task during training, forming the cross-entropy loss as shown in formula (8):

[0040] (8)

[0041] The loss function for training the entire network is global feature loss + reconstruction feature loss + local feature loss. , where we set weight = 10, weight = 1;

[0042] After completing the local feature learning, the local descriptor L=T(S) is obtained.

[0043] Step 4 is implemented in the following steps:

[0044] Step 1: Represent the image using a global descriptor, calculate the distance between the query image and other images according to the Euclidean distance formula, and return the n closest images.

[0045] Step 2: Further sort and filter the n images from Step 1 using local descriptors;

[0046] Step 3: Calculate the similarity between the query image and the n images sorted and filtered in Step 2 using the Euclidean distance formula, and return the top m images with the highest similarity. These m images are the final target images retrieved.

[0047] The beneficial effects of this invention are:

[0048] 1. An improved ResNet50 network is used to extract features with high-level semantic information. An attention mechanism is used to identify regions of interest in the image, avoiding the extraction of some useless features and obtaining more accurate features. At the same time, local and global features are fused to obtain richer image feature information, making the retrieval faster and more accurate.

[0049] 2. The traditional residual block has been improved. The improved residual block order is batch normalization layer, then convolutional layer, and finally ReLU activation function. The improved residual block arrangement method of this invention not only retains the identity mapping of the left path, but also maintains the learning ability of the right nonlinear network path.

[0050] 3. Automaton encoding is used to transform high-dimensional features into low-dimensional features and effectively integrate global and local features into a single network, reducing algorithm redundancy and improving retrieval efficiency;

[0051] 4. It uses a scoring attention mechanism, which focuses on the region of interest in the image, effectively extracting the regions of interest we need, improving the accuracy of retrieval, avoiding feature extraction of useless regions, and improving retrieval efficiency. Attached Figure Description

[0052] Figure 1 This is a flowchart of the image retrieval method of the present invention;

[0053] Figure 2 This is a structural diagram of the improved residual module of the present invention;

[0054] Figure 3This is a flowchart of the present invention based on local and global feature extraction;

[0055] Figure 4 This refers to the image to be retrieved selected in this embodiment of the invention;

[0056] Figure 5 These are images retrieved by the global descriptor in this embodiment of the invention;

[0057] Figure 6 This is the image that was finally retrieved in this embodiment of the invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] This invention is an image retrieval method based on the fusion of attention enhancement and autoencoding. It uses an improved ResNet50 network to extract global and local feature maps, obtains global and local feature descriptors based on the global and local feature maps, calculates image similarity, and obtains the target image by similarity comparison.

[0060] like Figure 1 As shown, please follow these steps:

[0061] Step 1: Improve the traditional ResNet50 model, train the residual network and form the network backbone, and obtain the global and local branches through the network backbone;

[0062] Step 2: Given an image, obtain two feature maps through the two branch networks in Step 1, namely the global feature map and the local feature map. Extract the deep activation feature D from the global feature map and the shallow activation feature S from the local feature map.

[0063] Step 3: Aggregate the deep activation features D extracted in Step 2 into a global feature and perform global feature learning to obtain a global feature descriptor. Based on the shallow activation features S extracted in Step 2, perform local feature learning through an attention mechanism to obtain a local feature descriptor.

[0064] Step 4: Calculate the similarity between the query image and the images in the Google Landmark dataset based on the local and global feature descriptors; rank and select the images in the Google Landmark dataset based on the similarity to obtain the target image.

[0065] like Figure 2 As shown, step 1 is implemented in the following steps:

[0066] Step 1.1: Improve the residual blocks of ResNet50. The residual blocks are arranged in the following order: batch normalization layer, then convolutional layer, and finally ReLU activation function layer.

[0067] Step 1.2: Train the ResNet residual network on the Google Landmark dataset. Number of iterations: 100k, learning rate: 1e-3, weight decay: 0.0005. After training, two branches, global and local, are obtained.

[0068] like Figure 3 As shown, step 2 is implemented in the following steps:

[0069] Given an image, we obtain global and local feature maps using the hierarchical representation of convolutional layers. From the output of conv4, we obtain the shallow activation features S in the local feature maps, denoted as: S R^(〖H_S×W_S×C〗_S ) , obtains the deep activation feature D in the global feature map from the output of conv5, denoted as: D R^(〖H_D×W_D×C〗_D ), where H, W, and C represent the height, width, and number of channels in each case; the deep activation feature D has 2048 channels and a feature dimension of 2048, while the shallow activation feature S has 1024 channels and a feature dimension of 128.

[0070] Step 3 involves obtaining the global feature descriptor.

[0071] Step 3.1a: By integrating the feature dimensions using a fully connected layer F and a linear mapping layer, the extraction of global features is completed. The formula for the global feature g is shown in formula (1):

[0072] (1)

[0073] Where F is the linear mapping matrix, b is the bias, and p is the norm of GeM Pooling. This represents an element. The hyperparameter p=3 for GeM Pooling.

[0074] Step 3.2a: The global feature learning employs normalized softmax and cross-entropy loss, also known as the "cosine classifier," and introduces ArcFace margin, which effectively reduces intra-class variance. The definition of ArcFace margin is shown in formula (2):

[0075] (2)

[0076] Where u is the cosine similarity, m is the arc margin, we set the arc margin m = 0.1, and c is a binary value;

[0077] The cross-entropy loss calculated using softmax normalization is shown in Equation (3):

[0078] (3)

[0079] The learnable scale parameter is initialized as: γ = = 45.25. Is class The L2 normalized classifier weights, where y is the one-hot ground truth label, and is 1 at class k;

[0080] Complete global feature learning to obtain a global descriptor. .

[0081] The specific steps for obtaining the local feature descriptor in step 3 are as follows:

[0082] Step 3.1b: Due to the large number of local features, an autoencoder (AE) structure is used to make the feature representation denser and reduce computational cost. This module is responsible for learning a suitable low-dimensional representation. Specifically, a 1x1 convolution is added as the encoder T to reduce the number of channels in the original feature map, thus obtaining a low-dimensional local feature representation. To facilitate training, another 1x1 convolution is added as the decoder to reconstruct the original feature map using the low-dimensional local features. The autoencoder consists of two layers of 1x1 convolutions with no stride; the first layer uses ReLU activation, and the second layer uses Softplus activation.

[0083] The loss function formula for an auto encoder is shown in (4):

[0084] (4)

[0085] S is the input. It is generated after deconvolution.

[0086] Step 3.2b: After dimensionality reduction, an attention network is used. The selection of local features mainly relies on a small attention module to select the most distinctive regions. Here, the attention heatmap is obtained by a small convolutional network. The attention network is used to assign weights to the low-dimensional local features extracted in step 3.1b. The selection of local features mainly relies on a small attention module to select the most distinctive regions. Here, the attention heatmap is obtained by a small convolutional network. The output y of the attention mechanism is the weighted sum of the weights of the convolutional features extracted by the network. The network output is shown in formula (5):

[0087] (5)

[0088] Scoring function It is based on local features Training yielded, among which It is a function The parameters.

[0089] Scoring function The parameters in are passed through Training is performed using backpropagation, where the gradient is shown in equation (6):

[0090] (6)

[0091] To prevent the scoring function from learning negative weights, this paper restricts... The scoring function is designed using a two-layer CNN with softplus activation at the top, and this paper uses a 1x1 convolutional filter.

[0092] Step 3.3b: To supervise the generation of the attention map, attention weights are used to integrate local features, as shown in formula (7):

[0093] (7)

[0094] Step 3.4b: This local feature will also complete the basic classification task during training, forming the cross-entropy loss as shown in formula (8):

[0095] (8)

[0096] The loss function for training the entire network is global feature loss + reconstruction feature loss + local feature loss. , where we set weight = 10, weight = 1;

[0097] After completing the local feature learning, the local descriptor L=T(S) is obtained.

[0098] Step 4 is implemented in the following steps:

[0099] Step 1: Represent the image using a global descriptor, calculate the distance between the query image and other images according to the Euclidean distance formula, and return the n closest images.

[0100] Step 2: Further sort and filter the n images from Step 1 using local descriptors;

[0101] Step 3: Calculate the similarity between the query image and the n images sorted and filtered in Step 2 using the Euclidean distance formula, and return the top m images with the highest similarity. These m images are the final target images retrieved.

[0102] Example

[0103] Figure 4 The query image is randomly selected; it is an image from the Eiffel dataset in the Paris6k dataset, and the retrieval method of this invention is used to retrieve it. Figure 5 The seven images with the highest similarity were selected using global feature descriptors. Figure 6 Through local descriptors Figure 5 The results are reordered and filtered from the search results.

Claims

1. An image retrieval method based on the fusion of attention enhancement and autoencoding, characterized in that, The improved ResNet50 network is used to extract global and local feature maps. Based on the global and local feature maps, global and local feature descriptors are obtained. Image similarity is calculated, and the target image is obtained by similarity comparison. The specific steps are as follows: Step 1: Improve the traditional ResNet50 model, train the residual network and form the network backbone, and obtain the global and local branch networks through the network backbone; Step 1 is implemented in the following steps: Step 1.1: Improve the residual blocks of ResNet50. The residual blocks are arranged in the following order: batch normalization layer, then convolutional layer, and finally ReLU activation function layer. Step 1.2: Train the ResNet residual network on the Google Landmark dataset. Number of iterations: 100k, learning rate: 1e-3, weight decay: 0.0005. After training, two branches, global and local, are obtained. Step 2: Given an image, obtain two feature maps through the two branch networks in Step 1, namely the global feature map and the local feature map. Extract the deep activation feature D from the global feature map and the shallow activation feature S from the local feature map. Step 3: Aggregate the deep activation features D extracted in Step 2 into a global feature and perform global feature learning to obtain a global feature descriptor. Based on the shallow activation features S extracted in Step 2, perform local feature learning through an attention mechanism to obtain a local feature descriptor. Step 3 involves obtaining the global feature descriptor. Step 3.1a: Use a fully connected layer F linear mapping layer to integrate the feature dimensions and complete the extraction of global features. The formula for global feature g is shown in (1): （1） Where F is the linear mapping matrix, b is the bias, and p is the norm of GeM Pooling. The element is represented by the hyperparameter p=3 in GeMPooling; Step 3.2a: The global feature learning adopts normalized softmax and cross-entropy loss, and reduces intra-class differences by introducing ArcFace margin. The ArcFace margin is calculated as shown in formula (2): （2） Where u is the cosine similarity, m is the arc margin, we set the arc margin m = 0.1, and c is a binary value; The cross-entropy loss calculated using softmax normalization is shown in Equation (3): （3） The learnable scale parameter is initialized as: γ = = 45.25; Is class The L2 normalized classifier weights, where y is the one-hot ground truth label, and is 1 at class k; Complete global feature learning to obtain a global descriptor. ; The specific steps for obtaining the local feature descriptor in step 3 are as follows: Step 3.1b: Use an autoencoder AE structure to represent local features, that is, add a 1x1 convolution as encoder T to reduce the number of channels in the original feature map to obtain a low-dimensional local feature representation; and to facilitate training, a 1x1 convolution will be added as decoder to reconstruct the original feature map using low-dimensional local features; the autoencoder consists of two layers of 1x1 convolutions without stride, and the activation function is ReLU in the first layer and Softplus in the second layer; The loss function formula for an auto encoder is shown in (4): （4） S is the input. It was generated after deconvolution; Step 3.2b: Use an attention network to assign weights to the low-dimensional local features extracted in step 3.1b. The selection of local features mainly relies on a small attention module to select the most distinctive regions. Here, the attention heatmap is obtained by a small convolutional network. The output y of the attention mechanism is the weighted sum of the weights of the convolutional features extracted by the network. The output of the network is shown in formula (5). （5） Scoring function It is based on local features Training yielded, among which It is a function Parameters, scoring function The parameters in are passed through Training is performed using backpropagation, where the gradient is shown in equation (6): （6） To prevent the scoring function from learning negative weights, restrictions are imposed. The scoring function is designed using a two-layer CNN with softplus activation at the top, and uses a 1x1 convolutional filter; Step 3.3b: Use attention weights to integrate local features and supervise the generation of the attention map, as shown in formula (7): （7） Step 3.4b: Local features complete the basic classification task during training, forming the cross-entropy loss as shown in formula (8): （8） The loss function for training the entire network is global feature loss + reconstruction feature loss + local feature loss. , where we set weight = 10, weight = 1; Complete local feature learning to obtain the local descriptor L=T(S); Step 4: Calculate the similarity between the query image and the images in the Google Landmark dataset based on the local and global feature descriptors; rank and select the images in the Google Landmark dataset based on the similarity to obtain the target image.

2. The image retrieval method based on attention enhancement and autoencoder fusion according to claim 1, characterized in that, Step 2 is implemented in the following steps: Given an image, we obtain global and local feature maps using the hierarchical representation of convolutional layers. From the output of conv4, we obtain the shallow activation features S in the local feature maps, denoted as: S R^(〖H_S×W_S×C〗_S ) , obtains the deep activation feature D in the global feature map from the output of conv5, denoted as: D R^(〖H_D×W_D×C〗_D ), where H, W, and C represent the height, width, and number of channels in each case; the deep activation feature D has 2048 channels and a feature dimension of 2048, while the shallow activation feature S has 1024 channels and a feature dimension of 128.

3. The image retrieval method based on attention enhancement and autoencoder fusion according to claim 1, characterized in that, Step 4 is implemented in the following steps: Step 4.1: Represent the image using a global descriptor, calculate the distance between the query image and other images according to the Euclidean distance formula, and return the n closest images. Step 4.2: Further sort and filter the n images from Step 1 using local descriptors; Step 4.3: Calculate the similarity between the query image and the n images sorted and filtered in Step 2 using the Euclidean distance formula, and return the top m images with the highest similarity. These m images are the final target images retrieved.

Images