A large-scale image retrieval hashing method focusing on inter-image block relationship

Abstract

The application discloses a large-scale image retrieval hash method focusing on the relationship between image blocks, characterized in that: firstly, multi-scale features are extracted from samples through a first self-attention calculation module and a second self-attention calculation module; then, a global average pooling layer, a hash layer and a classification layer are added to preliminarily extract features of images in a database and to use a trained hash retrieval model to perform hash coding to obtain a hash code retrieval library; features of an image to be retrieved are preliminarily extracted and the trained hash retrieval model is used to perform hash coding to obtain hash codes of the image to be retrieved; the hash code retrieval library is searched for data with the closest Hamming distance to the hash codes of the image to be retrieved, and the original sample corresponding to the data is displayed as a retrieval result, and the retrieval process is completed; the method has the advantages that the calculation amount is greatly reduced, rich multi-scale feature information can be extracted, the obtained hash codes are efficient and compact, and the image retrieval efficiency is relatively high.

Description

Technical Field

[0001] This invention relates to a large-scale image retrieval method, and more particularly to a large-scale image retrieval hashing method that focuses on the relationships between image blocks. Background Technology

[0002] In recent years, deep neural networks have achieved great success in feature extraction. Therefore, the combination of deep learning and hash learning has led to breakthroughs in hash-based image retrieval methods. Existing deep hashing methods use convolutional kernels to continuously extract abstract high-level features. Theoretically, their receptive field should be able to cover the entire image. However, many studies have shown that their actual receptive field is much smaller than the theoretical receptive field. This is not conducive to fully capturing global image information. Therefore, the features extracted by the convolutional kernels often lack global contextual information of the image, affecting the quality of the hash code. At the same time, although many deep hashing methods consider the similarity relationship of image depth features, they are difficult to maintain the semantic similarity of the original image in Hamming space because they do not make full use of the supervision information of the labels. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a large-scale image retrieval hashing method that focuses on the relationship between image blocks.

[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a large-scale image retrieval hashing method focusing on the relationship between image blocks, comprising the following steps:

[0005] Step 1): Select from the database The training set consists of 10 samples, using Construct a similarity matrix from the category labels of each sample;

[0006] Step 2): Randomly shuffle the training set, and then use the window merging module, the first self-attention calculation module and the second self-attention calculation module to perform preliminary feature extraction on the shuffled training set to obtain the feature map corresponding to each sample. The first self-attention calculation module adopts a window-based multi-head self-attention module, and the second self-attention calculation module adopts a shift window-based multi-head self-attention module.

[0007] Step 3): Construct the hash retrieval model to be trained, and input the feature map after preliminary feature extraction into the hash retrieval model to be trained to obtain the corresponding hash code and classification result;

[0008] Step 4): Define the loss function of the hash retrieval model to be trained, update the hash retrieval model to be trained through the backpropagation algorithm, and obtain the trained hash retrieval model after training is completed;

[0009] Step 5): Perform preliminary feature extraction on the images in the database and use the trained hash retrieval model to hash encode the images after preliminary feature extraction. Finally, define the resulting set as the hash code retrieval library. Perform preliminary feature extraction on the images to be retrieved and use the trained hash retrieval model to hash encode them to obtain the hash code of the images to be retrieved.

[0010] Step 6): Find the data with the closest Hamming distance to the hash code of the image to be retrieved in the hash code retrieval library, and display the original sample corresponding to the data as the retrieval result to complete the retrieval process.

[0011] The specific process of the above steps is as follows:

[0012] Step 1): Select from the database The training set consists of 10 samples. , , express For any sample, denote the corresponding label category matrix as follows: , ,in, express Any tag category, For the number of categories, use Construct a similarity matrix from the class labels of each sample. ,in, express Any two elements and The similarity, 1≤ i ≤ ≤ ,when and Belonging to the same category or more ,otherwise ;

[0013] Step 2)-1: ... Randomly shuffle the windows, set up window splitting and window merging modules, and... The window segmentation module segments the image into non-overlapping window images. Each non-overlapping window image has a window size of 4×4. The non-overlapping window images are then expanded in the channel direction to obtain a feature map with dimensions of 56×56×48.

[0014] Step 2)-2: Construct a first self-attention computation module and a second self-attention computation module. The first self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map. The second self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a shift-window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map.

[0015] Steps 2)-3: Use a linear embedding layer to map the feature map after window segmentation to 96 dimensions. Input the mapped feature map into the multi-head self-attention module based on window and the multi-head self-attention module based on shift window in turn. Output a feature map with a dimension of 56×56×96 and send it to the window merging module.

[0016] Step 2)-4: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the first self-attention calculation module and the second self-attention calculation module in sequence, and the output feature map with a dimension of 28×28×192 is sent to the window merging module again.

[0017] Step 2)-5: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the image processing module formed by the first self-attention calculation module and the second self-attention calculation module three times in sequence. Finally, the output feature map with a dimension of 14×14×384 is sent to the window merging module again.

[0018] Step 2)-6: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the first self-attention calculation module and the second self-attention calculation module in sequence, and the output feature map with a dimension of 7×7×768 is used as the feature map after preliminary feature extraction.

[0019] Step 3): Construct the hash retrieval model to be trained. Input the feature map after preliminary feature extraction into the hash retrieval model to obtain the corresponding hash code and classification result. The specific process is as follows:

[0020] Step 3)-1: Construct the hash retrieval model to be trained, including a pooling layer, a hashing layer, and a classification layer. Pass the feature map after preliminary feature extraction through the pooling layer, and apply global average pooling to obtain a feature vector with a dimension of 768×1. ;

[0021] Step 3)-2: ... The input is processed by the hash layer to obtain the feature vector output by the hash layer. , ,Will Binarization yields the hash code. , ,in, This represents the weight matrix of the hash layer. express transpose, Indicates the bias of the hash layer. It is the hyperbolic tangent function. For symbolic functions, then... The classification results are obtained after the classification layer. , , This represents the weight matrix of the classification layer. Indicates the bias of the classification layer;

[0022] Step 4): Define the loss function of the hash retrieval model to be trained, update the hash retrieval model to be trained through the backpropagation algorithm, and obtain the trained hash retrieval model after training. The specific process is as follows:

[0023] Step 4)-1: Define the loss function of the hash retrieval model to be trained as follows: ,in, , For hyperparameters, For classifying losses, , To quantify the loss, , For hash loss, , , It is a set of similar sample pairs. express The number of elements, It is a set of dissimilar sample pairs. express The number of elements, for transpose;

[0024] Step 4)-2: Set the maximum number of iterations. Based on the loss function, use the AdamW optimization algorithm to iteratively optimize the retrieval model to be trained until the maximum number of iterations is reached. Then stop the iteration process and obtain the trained hash retrieval model.

[0025] Step 5): Perform preliminary feature extraction on the images in the database and use the trained hash retrieval model to hash encode the images after preliminary feature extraction. Finally, define the resulting set as the hash code retrieval library. Perform preliminary feature extraction on the images to be retrieved and use the trained hash retrieval model to hash encode them to obtain the hash code of the images to be retrieved.

[0026] Step 6): Find the data with the closest Hamming distance to the hash code of the image to be retrieved in the hash code retrieval library, and display the original sample corresponding to the data as the retrieval result to complete the retrieval process.

[0027] Compared with existing technologies, the advantages of this invention are as follows: First, multi-scale features of the image are extracted from the sample through a feature extraction network constructed by a first self-attention calculation module and a second self-attention calculation module. The image is divided into several non-overlapping windows, and each window calculates its own self-attention, which greatly reduces the amount of computation. In order to learn global contextual information, window transformation is used to make the windows interconnected to learn global features. Furthermore, the first and second self-attention calculation modules adopt a layered structure of downsampling layer by layer to gradually increase the receptive field, which can extract rich multi-scale feature information. Then, a global average pooling layer is added after the feature extraction network, and additional hashing and classification layers are added. The network is trained under the joint optimization of hash loss, classification loss and quantization loss. By jointly optimizing hash loss, classification loss and quantization loss, feature representation and hash function can be learned from the input image at the same time. The obtained hash code is efficient and compact, thereby effectively improving the accuracy of image retrieval. Detailed Implementation

[0028] The present invention will now be described in further detail.

[0029] A large-scale image retrieval hashing method focusing on the relationships between image patches includes the following steps:

[0030] Step 1): Select from the database The training set consists of 10 samples. , , express For any sample, denote the corresponding label category matrix as follows: , ,in, express Any tag category, For the number of categories, use Construct a similarity matrix from the class labels of each sample. ,in, express Any two elements and The similarity, 1≤ i ≤ ≤ ,when and Belonging to the same category or more ,otherwise ;

[0031] Step 2): Randomly shuffle the training set, and then use the window merging module, the first self-attention calculation module, and the second self-attention calculation module to perform preliminary feature extraction on the shuffled training set to obtain the feature map corresponding to each sample. The first self-attention calculation module adopts a window-based multi-head self-attention module, and the second self-attention calculation module adopts a shift-window-based multi-head self-attention module. The specific process is as follows:

[0032] Step 2)-1: ... Randomly shuffle the windows, set up window splitting and window merging modules, and... The window segmentation module segments the image into non-overlapping window images. Each non-overlapping window image has a window size of 4×4. The non-overlapping window images are then expanded in the channel direction to obtain a feature map with dimensions of 56×56×48.

[0033] Step 2)-2: Construct a first self-attention computation module and a second self-attention computation module. The first self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map. The second self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a shift-window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map.

[0034] Steps 2)-3: Use a linear embedding layer to map the feature map after window segmentation to 96 dimensions. Input the mapped feature map into the multi-head self-attention module based on window and the multi-head self-attention module based on shift window in turn. Output a feature map with a dimension of 56×56×96 and send it to the window merging module.

[0035] Steps 2)-4: The window merging module merges the received feature maps to obtain a merged feature map. The merged feature map is then sequentially input into the first self-attention calculation module and the second self-attention calculation module, outputting a feature map with a dimension of 28×28×192, which is then sent back to the window merging module. The window merging module's merging of the received feature maps is an existing technique used to segment and recombine the original feature map according to a specific rule, arranging it along the channel dimension to change the original feature map's dimension to the desired dimension.

[0036] Step 2)-5: The window merging module merges the received feature maps to obtain the merged feature map, and then inputs the merged feature map into the image processing module formed by the first self-attention calculation module and the second self-attention calculation module three times in sequence;

[0037] Step 2)-6: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the first self-attention calculation module and the second self-attention calculation module in sequence, and the output feature map with a dimension of 7×7×768 is used as the feature map after preliminary feature extraction.

[0038] Step 3): Construct the hash retrieval model to be trained. Input the feature map after preliminary feature extraction into the hash retrieval model to obtain the corresponding hash code and classification result. The specific process is as follows:

[0039] Step 3)-1: Construct the hash retrieval model to be trained, including a pooling layer, a hashing layer, and a classification layer. Pass the feature map after preliminary feature extraction through the pooling layer, and apply global average pooling to obtain a feature vector with a dimension of 768×1. ;

[0040] Step 3)-2: ... The input is processed by the hash layer to obtain the feature vector output by the hash layer. , ,Will Binarization yields the hash code. , ,in, This represents the weight matrix of the hash layer. express transpose, Indicates the bias of the hash layer. It is the hyperbolic tangent function. For symbolic functions, then... The classification results are obtained after the classification layer. , , This represents the weight matrix of the classification layer. Indicates the bias of the classification layer;

[0041] Step 4): Define the loss function of the hash retrieval model to be trained, update the hash retrieval model to be trained through the backpropagation algorithm, and obtain the trained hash retrieval model after training. The specific process is as follows:

[0042] Step 4)-1: Define the loss function of the hash retrieval model to be trained as follows: ,in, , For hyperparameters, For classifying losses, , To quantify the loss, , For hash loss, , , It is a set of similar sample pairs. express The number of elements, It is a set of dissimilar sample pairs. express The number of elements, for transpose;

[0043] Step 4)-2: Set the maximum number of iterations. Based on the loss function, use the AdamW optimization algorithm to iteratively optimize the retrieval model to be trained until the maximum number of iterations is reached. Then stop the iteration process and obtain the trained hash retrieval model.

[0044] Step 5): Perform preliminary feature extraction on the images in the database and use the trained hash retrieval model to hash encode the images after preliminary feature extraction. Finally, define the resulting set as the hash code retrieval library. Perform preliminary feature extraction on the image to be retrieved and use the trained hash retrieval model to hash encode it to obtain the hash code of the image to be retrieved. The preliminary feature extraction process here is the same as in steps 2)-1 to 2)-6, only the target is different.

[0045] Step 6): Find the data with the closest Hamming distance to the hash code of the image to be retrieved in the hash code retrieval library, and display the original sample corresponding to the data as the retrieval result to complete the retrieval process.

Claims

1. A large-scale image retrieval hashing method focusing on the relationships between image blocks, characterized in that... Includes the following steps: Step 1): Select from the database The training set consists of 10 samples, using Construct a similarity matrix from the category labels of each sample; Step 2): Randomly shuffle the training set, and then use the window merging module, the first self-attention calculation module and the second self-attention calculation module to perform preliminary feature extraction on the shuffled training set to obtain the feature map corresponding to each sample. The first self-attention calculation module adopts a window-based multi-head self-attention module, and the second self-attention calculation module adopts a shift window-based multi-head self-attention module. Step 3): Construct the hash retrieval model to be trained, and input the feature map after preliminary feature extraction into the hash retrieval model to be trained to obtain the corresponding hash code and classification result; Step 4): Define the loss function of the hash retrieval model to be trained, update the hash retrieval model to be trained through the backpropagation algorithm, and obtain the trained hash retrieval model after training is completed; Step 5): Perform preliminary feature extraction on the images in the database and use the trained hash retrieval model to hash encode the images after preliminary feature extraction. Finally, define the resulting set as the hash code retrieval library. Perform preliminary feature extraction on the images to be retrieved and use the trained hash retrieval model to hash encode them to obtain the hash code of the images to be retrieved. Step 6): Find the data with the closest Hamming distance to the hash code of the image to be retrieved in the hash code retrieval library, and display the original sample corresponding to the data as the retrieval result to complete the retrieval process.

2. The large-scale image retrieval hashing method focusing on the relationship between image blocks according to claim 1, characterized in that... The specific process is as follows: Step 1): Select from the database The training set consists of 10 samples. , , express For any sample, denote the corresponding label category matrix as follows: , ,in, express Any tag category, For the number of categories, use Construct a similarity matrix from the class labels of each sample. ,in, express Any two elements and The similarity, 1≤ i ≤ ≤ ,when and Belonging to the same category or more ,otherwise ; Step 2)-1: ... Randomly shuffle the windows, set up window splitting and window merging modules, and... The window segmentation module segments the image into non-overlapping window images. Each non-overlapping window image has a window size of 4×4. The non-overlapping window images are then expanded in the channel direction to obtain a feature map with dimensions of 56×56×48. Step 2)-2: Construct a first self-attention computation module and a second self-attention computation module. The first self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map. The second self-attention computation module is used to feed the input feature map into an LN layer for normalization, and then input the normalized feature map into a shift-window-based multi-head self-attention module. Subsequently, the output feature map and the initially input feature map are residually concatenated, and the result is input into a two-layer multilayer perceptron with a GELU nonlinear activation function to obtain the output feature map. Steps 2)-3: Use a linear embedding layer to map the feature map after window segmentation to 96 dimensions. Input the mapped feature map into the multi-head self-attention module based on window and the multi-head self-attention module based on shift window in turn. Output a feature map with a dimension of 56×56×96 and send it to the window merging module. Step 2)-4: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the first self-attention calculation module and the second self-attention calculation module in sequence, and the output feature map with a dimension of 28×28×192 is sent to the window merging module again. Step 2)-5: The window merging module merges the received feature maps to obtain the merged feature map, and then inputs the merged feature map into the image processing module formed by the first self-attention calculation module and the second self-attention calculation module three times in sequence; Step 2)-6: The window merging module merges the received feature maps to obtain the merged feature map. Then, the merged feature map is input into the first self-attention calculation module and the second self-attention calculation module in sequence, and the output feature map with a dimension of 7×7×768 is used as the feature map after preliminary feature extraction. Step 3): Construct the hash retrieval model to be trained. Input the feature map after preliminary feature extraction into the hash retrieval model to obtain the corresponding hash code and classification result. The specific process is as follows: Step 3)-1: Construct the hash retrieval model to be trained, including a pooling layer, a hashing layer, and a classification layer. Pass the feature map after preliminary feature extraction through the pooling layer, and apply global average pooling to obtain a feature vector with a dimension of 768×1. ; Step 3)-2: ... The input is processed by the hash layer to obtain the feature vector output by the hash layer. , ,Will Binarization yields the hash code. , ,in, This represents the weight matrix of the hash layer. express transpose, Indicates the bias of the hash layer. It is the hyperbolic tangent function. For symbolic functions, then... The classification results are obtained after the classification layer. , , This represents the weight matrix of the classification layer. Indicates the bias of the classification layer; Step 4): Define the loss function of the hash retrieval model to be trained, update the hash retrieval model to be trained through the backpropagation algorithm, and obtain the trained hash retrieval model after training. The specific process is as follows: Step 4)-1: Define the loss function of the hash retrieval model to be trained as follows: ,in, , For hyperparameters, For classifying losses, , To quantify the loss, , For hash loss, , , It is a set of similar sample pairs. express The number of elements, It is a set of dissimilar sample pairs. express The number of elements, for transpose; Step 4)-2: Set the maximum number of iterations. Based on the loss function, use the AdamW optimization algorithm to iteratively optimize the retrieval model to be trained until the maximum number of iterations is reached. Then stop the iteration process and obtain the trained hash retrieval model. Step 5): Perform preliminary feature extraction on the images in the database and use the trained hash retrieval model to hash encode the images after preliminary feature extraction. Finally, define the resulting set as the hash code retrieval library. Perform preliminary feature extraction on the images to be retrieved and use the trained hash retrieval model to hash encode them to obtain the hash code of the images to be retrieved. Step 6): Find the data with the closest Hamming distance to the hash code of the image to be retrieved in the hash code retrieval library, and display the original sample corresponding to the data as the retrieval result to complete the retrieval process.

Images