Hash retrieval method based on spatial crossover shift and Fourier frequency domain learning

Abstract

The invention discloses a Hash retrieval method based on spatial crossover shift and Fourier frequency domain learning, and mainly relates to the field of deep Hash image retrieval. Comprising the following steps: S1, respectively and correspondingly dividing an image data set and category label information corresponding to the image data set into a training set, a query set and a database; s2, preprocessing the image information; s3, constructing a Transform deep hash retrieval architecture based on spatial cross shift and Fourier frequency domain learning; s4, starting to train the images in batches in turn; s5, randomly inputting a query set, generating query Hash codes through a network, calculating Hamming space distances from the query Hash codes to database Hash codes, sorting the distances from small to large, and finding corresponding image groups according to previous several returned Hash code results; according to the method, the high-level semantic expression of the deep features can be obtained, interactive learning between different domains can be promoted, the semantic information accuracy can be improved, and the robustness of the model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep hash image retrieval, and specifically to a hash retrieval method based on spatial cross-shift and Fourier frequency domain learning. Background Art

[0002] The development of multimedia technology has resulted in tens of thousands of graphic and text data pouring into various storage systems every day. Finding similar images has become an urgent need in daily life. Among them, hash image retrieval has gradually occupied the field of image retrieval due to its advantages of high computational efficiency and high accuracy. Its core mechanism involves using hash functions to map the original high-dimensional, difficult-to-process complex continuous data into low-dimensional, simple, discrete binary coded data. By measuring the distance between these discrete data in the Hamming space, the similarity between images is determined, thereby retaining the similarity structure of the original data in the compressed space.

[0003] With the continuous development of deep learning technology, many excellent hash image retrieval algorithms have gradually surfaced. They have made progress to a certain extent. However, these algorithms often only focus on obtaining global or local features of target categories from the spatial domain, ignoring the influence of the frequency domain on the semantic information of image channels. There are still limitations, which makes it difficult for the network to cope with images with diverse targets and high complexity. Secondly, the information interaction between different domains determines the high-level semantic expressions that the network can obtain, which has not been fully explored. Finally, the hash function represents a mapping relationship from continuous to discrete, and in this process, it is easy to cause information loss. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and to provide a hash retrieval method based on spatial cross-shift and Fourier frequency domain learning, which can obtain high-level semantic expressions of deep features, promote interactive learning between different domains to improve the accuracy of semantic information, and improve the robustness of the model.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A hash retrieval method based on spatial cross-shift and Fourier frequency domain learning comprises the following steps:

[0007] S1. Divide the image data set and the category label information corresponding to the image data set into: a training set, a query set and a database;

[0008] S2, preprocess the image information, crop each input image into the same pixel size, and feed it into the model;

[0009] S3. Construct a Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning, which includes a CAFormer feature extraction backbone, a spatial-spectral dual-domain shift perception module, a hash learning module, and a loss function. The CAFormer feature extraction backbone and the spatial-spectral dual-domain shift perception module are used to learn the rich semantic features of image target categories; the hash learning module is used to discretize continuous feature vectors through hash mapping to obtain a hash code representing the input image; the loss function is used to guide the optimization direction of the hash function;

[0010] S4, start training images in batches, take M small batches of images in the training set, train according to the Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning in step S3, and use the loss for back-propagation update optimization, and save the best hash retrieval model weights during the training process;

[0011] S5. Randomly input a query set, generate a query hash code through the network, calculate the Hamming space distance with the database hash code, sort the distances from small to large, and find the corresponding image group based on the first few returned hash code results.

[0012] Preferably, in step S3, CAFormer feature extraction backbone and spatial-spectral dual-domain shift perception module are used as feature extraction, and the output of each stage is: Model training is performed through CAFormer-s18 pre-training.

[0013] Preferably, in step S3, the workflow of the space-spectrum dual-domain shift perception module includes:

[0014] First, the input feature map Fi is shifted up and down as a whole, multiple position distribution relationships are established in the feature tensor, and the feature dimension is restored by filling '0' to obtain two different spatial shift information representations F ↑ and F ↓ , and is fused with the initial input information to promote spatial position feature learning. Assuming that F represents the input of this module, it is specifically expressed as:

[0015] F ↑ , F ↓ =shift(R(F))

[0016] F1=F ↑ +F ↓ +R(F)

[0017] Among them, R represents the dimension reshaping operation, and shift represents the feature movement operation;

[0018] Then, the fused learned position features are sent to two branches, and the average pooling of H×1 and 1×W is used to aggregate the spatial information in height and width, respectively, to capture the long-distance dependencies of this information in different spatial dimensions, and obtain two bar feature maps F W and F H , and then transpose along the two dimensions H and W to obtain two other bar features F W′ and F H′ , the implementation process is expressed as:

[0019] F W F H =AvgPool2d(F1)

[0020] F H′ , F W′ =T(F W ), T(F H )

[0021] Among them, AvgPool2d represents the 2D average pooling operation, and T represents transpose;

[0022] Next, the bar feature maps of the same size are summed element by element to enhance the semantic richness of the features, which is specifically expressed as:

[0023] F2=expand(F H +F H′ )+expand(F W +F W′ )

[0024] Among them, expand means to expand the bar feature.

[0025] Preferably, in step S3, Fourier transform is introduced in the space-spectrum dual-domain shift perception module to convert spatial information into frequency domain information, and the category information of the frequency domain attention channel is used, and the corresponding weights of the feature information adaptive learning are optimized by weighting, so that the frequency domain features are close to the image category and far away from other categories; the frequency domain is then converted back to the spatial domain through inverse Fourier transform and fused with the input information in a residual manner to obtain a more detailed semantic feature representation, which is sent to the next stage. The adaptive frequency domain learning of this part is specifically expressed as:

[0026]

[0027] Among them, R represents the dimension reshaping operation, represents the Fourier transform, represents the learnable weight matrix, represents the inverse Fourier transform, and F is the input of the spatial-spectral dual-domain shift perception module.

[0028] Preferably, in step S3, the loss function includes center loss, quantization loss and bit decoupling loss, and the final loss function is:

[0029] L=L C +λL Q +αL D

[0030] Among them, λ is a trade-off hyperparameter for balancing quantization loss, α is a trade-off hyperparameter for balancing bit decoupling loss, and L C is the center loss, L Q is the quantization loss, L D is the bit decoupling loss.

[0031] Preferably, the center loss is expressed as follows:

[0032]

[0033] Among them, b i represents the hash code of the i-th sample in the image sample set after hash mapping, C i represents the central hash code of the i-th sample, and M is the number of images.

[0034] Preferably, the quantization loss is expressed as follows:

[0035]

[0036] Preferably, the bit decoupling loss is expressed as follows:

[0037]

[0038] in, is the penalty loss coefficient, which is used to adjust the size of the loss. I represents the L×L unit matrix; mean represents the calculation of the average similarity of this matrix.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] 1. The present invention combines spatial shift and frequency domain learning, and proposes a hash image retrieval method based on spatial cross shift and Fourier frequency domain learning to efficiently perceive visual Transformer features. The framework is mainly based on CAFormer, supplemented by the spatial-spectral dual-domain shift perception module, which helps the model learn rich semantic representations. In addition, the present invention designs a bit decoupling loss to make each bit independent of each other and enhance the discrimination ability of hash codes.

[0041] 2. The present invention proposes a unified deep hash end-to-end retrieval framework, which applies a four-stage network. Each stage adopts the fixed structure of CAFormer and follows the spatial-spectral dual-domain shift perception module to further enhance the features and obtain the high-level semantic expression of deep features.

[0042] 3. The present invention comprehensively considers the perception ability of spatial position and frequency domain for visual elements, embeds a spatial-spectral dual-domain shift perception module, establishes the spatial position relationship distribution, and further focuses on the target category characteristics through the spectrum, allowing the model to pay attention to both spatial features and frequency domain features, promoting interactive learning between different domains to improve the accuracy of semantic information.

[0043] 4. The present invention application proposes a bit decoupling loss constraint to reduce the correlation between the hash code bits output, and adds it to the overall loss auxiliary model optimization learning, so as to generate easily distinguishable and representative hash binary discrete codes while retaining the similarity information in the original feature space as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall framework diagram of the present invention;

[0045] Figure 2 It is a framework diagram of the spatial-spectral dual-domain shift perception module. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0047] Embodiment: The present invention is a hash retrieval method based on spatial cross-shift and Fourier frequency domain learning, and the specific implementation steps are as follows:

[0048] Step 1: Divide the image dataset and its corresponding category label information into three parts: training set, query set and database;

[0049] Step 2: Preprocess the image information, crop each input image into a size of 224*224 pixels, and feed it into the model;

[0050] Step 3: Construct a Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning, which includes a CAFormer (using CAFormer-s18 pre-trained weights) feature extraction backbone, a spatial-spectral dual-domain shift perception module, a hash learning module, and a loss function (including center loss, quantization loss, and bit decoupling loss). The CAFormer feature extraction backbone and the spatial-spectral dual-domain shift perception module are used to learn the rich semantic features of image target categories. The hash layer is used to discretize continuous feature vectors through hash mapping to obtain a hash code representing the input image. The loss function is used to guide the optimization direction of the hash function.

[0051] Step 4: Start training images in batches, take M small batches of images from the training image set, train according to the Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning, and use loss to perform back propagation update optimization, and save the best hash retrieval model weight of the present invention during the training process;

[0052] Step 5: Randomly input a query set and generate a query hash code through the network. Calculate the Hamming space distance with the database hash code, sort the distances from small to large, and find the corresponding image group based on the first few returned hash code results.

[0053] The Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning in step 3 above is as follows Figure 1 As shown, the backbone network of CAFormer and the spatial-spectral dual-domain shift perception module are first used as feature extraction, and the output of each stage is: The model is trained through CAFormer-s18 pre-training to achieve hybrid semantic feature attention in spatial and frequency domains, reduce interference from image background and irrelevant target objects, emphasize the prominent areas of semantic information, and efficiently capture the target semantic information in feature extraction, thereby providing sufficient information for the network to generate distinctive binary hash codes.

[0054] The present invention adopts the spatial-spectral dual-domain shift perception module to learn the feature maps generated at each stage, and makes it easier for the model to understand the features of the same category images at different positions to reduce the interference of irrelevant background and non-target areas, such as Figure 2 First, the module moves the input feature map Fi up and down as a whole, establishes multiple position distribution relationships in the feature tensor, fills the restored feature dimension with '0', and obtains two different spatial shift information representations F ↑ and F ↓, and is fused with the initial input information to promote spatial position feature learning. Assuming that F represents the input of this module, it is specifically expressed as:

[0055] F ↑ , F ↓ =shift(R(F))

[0056] F1=F ↑ +F ↓ +R(F)

[0057] Among them, R represents the dimension reshaping operation, and shift represents the feature movement operation;

[0058] Then, the fused learned position features are sent to two branches, and the average pooling of H×1 and 1×W is used to aggregate the spatial information in height and width, respectively, to capture the long-distance dependencies of this information in different spatial dimensions, and obtain two bar feature maps F W and F H , and then transpose along the two dimensions H and W to obtain two other bar features F W′ and F H′ , the implementation process is expressed as:

[0059] F W F H =AvgPool2d(F1)

[0060] F H′ , F W′ =T(F W ), T(F H )

[0061] Among them, AvgPool2d represents the 2D average pooling operation, and T represents transpose;

[0062] Next, the bar feature maps of the same size are summed element by element to enhance the semantic richness of the features, which is specifically expressed as:

[0063] F2=expand(F H +F H′ )+expand(F W +F W′ )

[0064] Among them, expand means to expand the bar feature.

[0065] When there is a situation in the image where the current label category is similar to other categories, the network may cause recognition errors due to the small difference between categories. The category information of these targets often exists in the channel. Therefore, the present invention introduces Fourier transform in the spatial-spectral dual-domain shift perception module to convert spatial information into frequency domain information, and uses the frequency domain to focus on the category information of the channel, and optimizes the corresponding weights of the adaptive learning of feature information by weighting, so that the frequency domain features are close to the image category and away from other categories. The frequency domain is then converted back to the spatial domain through the inverse Fourier transform and fused with the input information in a residual manner to obtain a more detailed semantic feature representation, which is sent to the next stage, promoting interactive learning between different domains, helping the model learn the differences between categories, reducing the occurrence of misjudgments, and laying the foundation for generating more reliable potential hash codes. The adaptive frequency domain learning of this part is specifically expressed as:

[0066]

[0067] Among them, R represents the dimension reshaping operation, represents the Fourier transform, represents the learnable weight matrix, represents the inverse Fourier transform, and F is the input of the spatial-spectral dual-domain shift perception module.

[0068] The spatial-spectral dual-domain shift perception module analyzes the spatial position and category spectrum information, understands and learns the dual-domain information representation, improves the semantic accuracy, clusters the semantics of similar images closer, and provides sufficient information for the network to generate distinctive binary hash codes.

[0069] Loss function:

[0070] Definition: The image dataset of N samples is x i ∈R d represents the i-th image in the dataset, d represents the image dimension, and y i ∈(0,1) C Represents image x i The corresponding semantic label vector, C is the total number of categories, when y i =1, indicating x i Belongs to the current y i Corresponding to this category, on the contrary, x i If it does not belong to this category i = 0. The L-bit compact hash code that can represent high-dimensional semantic features in the Hamming space is learned through hash mapping.

[0071] The loss function is an important factor for the hash code to effectively reflect the similarity relationship between images. The present invention integrates the existing center loss, quantization loss and the proposed bit decoupling loss to optimize the generation of hash codes.

[0072] Bit decoupling loss: In order to make the generated hash codes maintain better similarity between the same category and have better distinguishability between different categories, the present invention proposes a bit decoupling loss, which constrains each bit of the hash code to make the bits independent of each other, reduce redundancy, and improve the model's ability to learn binary codes. To achieve this goal, the present invention considers using the correlation relationship between labels to achieve the independence of hash codes. If the correlation between labels is higher, the code will more strongly penalize the correlation between hash codes. We measure the correlation between label vectors through the autocorrelation matrix between label vectors.

[0073] Specifically, given the true label vectors of M mini-batch images, these M label vectors are stacked into a matrix Y = (y1, y2, ..., y i ,...,y M )∈R M×C Since each row of Y represents the label vector of a different image, the present invention calculates the mutual correlation coefficients between them through Y to obtain a similarity weight matrix Each element in [i,j] represents the dot product between the label vector of the i-th image and the label vector of the j-th image. The larger the dot product, the more similar the two images are, and therefore the higher the correlation between them. The average value of this weight matrix is ​​taken to get the average similarity between all image labels as a whole.

[0074]

[0075] Among them, mean means calculating the average similarity of this matrix.

[0076] On the other hand, the hash codes generated by the hash network for the M images are stacked into a matrix B = (b1, b2, ..., b i ,...,b M )∈R M×L , the matrix is ​​used to improve the quality of the hash code. Ideally, the hash codes should be independent of each other, which means that the covariance matrix obtained from the hash code matrix should be close to the identity matrix, that is, the diagonal elements are close to 1 and the off-diagonal elements are close to 0. Therefore, the present invention constructs the decoupling loss by optimizing the difference between the hash code covariance matrix and the identity matrix. The smaller the difference, the more independent the hash codes are.

[0077]

[0078] in, is the penalty loss coefficient, which is used to adjust the size of the loss, and I represents the L×L unit matrix.

[0079] The loss in the present invention utilizes the independence between the hash code matrix B and the label matrix Y, which can avoid the problem that the hash codes generated by different images are too similar, enhance the recognition ability of the image retrieval system, and ensure that even if the labels of the images are similar, the generated hash codes can effectively distinguish different images, thereby improving the optimization problem of the hash codes and improving the accuracy of the retrieval results.

[0080] Center loss: Image retrieval datasets are divided into single-label and multi-label. Multi-label image categories are numerous and complex, which leads to the central similarity metric, which aims to cluster the category hash centers of multi-target images into a hash centroid through weight distribution. The hash centroid of a single-target image is the hash center itself. Fast retrieval is completed by calculating the Hamming distance between the hash code bi and the hash centroid. In this process, the optimization process of the center loss is defined as:

[0081]

[0082] Among them, b i represents the hash code of the i-th sample in the image sample set after hash mapping, C i represents the central hash code of the i-th sample, and M is the number of images.

[0083] Quantization loss: In order to increase the discreteness of the hash code and constrain the output of the network to be close to the true values ​​+1 and -1, we perform quantization on the continuous hash code u in the output part of the network. i Applying tanh for quantization inevitably results in an error between the output and the true value. The present invention continues the existing L2 norm to reduce the difference between the quantized hash code and the true binary code, improves the consistency of similar semantics, and enables the network to generate high-quality binary codes. The quantization loss is expressed as follows:

[0084]

[0085] The present invention combines the above bit decoupling loss, center loss and quantization loss to jointly optimize the network, and the final loss of the entire deep hash retrieval framework is:

[0086] L=L C +λL Q +αL D

[0087] Among them, λ is a trade-off hyperparameter for balancing quantization loss, and α is a trade-off hyperparameter for balancing bit decoupling loss.

[0088] In order to fully explore the potential connection between spatial information and frequency domain information, the present invention focuses on the position distribution of spatial information and further processes it through the spectrum, and proposes a decoupling loss that can promote the independence of bits. This method reduces the situation where the discrete values ​​output by the model are too similar by penalizing the correlation between labels, effectively ensuring that the binary output of the hash code is more discriminative.

[0089] The present invention adopts CAFormer as the backbone of information extraction, because its excellent progressive pyramid learning method can provide the characteristics of images at different resolutions, and the hash code generated in hash retrieval is guaranteed. However, although the category features can be extracted in this way, there is a mapping burden between the features and the hash code, and no interaction is performed between different domains. The influence of the frequency domain on the semantic information of the image channel is ignored, and the problem of easily losing distinguishing information still exists. In order to solve this problem, the present invention distributes the features extracted by the backbone network along the spatial cross-shift learning position, and uses Fourier to perform category perception brought by the target features. Because of its excellent mixed attention in the spatial domain and frequency domain, it can fully enable the interactive learning of features between different domains, reduce interference from the image background and irrelevant target objects, emphasize the prominent area of ​​the semantic information of attention, and efficiently capture the target semantic information in feature extraction. In this way, if you want to improve the quality of hash retrieval, it is more important to grasp the independence between bits and reduce redundancy. Therefore, the present invention proposes a bit decoupled loss target to adjust the properties of the hash code. The loss optimizes the bit-independent constraints to reduce the bit correlation, generate different bits independent of each other, and easily distinguish the best hash code, thereby improving the robustness of the model.

Claims

1. A hash retrieval method based on spatial cross-shift and Fourier frequency domain learning, characterized in that: Includes steps: S1. Divide the image data set and the category label information corresponding to the image data set into: a training set, a query set and a database; S2, preprocess the image information, crop each input image into the same pixel size, and feed it into the model; S3. Construct a Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning, which includes a CAFormer feature extraction backbone, a spatial-spectral dual-domain shift perception module, a hash learning module, and a loss function. The CAFormer feature extraction backbone and the spatial-spectral dual-domain shift perception module are used to learn the rich semantic features of image target categories; the hash learning module is used to discretize continuous feature vectors through hash mapping to obtain a hash code representing the input image; the loss function is used to guide the optimization direction of the hash function; S4, start training images in batches, take M small batches of images in the training set, train according to the Transformer deep hash retrieval architecture based on spatial cross-shift and Fourier frequency domain learning in step S3, and use the loss for back-propagation update optimization, and save the best hash retrieval model weights during the training process; S5. Randomly input a query set, generate a query hash code through the network, calculate the Hamming space distance with the database hash code, sort the distances from small to large, and find the corresponding image group based on the first few returned hash code results.

2. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 1 is characterized in that: In step S3, the CAFormer feature extraction backbone and the spatial-spectral dual-domain shift perception module are used as feature extraction, and the output of each stage is: Model training is performed through CAFormer-s18 pre-training.

3. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 1 is characterized in that: In step S3, the workflow of the spatial-spectral dual-domain shift perception module includes: First, the input feature map Fi is shifted up and down as a whole, multiple position distribution relationships are established in the feature tensor, and the feature dimension is restored by filling '0' to obtain two different spatial shift information representations F ↑ and F ↓ , and is fused with the initial input information to promote spatial position feature learning. Assuming that F represents the input of this module, it is specifically expressed as: F ↑ ,F ↓ =shift(R(F)) F1=F ↑ +F ↓ +R(F) Among them, R represents the dimension reshaping operation, and shift represents the feature movement operation; Then, the fused learned position features are sent to two branches, and the average pooling of H×1 and 1×W is used to aggregate the spatial information in height and width, respectively, to capture the long-distance dependencies of this information in different spatial dimensions, and obtain two bar feature maps F W and F H , and then transpose along the two dimensions H and W to obtain two other bar features F W′ and F H′ , the implementation process is expressed as: F W ,F H =AvgPool2d(F1) F H′ ,F W′ =T(F W ),T(F H ) Among them, AvgPool2d represents the 2D average pooling operation, and T represents transpose; Next, the bar feature maps of the same size are summed element by element to enhance the semantic richness of the features, which is specifically expressed as: F2=expand(F H +F H′ )+expand(F W +F W′ ) Among them, expand means to expand the bar feature.

4. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 3 is characterized in that: In step S3, Fourier transform is introduced into the spatial-spectral dual-domain shift perception module to convert spatial information into frequency domain information, and the category information of the frequency domain attention channel is used. The corresponding weights of the adaptive learning of feature information are optimized by weighting, so that the frequency domain features are close to the image category and far away from other categories. The frequency domain is then converted back to the spatial domain through inverse Fourier transform and fused with the input information in a residual manner to obtain a more detailed semantic feature representation, which is sent to the next stage. The adaptive frequency domain learning of this part is specifically expressed as: Among them, R represents the dimension reshaping operation, represents the Fourier transform, represents the learnable weight matrix, represents the inverse Fourier transform, and F is the input of the spatial-spectral dual-domain shift perception module.

5. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 1 is characterized in that: In step S3, the loss function includes center loss, quantization loss and bit decoupling loss, and the final loss function is: L=L C +λL Q +αL D Among them, λ is a trade-off hyperparameter for balancing quantization loss, α is a trade-off hyperparameter for balancing bit decoupling loss, and L C is the center loss, L Q is the quantization loss, L D is the bit decoupling loss.

6. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 5, characterized in that: The center loss is expressed as follows: Among them, b i represents the hash code of the i-th sample in the image sample set after hash mapping, C i represents the central hash code of the i-th sample, and M is the number of images.

7. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 5, characterized in that: The quantization loss is expressed as follows:

8. The hash retrieval method based on spatial cross-shift and Fourier frequency domain learning according to claim 5 is characterized in that: The bit decoupling loss is expressed as follows: in, is the penalty loss coefficient, which is used to adjust the size of the loss. I represents the L×L unit matrix; mean represents the calculation of the average similarity of this matrix.

Images