Remote sensing image retrieval optimization method based on automatic weight distribution and comparison hash

Abstract

The invention provides a remote sensing image retrieval optimization method based on automatic weight distribution and comparative Hash, a comparative Hash network model based on automatic weight distribution is constructed, and the performance of the model is optimized by adopting a double-stage training strategy. In the feature learning stage, automatic weighting comparison loss is provided, Gaussian weighting and a dynamic adjustment strategy are introduced to improve a traditional loss function, different weight values are given according to the importance between sample pairs, and learning of the model on key sample pairs is enhanced; a threshold value is set to identify the difficult-to-negative sample, so that the model is not interfered by the difficult-to-negative sample. In the Hash learning stage, a Hash layer is added to convert high-dimensional image representation into binary Hash codes, and quantization loss is introduced to learn the spliced and fused Hash codes, so that a semantic similar structure among data in a Hamming space can be kept, and meanwhile, the identification of the Hash codes is improved by utilizing automatic weighting comparison loss. According to the method, the target remote sensing image can be effectively retrieved, and the retrieval speed is relatively high.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and remote sensing image processing, and in particular to a remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing, which aims to improve the efficiency and accuracy of remote sensing image retrieval. Background Art

[0002] With the development of earth observation technology, the number of remote sensing images has shown an explosive growth trend. How to retrieve target images from a huge number of remote sensing databases has become a current research hotspot. Remote sensing image retrieval can be mainly divided into text-based remote sensing image retrieval and content-based remote sensing image retrieval. Text-based remote sensing image retrieval draws on text search technology (Text-based Image Retrieval, TBIR). The TBIR method requires a text description of the image and then searches for the target retrieval image based on the text description. However, this method requires a lot of human resources to annotate the data, and the image information description ability is insufficient, which makes the TBIR method difficult to apply to today's massive remote sensing image data scenarios. The content-based image retrieval method (Content-Based Image Retrieval, CBIR) focuses on understanding the characteristics of the image itself, using the content features of the image as an index to build an image database, and then realize retrieval. Compared with the TBIR method, the CBIR method is more adaptable to the needs of modern remote sensing image retrieval.

[0003] Depending on whether a supervisory signal is needed during the network training process, CBIR methods can be divided into supervised and unsupervised methods. In supervised learning, building annotated datasets is a time-consuming and labor-intensive task. Especially for remote sensing image datasets, the annotation process is highly dependent on manual labor and expertise, which requires a lot of time and economic costs. In order to overcome the limitations of traditional supervised learning, self-supervised learning as an unsupervised learning method has received widespread attention. Among them, contrastive learning is an important technology. The core idea is to bring similar samples closer and push dissimilar samples away, so as to learn discriminative features. Contrastive learning methods can be roughly divided into negative sample-based methods, clustering-based methods, asymmetric network-based methods, and information maximization-based methods.

[0004] Although contrastive learning can effectively learn discriminative features, there are three main difficulties when applied to remote sensing image retrieval tasks: First, the weight distribution problem. The standard contrast loss usually regards all sample pairs as equally important without considering the differences between samples. Second, the accuracy problem of positive and negative samples. The traditional construction method defines different views of the same image as positive sample pairs and views of different images as negative sample pairs. This construction method will identify images belonging to the same category as negative samples, which may cause the model to learn common features of different categories and ignore the significant similarities between different images in the same category, thereby reducing the model's discrimination accuracy. Third, the high-dimensional feature problem. Although the contrastive learning method can learn high-quality features, the high-dimensional deep features obtained often cause "dimensionality disaster" and also reduce retrieval efficiency. Therefore, how to effectively solve the above difficulties needs to be studied urgently. Summary of the invention

[0005] In view of this, the present invention provides a remote sensing image retrieval optimization method based on automatic weight allocation and contrast hashing. Compared with the existing remote sensing image retrieval method, the method of the present invention uses contrast learning to effectively solve the problem of dependence on image labels. The commonly used contrast loss usually regards all sample pairs as equally important without considering the differences between samples. The present invention proposes automatic weighted contrast loss, which Gaussian weights the similarities between sample pairs, strengthens the model's learning of key sample pairs, further brings positive sample pairs closer, and pushes negative sample pairs away. At the same time, the weight value is dynamically adjusted to reduce the interference of difficult negative samples on model training. Finally, the hash algorithm can be used to map high-dimensional features into discrete hash codes to improve retrieval efficiency.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows.

[0007] A remote sensing image retrieval optimization method based on automatic weight allocation and contrast hashing includes the following steps:

[0008] S1: constructing a comparative hash network model based on automatic weight allocation, wherein the comparative hash network model based on automatic weight allocation mainly includes a feature learning stage and a hash learning stage;

[0009] S11: The feature learning stage includes three components: data enhancement module, encoder module, and automatic weighted contrast loss. The data enhancement module is used to obtain remote sensing image data and obtain enhanced views of each input image. The encoder module is used to extract image features of depth partitions from the enhanced views. The automatic weighted contrast loss is calculated by dynamically adjusting the weight factor to train and adjust the network model. Specifically, the similarity matrix and weight matrix are obtained by calculating the similarity and weight factor between image features, and the weight value of the difficult negative sample is dynamically adjusted. The automatic weighted contrast loss is calculated by the similarity matrix and the dynamically adjusted weight matrix, and then the automatic weighted contrast loss is used to train and adjust the network model in the feature learning stage to obtain fine-tuned network model parameters.

[0010] S12: The hash learning stage shares the data enhancement module and encoder module with the feature learning stage, and also includes a hash layer and a joint loss. In the hash learning stage, the fine-tuned network model parameters obtained in the feature learning stage are first loaded, and the image is input into the trained and parameter-adjusted network model for feature extraction to obtain optimized image features and embedding vectors. The hash layer converts the optimized embedding vector into a continuous hash code, and uses the continuous hash code to calculate the optimized automatic weighted contrast loss. The continuous hash code is then quantized into a binary hash code to obtain two sets of discrete binary hash code sets. The quantization loss in the quantization process is calculated based on the two sets of binary hash codes obtained, and the combined total loss function is calculated through the optimized automatic weighted contrast loss and quantization loss. The network model is trained and learned using the total loss function to obtain the trained network model parameters. The hash learning stage is used to learn the concatenated and fused hash codes to maintain the semantic similarity structure between data.

[0011] S2: Apply the trained network model to remote sensing image retrieval, query the remote sensing dataset image that is most similar to the query image through the network model, and output the retrieval result.

[0012] Furthermore, each remote sensing image data generates two enhanced views with different perspectives after passing through the data enhancement module. The methods of enhancing image data by the data enhancement module include random cropping, scaling, horizontal flipping, random color jittering, random grayscale and random Gaussian blur.

[0013] Furthermore, the encoder module includes a backbone network and a nonlinear projector, the backbone network is a ResNet50 model, and the backbone network, i.e., the ResNet50 model, includes an input layer, an initial convolutional layer, a residual block, a global average pooling layer, and a fully connected layer, which are used to extract features and obtain feature representations of the image; the nonlinear projector uses a multilayer perceptron with a hidden layer to map the feature representation extracted by the backbone network to a space where contrast loss is applied, and converts it to obtain a low-dimensional feature representation of the image, and the nonlinear projector is composed of a fully connected layer, a ReLU activation function, and another fully connected layer connected in sequence.

[0014] Furthermore, the specific process of training and adjusting the parameters of the network model using automatic weighted contrast loss is as follows:

[0015] 1) Similarity calculation: Through data enhancement and encoder modules, two sets of embedding vectors of the same batch of input images can be obtained. The embedding vectors from the same image are defined as positive sample pairs, and the embedding vectors from different images are defined as negative sample pairs. The cosine similarity between the two sets of embedding vectors is calculated to obtain a similarity matrix.

[0016] 2) Gaussian weighting: Gaussian weighting technology is used to convert the similarity relationship between sample pairs into weight factors, and then obtain a weight matrix; the mathematical expression of the weight value in the weight matrix is ​​as follows:

[0017]

[0018] Among them, w ij represents the weight value of the i-th row and j-th column in the weight matrix W, s ij represents the augmented view of the i-th input image and the enhanced view of the jth input image The similarity value between them; μ is the mean, which represents the center position of the Gaussian distribution curve, that is, the central tendency of the data distribution; σ is the standard deviation, which is used to measure the degree of dispersion of the data with respect to the mean;

[0019] 3) Dynamic adjustment: Set the similarity threshold λ, select samples with similarity greater than the threshold from the negative sample set and define them as hard negative samples, and set the weight value φ corresponding to the hard negative sample to a sufficiently small value that is infinitely close to 0, and obtain the weighted weight matrix; the mathematical expression for dynamic adjustment of the weight value is as follows:

[0020] w i ' j =w ij ·H(s ij -λ)+φ·(1-H(s ij -λ)),

[0021] Where H(ξ) represents the indicator function. When ξ≥0, H(ξ)=1, and when ξ<0, H(ξ)=0; w i ' j represents the weight value of the i-th row and j-th column in the weighted weight matrix after adjustment, corresponding to the weight value of the negative sample pair consisting of the i-th input image and the j-th input image after weighting; w ij represents the weight value of the i-th row and j-th column in the weight matrix W before adjustment, corresponding to the weight value of the negative sample pair consisting of the i-th input image and the j-th input image before adjustment; φ represents the weight value of the difficult negative sample, which should be set close to 0; λ represents the similarity threshold;

[0022] 4) Propose automatic weighted contrast loss: Multiply the positive and negative sample items in the contrast loss with their corresponding weighted weight values ​​to obtain the automatic weighted contrast loss; the calculation formula of the automatic weighted contrast loss is as follows:

[0023]

[0024] Among them, L AWC (i) represents the automatic weighted contrast loss, Represents the input image x i Generated enhanced view The embedding vector obtained after mapping is Represents the input image x i Generated enhanced view The embedding vector obtained after mapping, sim(·,·) represents the cosine similarity, Indicates the i-th input image in the enhanced view Figure 1 or enhanced vision Figure 2 The following features, It means that the jth input image is in the enhanced view Figure 1 or enhanced vision Figure 2 The following characteristics, w i ' i represents the weight value of the positive sample pair of the i-th input image after weighting, w i ' j represents the weight value of the negative sample pair consisting of the i-th input image and the j-th input image after weighting, τ represents the temperature coefficient, and exp(·) represents the exponential function with the constant e as the base;

[0025] 5) Using automatic weighted contrast loss to train the contrast hash network model based on automatic weight allocation in the feature learning phase to obtain fine-tuned network model parameters.

[0026] Furthermore, in the hash learning phase, the optimized automatic weighted contrast loss is expressed as:

[0027]

[0028] In the formula, represents the rewritten automatic weighted contrast loss, Indicates the i-th input image in the enhanced view Figure 1 The consecutive hash codes under Indicates the i-th input image in the enhanced view Figure 2 The consecutive hash codes under Indicates the i-th input image in the enhanced view Figure 1 or enhanced vision Figure 2 The consecutive hash codes under Indicates the jth input image in the enhanced view Figure 1 or enhanced vision Figure 2 The consecutive hash codes for the following.

[0029] Furthermore, in the hash learning phase, the continuous hash codes are transformed into and Quantized to binary hash code and b i 2 , and avoid the gradient vanishing problem of the network through the back propagation function, and obtain two sets of discrete binary hash code sets corresponding to the two sets of enhanced views; the symbolic function is expressed as:

[0030]

[0031] Where b i represents a binary hash code, represents the continuous hash code of the i-th input image;

[0032] The back propagation function is expressed as:

[0033]

[0034] Among them, b t+1 The binary hash code representing the t+1th gradient, b t represents the binary hash code of the t-th gradient, L represents the loss function, θ represents the model parameters used for the t-th gradient descent iteration, and r represents the model learning rate used for the t-th gradient descent iteration;

[0035] Furthermore, in the hash learning stage, the expression for calculating the quantization loss in the quantization process based on the two sets of binary hash codes obtained and the embedding vector obtained in the hash learning stage is:

[0036]

[0037] in, and b i 2Represents the binary hash code obtained after mapping by the symbolic function, L Q represents the quantization loss, sim(·,·) represents the cosine similarity;

[0038] Then the optimized automatic weighted contrast loss and quantization loss are added to obtain the combined total loss function, which is used to jointly optimize the network model; the calculation formula of the total loss function is:

[0039]

[0040] Among them, L total Represents the total loss function, which is used for training the network model in the hash learning phase, N represents the batch size, and i represents the i-th original input image.

[0041] Furthermore, the specific process of step S2 is:

[0042] First, load and freeze the parameters of the contrastive hash network model based on automatic weight allocation that has been trained with a two-stage strategy, then input the query image and the remote sensing dataset image into the trained contrastive hash network model based on automatic weight allocation, and extract the discrete binary hash codes of the query image and the remote sensing dataset image; then calculate the Hamming distance between the query image and the remote sensing dataset image through the discrete binary hash code, and measure the similarity of the query image and the remote sensing dataset image; the Hamming distance can quantify the degree of difference between the two hash codes, and the smaller the Hamming distance, the more similar the two images are. Sort the Hamming distances from small to large, and the remote sensing dataset image with the smallest Hamming distance will be ranked in front, indicating that it is most similar to the query image; finally, output the remote sensing dataset image with the smallest Hamming distance, and you can get the remote sensing dataset image that is most similar to the query image, and get the retrieval result.

[0043] Compared with the prior art, the remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing provided by the present invention has the following advantages:

[0044] First, a Gaussian weighting strategy is introduced in the proposed automatic weighted contrast loss. The present invention can automatically assign weights according to the similarity of sample pairs, solving the problem that the standard contrast loss usually regards all sample pairs as equally important, so that the model can further bring the positive sample pairs closer and push the negative sample pairs away. Second, a dynamic adjustment strategy is introduced in the proposed automatic weighted contrast loss. First, the difficult negative samples are identified by setting a similarity threshold, and then the weight value of the difficult negative sample is updated to a sufficiently small value, which can effectively avoid images belonging to the same category from being mistakenly defined as negative samples, thereby improving the accuracy of the positive and negative sample division, reducing the interference of difficult negative samples on model training, and helping the model to better learn the significant similarities of images within the class, thereby improving the discrimination accuracy of the model. Third, the introduction of the hash algorithm can effectively retain important information in high-dimensional features, and improve the retrieval efficiency while maintaining a high retrieval accuracy. At the same time, a backpropagation function is defined to avoid the gradient vanishing problem.

[0045] The present invention proposes a remote sensing image retrieval optimization method based on automatic weight allocation and contrast hashing, aiming to solve the problem of supervised hashing method's dependence on labeled images in high-resolution remote sensing image retrieval. In the feature learning stage, the automatic weighted contrast loss can further bring the positive sample pairs closer and push the negative sample pairs farther by Gaussian weighting the positive and negative sample pairs. Then, the weight values ​​of the difficult negative samples in the negative sample set are dynamically adjusted to cleverly weaken the adverse effects of the difficult negative samples on the model, significantly increase the feature differences between the positive and negative sample pairs, and enhance the model's discriminative power. In the hash learning stage, in order to prevent the gradient from disappearing, the sign function is used to forward propagate the quantization error, and then the automatic weighted contrast loss and the quantization loss are introduced to jointly optimize the model, so that the model can generate high-quality hash codes for remote sensing image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to better describe the embodiments of the present invention, the following provides illustrations of the embodiments, which together with the description are used to explain the principles of the present invention.

[0047] Figure 1 It is a two-stage flow chart of a comparative hash network model based on automatic weight allocation in an embodiment of the present invention.

[0048] Figure 2 The figure is a flow chart of remote sensing image retrieval in an embodiment of the present invention.

[0049] Figure 3 This is an example diagram of UCM data set categories in an embodiment of the present invention.

[0050] Figure 4 exemplifies the categories of AID data sets in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0052] The embodiment of the present invention proposes a remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing, which specifically includes the following steps:

[0053] S1: Construct a contrastive hash network model based on automatic weight allocation, which mainly includes a feature learning stage and a hash learning stage. First, the feature learning stage uses the data enhancement module and the encoder module to obtain the features of different views. At the same time, the feature learning stage innovatively introduces Gaussian weighting and dynamic adjustment strategy to propose automatic weighted contrast loss to adjust the parameters of the network model, so that the weight matrix can express the similarity of different sample pairs, while identifying and reducing the interference of difficult negative samples on model training. Finally, the hash learning stage introduces the hash algorithm into the network model, and combines the automatic weighted contrast loss with the quantization loss to optimize the contrastive hash network model based on automatic weight allocation.

[0054] The structure of the comparative hash network model based on automatic weight allocation constructed by the present invention is as follows: Figure 1As shown. The feature learning stage includes three components: data enhancement module, encoder module, and automatic weighted contrast loss. In the feature learning stage, each batch of N remote sensing images obtains different enhanced views through the data enhancement module, and then the encoder module is used to extract potential features from the enhanced views. Then the similarity between the features is calculated to obtain the similarity matrix S. The element values ​​on the main diagonal of the similarity matrix S are the similarities of the positive sample pairs, and the others are the similarities of the negative sample pairs. The similarity matrix S is Gaussian weighted to obtain the weight matrix W. The element values ​​on the main diagonal of the weight matrix W are the weight values ​​of the positive sample pairs, and the other element values ​​are the weight values ​​of the negative sample pairs. When the similarity value of the negative sample pair reaches the threshold, it is defined as a difficult negative sample. The weight value of the difficult negative sample is dynamically adjusted, and the weight value of the difficult negative sample in the weight matrix is ​​set to the same minimum value, and then the weighted weight matrix is ​​calculated (the low weight value of the difficult negative sample is marked in the form of a black block in the matrix). Next, the weighted weight matrix is ​​used to optimize the contrast loss function, and an automatic weighted contrast loss is designed; the automatic weighted contrast loss is used to train and adjust the contrast hash network model based on automatic weight allocation, and the fine-tuning network model parameters are obtained. The hash learning stage shares the data enhancement module and the encoder module with the feature learning stage, and also includes a hash layer and a joint loss back propagation training network model. In the hash learning stage, the fine-tuning network model parameters obtained in the feature learning stage are first loaded, and the image is input into the network model after training and parameter adjustment for feature extraction to obtain optimized image features and embedding vectors, and then the optimized embedding vectors are converted into continuous hash codes using the hash layer, and the optimized automatic weighted contrast loss is calculated using the continuous hash code; then the continuous hash code is quantized into a binary hash code, and the gradient vanishing problem of the network is avoided by the back propagation function to obtain two sets of discrete binary hash code sets; the quantization loss in the quantization process is calculated based on the two sets of binary hash codes obtained, and then the combined total loss function is calculated by the optimized automatic weighted contrast loss and the quantization loss, and the network model is trained and learned using the total loss function to obtain the trained network model parameters. Finally, the trained network model is applied to remote sensing image retrieval.

[0055] S11: In the network model provided by the present invention, the feature learning stage includes three components, namely, a data enhancement module, an encoder module, and training and parameter adjustment using automatic weighted contrast loss.

[0056] S111: Data augmentation module, given N input images from the same batch in the training set By introducing a rich and diverse set of data augmentation strategies, we generate two different enhanced views for each image. and The input image x i Two enhanced views generated Defined as similar views, that is, positive sample pairs, similarly, the input image x j Two enhanced views generated It also belongs to similar views and is a positive sample pair. and represents a pair of dissimilar views generated by different input images, i.e., a pair of negative samples, where i≠j. The data enhancement strategies used in this embodiment include random cropping, scaling, horizontal flipping, random color jittering, random grayscale, and random Gaussian blurring, etc., aiming to increase the generalization ability of the network model.

[0057] S112: Encoder module. The contrast hash network model based on automatic weight allocation of the present invention is provided with two parallel and weight-sharing branches to jointly learn the representation of the input image data. The core component of the two-stage network model is the encoder module. The encoder module in this embodiment not only integrates a powerful backbone network for basic feature extraction, but also adds a nonlinear projector, the purpose of which is to further map the extracted high-level features to an embedding space suitable for self-supervised learning. The backbone network f(·) usually uses a convolutional neural network (CNN). This article selects the ResNet50 model in the convolutional neural network category as the backbone network. ResNet-50 is a deep residual network with powerful modeling capabilities and good feature extraction capabilities. The backbone network f(·), i.e., the ResNet50 model, includes an input layer, an initial convolutional layer, a residual block, a global average pooling layer, and a fully connected layer, which are used to extract features and obtain feature representations of images. The nonlinear projector g(·) (i.e., Projection) uses a multi-layer perceptron (MLP) with one hidden layer to map the feature representation extracted by the backbone network f(·) to the space where the contrast loss is applied. The nonlinear projector consists of a fully connected layer, a ReLU activation function, and another fully connected layer connected in sequence.

[0058] Each pair of enhanced views and After inputting into the backbone network for feature extraction, the respective feature representations are obtained and Where N is the batch size. These feature representations are the output of the backbone network f(·) through the average pooling layer, and the feature dimension is 2048. Next, the nonlinear projector g(·) transforms the feature representation and feature representation Mapped to a low-dimensional embedding space to obtain an embedding vector and The feature dimension is 256. The conversion process can be summarized as follows:

[0059]

[0060] in, Represents enhanced view The embedding vector obtained after mapping is Represents enhanced view The embedding vector obtained after mapping; the mapping process corresponding to f(·) is represents the backbone network mapping, and the mapping process corresponding to g(·) is represents the nonlinear projector mapping, H and W are the height and width of the input image, and C is the number of channels.

[0061] S113: Use automatic weighted contrast loss for training and parameter adjustment. The core of contrast loss is to minimize the distance between positive sample pairs and maximize the distance between positive and negative sample pairs. From the definition of positive and negative samples, we can know that positive sample pairs usually have higher similarity. Therefore, in order to further optimize the loss function, this paper proposes automatic weighted loss, which can highlight the learning of more valuable sample pairs, and reduce the interference of difficult negative samples on the network model, so as to maintain the stability of network model learning. The specific process is as follows:

[0062] 1) Similarity calculation: for N input image samples in the same batch The two embedding vector sets of image features obtained by the data enhancement module and the encoder module are: and Calculate the cosine similarity of two embedding vector sets and get the similarity matrix S = {s ij}∈R N×N , R represents the set of real numbers, then s ij represents the augmented view of the i-th input image and the enhanced view of the jth input image The similarity value between .

[0063] 2) Gaussian weighting: The similarity relationship between sample pairs is converted into a weight factor through Gaussian weighting technology. Positive sample pairs with high similarity are given higher weights, while negative sample pairs with low similarity are assigned lower weights, thereby forming a decreasing weight distribution pattern. This mechanism plays a dual effect in the optimization network model of the present invention. It not only strengthens the intimacy between positive sample pairs and makes their feature vector distances closer, but also increases the distinction between positive and negative sample pairs, pushes away the feature differences between them, and deepens the interval. Gaussian weighting is a function form based on Gaussian distribution. In this embodiment, the similarity matrix S is weighted by a Gaussian function to obtain a weight matrix W. The mathematical expression of the weight values ​​in the weight matrix W is as follows:

[0064]

[0065] Among them, w ijRepresents the weight value of the i-th row and j-th column in the weight matrix W. μ is the mean, which represents the center position of the Gaussian distribution curve, that is, the central tendency of the data distribution. σ is the standard deviation, which is used to measure the degree of dispersion of the data with respect to the mean.

[0066] 3) Dynamic adjustment: Input image x i Two enhanced views generated is defined as a similar view, i.e., a positive sample pair. and The enhanced views generated by representing different input images are defined as dissimilar view pairs, i.e., negative sample pairs, where i≠j. However, if different views of the same image are univocally identified as positive samples, and all views of different images are identified as negative samples, it is easy to ignore the significant similarities between different images in the category, resulting in a reduction in the discrimination accuracy of the model. In this embodiment, samples with high similarity rankings are selected from the negative sample set and defined as difficult negative samples, and the weights of difficult negative samples are reduced to reduce interference with network model training. A high degree of similarity indicates that the difficult negative sample is likely to be of the same category as the input image to be queried. Therefore, a similarity threshold λ is set, and samples with a similarity greater than the threshold are selected as difficult negative samples, and the weight value φ corresponding to the difficult negative sample is set to be sufficiently small, i.e., to a value infinitely close to 0. The weighted weight matrix is ​​obtained, which can indirectly reduce the interference of difficult negative samples on model training. The mathematical expression for dynamically adjusting the weight value is as follows:

[0067] w i ' j =w ij ·H(s ij -λ)+φ·(1-H(s ij -λ)) (3)

[0068] Wherein, H(ξ) represents the indicator function. When ξ≥0, H(ξ)=1, and when ξ<0, H(ξ)=0. i ' j represents the weight value of the i-th row and j-th column in the weighted weight matrix after adjustment, corresponding to the weight value of the negative sample pair consisting of the i-th input image and the j-th input image after weighting, w ij Represents the weight value of the i-th row and j-th column in the weight matrix W before adjustment, corresponding to the weight value of the negative sample pair consisting of the i-th input image and the j-th input image before adjustment. φ represents the weight value of the difficult negative sample and should be set close to 0. λ represents the similarity threshold.

[0069] 4) Propose automatic weighted contrast loss: multiply the similarity of positive and negative sample items in the contrast loss by their corresponding weighted weight values, which can automatically adjust the contribution of different samples to the learning of the network model, further bring the positive sample pairs closer and push the negative sample pairs further away. At the same time, assign lower weight values ​​to difficult negative samples, which reduces the interference of these difficult negative samples on the learning of the network model, and helps the network model learn features from the samples more evenly. Define automatic weighted contrast loss as follows:

[0070]

[0071] Among them, L AWC (i) represents the automatic weighted contrast loss, sim(·,·) represents the cosine similarity, Indicates the i-th input image in the enhanced view Figure 1 or enhanced vision Figure 2 The following features, It means that the jth input image is in the enhanced view Figure 1 or enhanced vision Figure 2 The following characteristics, w i ' i represents the weight value of the positive sample pair of the i-th input image after weighting, w i ' j represents the weight value of the negative sample pair consisting of the i-th input image and the j-th input image after weighting, τ represents the temperature coefficient, and exp(·) represents the exponential function with the constant e as the base;

[0072] 5) Use automatic weighted contrast loss L AWC (i) Training the comparison hash network model based on automatic weight allocation once to obtain fine-tuned network model parameters.

[0073] S12: In the network model provided by the present invention, the hash learning stage includes four components, namely, a data enhancement module, an encoder module, a hash layer, and a joint loss back propagation training network model based on quantization loss and automatic weighted contrast loss, wherein the data enhancement module and the encoder module are shared with the feature learning stage, and directly share the information obtained by the data enhancement module and the encoder module in the feature learning stage.

[0074] S121: In the hash learning phase, first you need to load and freeze the fine-tuned network model parameters obtained in the feature learning phase, and use the trained fine-tuned network model to re-acquire the input image in the input image Two enhanced views of and And obtain the new embedding vector obtained by training the fine-tuned data enhancement module and encoder module and

[0075]

[0076] S122: Then the embedding vector obtained in the hash learning phase is reduced in dimension through the hash layer. The hash layer is defined as follows:

[0077]

[0078] The hashing layer fc(·) uses a fully connected layer to generate hash codes. The fully connected layer can embed the vector and embedding vector Dimensionality reduction to K-dimensional continuous hash representation and

[0079] S123: Using continuous hash codes, the automatic weighted contrast loss is rewritten as:

[0080]

[0081] In the formula, represents the rewritten automatic weighted contrast loss, Indicates the i-th input image in the enhanced view Figure 1 or enhanced vision Figure 2 The consecutive hash codes under Indicates the jth input image in the enhanced view Figure 1 or enhanced vision Figure 2 The continuous hash code under w i ' i Represents the weight value of the positive sample pair of the i-th input image after weighting.

[0082] The continuous hash code is quantized into a binary hash code through the symbolic function sgn(·), which can be expressed as:

[0083]

[0084] Where b i represents a binary hash code, Represents the continuous hash code of the i-th input image.

[0085] At this time, after mapping the continuous hash code into a discrete binary hash code, the traditional back propagation method is generally used to optimize the parameters of continuous values, but cannot directly process binary discrete outputs, which means that we need to solve the problem of how to make the network output discrete hash codes during the gradient descent process. Most of the previous work chose to use a continuous relaxation method to perform binary hash optimization, replacing the discrete sign function with a smooth activation function (such as tanh or sigmoid). These relaxation schemes will introduce quantization errors, resulting in suboptimal hash codes. Some work strictly uses a sign function to forward propagate the quantization error, and then transmits the gradient completely to the previous layer of the network structure, effectively preventing the gradient from disappearing, and updating all bits together. The present invention uses a defined back propagation function to avoid the gradient vanishing problem in the network. The back propagation function is defined as follows:

[0086]

[0087] Among them, b t+1 The binary hash code representing the t+1th gradient, b t represents the binary hash code of the t-th gradient, L represents the loss function, θ represents the model parameters used for the t-th gradient descent iteration, and r represents the model learning rate used for the t-th gradient descent iteration. and hash representation The binary hash code is obtained after mapping through the symbol function and the binary hash code b i 2 ;

[0088] Then according to the binary hash code obtained and the binary hash code b i 2 The embedding vector obtained in the hash learning phase is used to calculate the quantization loss in the quantization process, and the quantization loss is defined as:

[0089]

[0090] Among them, L Q represents the quantization loss, and sim(·,·) represents the cosine similarity.

[0091] S124: Combine the rewritten automatic weighted contrast loss and quantization loss to obtain the total loss function:

[0092]

[0093] Among them, L total Represents the total loss function, which is used for training the network model in the hash learning phase, N represents the batch size, and i represents the i-th original input image.

[0094] S125: Using the total loss function L total The comparative hash network model based on automatic weight allocation is trained and learned to obtain trained network model parameters.

[0095] S2: Finally, the comparative hash network model based on automatic weight allocation proposed in the present invention is applied to the remote sensing image retrieval task. The two most critical issues in remote sensing image retrieval are feature extraction and similarity measurement. The former is committed to mining key image information and converting it into high-quality features, while the latter judges the degree of difference between images by measuring the similarity between features. The present invention constructs a high-performance hash network model through a carefully designed two-stage training strategy. The network model can quantize high-dimensional real-valued features into low-dimensional discrete hash codes to improve the retrieval efficiency of remote sensing images.

[0096] Remote sensing image retrieval process Figure 2 As shown in the figure, first load and freeze the parameters of the contrastive hash network model based on automatic weight allocation that has been trained with a two-stage strategy, input the query image and the remote sensing dataset image into the trained contrastive hash network model based on automatic weight allocation, and extract the discrete binary hash codes of the query image and the remote sensing dataset image. Then calculate the Hamming distance between the query image and the remote sensing dataset image through the discrete binary hash code, and measure the similarity of the query image and the remote sensing dataset image. The Hamming distance can quantify the degree of difference between the two hash codes. The smaller the Hamming distance, the more similar the two images are. Sort the Hamming distance from small to large, and the remote sensing dataset image with the smallest Hamming distance will be ranked in front, indicating that it is most similar to the query image. Finally, output the remote sensing dataset image with the smallest Hamming distance, and you can get the remote sensing dataset image that is most similar to the query image, and get the retrieval result.

[0097] The present invention is evaluated on two common remote sensing datasets. The UCM dataset is a land use image dataset for research purposes. It contains 2,100 images, including 21 scenes such as agriculture, airplanes, and baseball diamonds. Each type has 100 images, and the size of each image is 256×256 pixels. For example, see Figure 3 The AID dataset collects high-resolution aerial remote sensing images from different regions, time periods, and perspectives. The dataset contains 10,000 images. The number of images in each category is inconsistent. It includes more than 30 categories such as cities, farmlands, forests, and industrial areas. The size of each image is 600×600 pixels. For example images, see Figure 4 .

[0098] The method proposed in the present invention and some existing hash algorithms are used for remote sensing image retrieval. Among them, 6 existing competitive or classic unsupervised deep hashing methods are re-implemented based on the open source code provided by the original author, including BiHalf algorithm, CIBHash algorithm, SDC algorithm, SDC_SimCLR algorithm, GreedyHash algorithm and DCSH algorithm. At the same time, CIBHash algorithm, SDC_SimCLR algorithm and DCSH algorithm are all unsupervised hash algorithms based on contrastive learning structure. The experimental results are shown in the following table.

[0099] In terms of qualitative retrieval results, Tables 1 and 2 show the comparison of mAP (mean average precision) retrieval results with other unsupervised hashing methods on the UCM and AID datasets. On these two datasets, we compared the retrieval performance of 16-bit, 32-bit, 64-bit and 128-bit hash codes respectively. Most hashing methods achieved higher average precision with increasing the number of bits, which shows that the length of the hash code has a great influence on the retrieval performance. Longer hash codes can better retain image feature information and improve retrieval performance. The method of the present invention shows high retrieval accuracy on both UCM and AID datasets, and its average precision is much higher than that of other methods. In particular, under 64-bit and 128-bit, the average precision of retrieval on the UCM dataset using the method of the present invention reached 80.46% and 77.80%, respectively, and the average precision of retrieval on the AID dataset reached 84.27% and 84.62%, respectively. It shows that the method of the present invention is effective and stable in the hash retrieval task of remote sensing images.

[0100] Table 1 Comparison of average precision (mAP) retrieval results on the UCM dataset and different hashing methods

[0101]

[0102] Table 2 Comparison of average precision (MAP) retrieval results on the AID dataset and different hashing methods

[0103]

[0104] Tables 3 and 4 show the comparison of ANMRR retrieval results of different hashing methods on the UCM and AID datasets. According to the experimental results, it can be seen that on the UCM and AID datasets, the method of the present invention performs well at all digits and has a lower ANMRR value (average normalized corrected retrieval rank), which indicates that the method of the present invention has good performance in the image retrieval task and can effectively maintain the order of the retrieval results. In contrast, the performance of other hashing methods is relatively poor, especially at a shorter hashing digit, the ANMRR value is higher, indicating a decrease in retrieval performance. Therefore, the experimental results show that the method of the present invention has stable performance under different datasets and digits, and can be used as a reliable image retrieval method.

[0105] Table 3 Comparison of ANMRR retrieval results on the UCM dataset and different hashing methods

[0106]

[0107] Table 4 Comparison of ANMRR retrieval results on the AID dataset and different hashing methods

[0108]

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing, characterized in that: The following steps are involved: S1: constructing a comparative hash network model based on automatic weight allocation, wherein the comparative hash network model based on automatic weight allocation includes a feature learning stage and a hash learning stage; S11: The feature learning stage includes three components: data enhancement module, encoder module, and automatic weighted contrast loss; the data enhancement module is used to acquire remote sensing image data and obtain an enhanced view of each input image, and the encoder module is used to extract image features of depth partitions from the enhanced view; Automatic weighted contrast loss is calculated by dynamically adjusting the weight factor, which is used to train and adjust the network model parameters to obtain fine-tuned network model parameters; S12: The hash learning stage shares the data enhancement module and encoder module with the feature learning stage, and also includes a hash layer and a joint loss. In the hash learning stage, the fine-tuned network model parameters obtained in the feature learning stage are first loaded, and the image is input into the trained and parameter-adjusted network model for feature extraction to obtain optimized image features and embedding vectors. The hash layer converts the optimized embedding vector into a continuous hash code, and uses the continuous hash code to calculate the optimized automatic weighted contrast loss. The continuous hash code is then quantized into a binary hash code to obtain two sets of discrete binary hash code sets. The quantization loss in the quantization process is calculated based on the two sets of binary hash codes obtained, and then the combined total loss function is calculated through the optimized automatic weighted contrast loss and the quantization loss. The network model is trained and learned using the total loss function to obtain the trained network model parameters. S2: Apply the trained network model to remote sensing image retrieval, query the remote sensing dataset image that is most similar to the query image through the network model, and output the retrieval result.

2. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 1 is characterized in that: After each remote sensing image data passes through the data enhancement module, two enhanced views with different perspectives are generated. The methods used by the data enhancement module to enhance image data include random cropping, scaling, horizontal flipping, random color jittering, random grayscale and random Gaussian blur.

3. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 2 is characterized in that: The encoder module includes a backbone network and a nonlinear projector. The backbone network is a ResNet50 model. The backbone network, i.e., the ResNet50 model, includes an input layer, an initial convolutional layer, a residual block, a global average pooling layer, and a fully connected layer, which are used to extract features and obtain a high-dimensional feature representation of the image. The nonlinear projector uses a multilayer perceptron with a hidden layer to map the feature representation extracted by the backbone network to a space where contrast loss is applied, and converts it to obtain a low-dimensional feature representation of the image. The nonlinear projector is composed of a fully connected layer, a ReLU activation function, and another fully connected layer connected in sequence.

4. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 3 is characterized in that: The specific process of training and adjusting the parameters of the network model using the automatic weighted contrast loss is as follows: through data enhancement and encoder modules, two sets of embedding vectors of the same batch of input images can be obtained, and the embedding vectors from the same image are defined as positive sample pairs, and the embedding vectors from different images are defined as negative sample pairs; the cosine similarity between the two sets of embedding vectors is calculated to obtain a similarity matrix; Use Gaussian weighting technology to convert the similarity relationship of sample pairs into weight factors, and then obtain the weight matrix; set the similarity threshold , select samples with similarity greater than the threshold from the negative sample set and define them as hard negative samples, and set the weight value corresponding to the hard negative sample The weight matrix is ​​set to a value that is small enough and infinitely close to 0 to obtain a weighted weight matrix; the positive and negative sample items are multiplied by their corresponding weighted weight values ​​to obtain an automatic weighted contrast loss; the automatic weighted contrast loss is used to train the contrast hash network model based on automatic weight allocation in the feature learning phase to obtain fine-tuning network model parameters.

5. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 4 is characterized in that: The hash layer uses a fully connected layer to reduce the embedding vector output by the encoder module into a K-dimensional continuous hash code.

6. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 5 is characterized in that: In the hash learning phase, the continuous hash codes are quantized into binary hash codes using the sign function sgn(·).

7. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 6 is characterized in that: The specific steps of calculating the combined total loss function include: concatenating and fusing the discrete binary hash codes corresponding to the two enhanced views; calculating the quantization loss between the low-dimensional embedding vector output by the encoder module and the binary hash code; and calculating the optimized automatic weighted contrast loss with the continuous hash code at the same time; adding the quantization loss and the optimized automatic weighted loss to obtain the combined total loss function, which is used to train and learn the network model in the hash learning stage.

8. The remote sensing image retrieval optimization method based on automatic weight allocation and comparative hashing according to claim 7 is characterized in that: The specific process of step S2 is: first load and freeze the parameters of the contrastive hash network model based on automatic weight allocation that has been trained by the two-stage strategy, then input the query image and the remote sensing dataset image into the trained contrastive hash network model based on automatic weight allocation, and extract the discrete binary hash codes of the query image and the remote sensing dataset image; Then, the Hamming distance between the query image and the remote sensing dataset image is calculated through discrete binary hash codes to measure the similarity between the query image and the remote sensing dataset image. The Hamming distance can quantify the degree of difference between the two hash codes. The smaller the Hamming distance, the more similar the two images are. The Hamming distances are sorted from small to large. The remote sensing dataset image with the smallest Hamming distance will be ranked in front, indicating that it is most similar to the query image. Finally, the remote sensing dataset image with the smallest Hamming distance is output, and the remote sensing dataset image that is most similar to the query image can be obtained, and the retrieval result is obtained.

Images