Remote sensing image retrieval optimization method based on automatic weight assignment and contrastive hashing

Abstract

The application provides a remote sensing image retrieval optimization method based on automatic weight distribution and contrastive hashing, constructs a contrastive hashing network model based on automatic weight distribution, and adopts a two-stage training strategy to optimize the model performance. In the feature learning stage, an automatic weighted contrastive loss is proposed, a Gaussian weighting and dynamic adjustment strategy is introduced to improve the traditional loss function, different weight values are given according to the importance of sample pairs, and the learning of the model on key sample pairs is strengthened; a threshold is set to identify difficult negative samples, so that the model is not disturbed by the difficult negative samples. In the hash learning stage, the hash layer is added to convert the high-dimensional image representation into binary hash code, the quantization loss is introduced to learn the hash code after splicing and fusion, so that the semantic similar structure between data in the Hamming space can be kept, and the automatic weighted contrastive loss is used to improve the identification of the hash code. The application can effectively retrieve the target remote sensing image, and the retrieval speed is relatively fast.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and remote sensing image processing, specifically, it relates to a remote sensing image retrieval optimization method based on automatic weight distribution and contrastive hashing, which aims to improve the efficiency and accuracy of remote sensing image retrieval. BACKGROUND

[0002] With the development of earth observation technology, the number of remote sensing images shows an explosive growth trend. How to retrieve target images from a large number of remote sensing databases has become a current research hotspot. Remote sensing image retrieval can be mainly divided into text-based remote sensing image retrieval and content-based remote sensing image retrieval. Text-based remote sensing image retrieval draws on the text search technology (Text-based Image Retrieval, TBIR), and the TBIR method needs to describe the image with text, and then find the target retrieval image according to the text description. However, this method requires a large number of human resources to label data, and the image information expression ability is insufficient, which makes it difficult for the TBIR method to be applied to the current massive remote sensing image data scene. The content-based image retrieval method (Content-Based Image Retrieval, CBIR) focuses on understanding the features of the image itself, and uses the content features of the image as an index to construct an image database, and then realizes retrieval. Compared with the TBIR method, the CBIR method is more suitable for the needs of modern remote sensing image retrieval.

[0003] According to whether a supervision signal is needed in the network training process, the CBIR method can be divided into supervised and unsupervised. In supervised learning, constructing a labeled data set is a time-consuming and labor-intensive task, especially for remote sensing image data sets, the labeling process is highly dependent on manual labor and professional knowledge, which requires a large amount of time and economic cost. In order to overcome the limitations of traditional supervised learning, self-supervised learning as an unsupervised learning method has received extensive attention. Among them, contrastive learning is an important technology, the core idea of which is to pull similar samples and push away dissimilar samples, so as to learn features with discriminative power. The contrastive learning method can be roughly divided into negative sample-based method, clustering-based method, asymmetric network-based method and information maximization-based method.

[0004] Although contrastive learning can effectively learn features with discriminability, there are three main difficulties when applied to remote sensing image retrieval tasks: first, the weight allocation problem, the standard contrastive loss usually considers all sample pairs as equally important, without considering the difference between samples; second, the accuracy of positive and negative samples, the traditional construction method defines different views of the same image as positive sample pairs and views of different images as negative sample pairs, this construction method will identify images belonging to the same class as negative samples, which may cause the model to learn common features of different classes and ignore the significant similarity of different images within the class, thereby reducing the discriminant accuracy of the model; third, the high-dimensional feature problem, although the contrastive learning method can learn high-quality features, the high-dimensional deep features obtained often cause the curse of dimensionality and also reduce the retrieval efficiency. Therefore, how to effectively solve the above difficulties needs to be studied. SUMMARY

[0005] Therefore, the present application provides a remote sensing image retrieval optimization method based on automatic weight allocation and contrastive hashing. Compared with existing remote sensing image retrieval methods, the present application uses contrastive learning to effectively solve the dependence on image labels. However, the commonly used contrastive loss usually considers all sample pairs as equally important, without considering the difference between samples. The present application proposes an automatic weighted contrastive loss, which performs Gaussian weighting on the similarity between sample pairs, strengthens the model's learning of key sample pairs, further closes positive sample pairs and pushes away negative sample pairs. At the same time, the weight value is dynamically adjusted to weaken the interference of difficult negative sample pairs on model training. Finally, the hash algorithm can map high-dimensional features to discrete hash codes, improving retrieval efficiency.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows.

[0007] A remote sensing image retrieval optimization method based on automatic weight allocation and contrastive hashing, comprising the following steps:

[0008] S1: constructing a contrastive hashing network model based on automatic weight allocation, the contrastive hashing network model based on automatic weight allocation mainly includes a feature learning stage and a hash learning stage;

[0009] S11: the feature learning stage includes a data augmentation module, an encoder module, and an automatic weighted contrastive loss; the data augmentation module is used to obtain remote sensing image data and obtain an enhanced view of each input image, the encoder module is used to extract deep partitioned image features from the enhanced view; the automatic weighted contrastive loss is obtained by dynamically adjusting a weight factor to train and adjust the network model, specifically: the similarity matrix and the weight matrix are obtained by calculating the similarity between the image features and the weight factor, and the weight value of the difficult negative sample is dynamically adjusted, the automatic weighted contrastive loss is calculated through the similarity matrix and the dynamically adjusted weight matrix, and the automatic weighted contrastive loss is used to train and adjust the network model in the feature learning stage to obtain the fine-tuned network model parameters;

[0010] S12: the hash learning stage shares the data augmentation module and the encoder module with the feature learning stage, and further includes a hash layer and a joint loss; the hash learning stage first loads the fine-tuned network model parameters obtained in the feature learning stage, inputs the image into the trained and adjusted network model for feature extraction, obtains the optimized image features and embedding vectors; the hash layer converts the optimized embedding vectors into continuous hash codes, and uses the continuous hash codes to calculate the optimized automatic weighted contrastive loss; then the continuous hash codes are quantized into binary hash codes to obtain two sets of discrete binary hash codes; the quantization loss in the quantization process is calculated according to the two sets of binary hash codes, and the total loss function after jointing is calculated through the optimized automatic weighted contrastive loss and the quantization loss, and the network model is trained and learned using the total loss function to obtain the trained network model parameters; the hash learning stage is used to learn the spliced and fused hash codes to maintain the semantic similarity structure between data;

[0011] S2: the trained network model is applied to remote sensing image retrieval, and the network model is used to query the most similar remote sensing data set image from the remote sensing data set to the image to be queried, and the retrieval result is output.

[0012] Further, each remote sensing image data generates two enhanced views with different perspectives after the data augmentation module, and the data augmentation module enhances the image data by random cropping, scaling, horizontal flipping, random color jittering, random grayscale, and random Gaussian blur.

[0013] Further, the encoder module comprises a backbone network and a nonlinear projector, the backbone network is a ResNet50 model, the backbone network, i.e., the ResNet50 model, comprises an input layer, an initial convolutional layer, a residual block, a global average pooling layer and a fully connected layer, and is used for feature extraction to obtain a feature representation of an image; the nonlinear projector uses a multilayer perception with one hidden layer to map the feature representation extracted by the backbone network to a space where the contrast loss is applied, and converts to obtain a low-dimensional feature representation of the image, the nonlinear projector is composed of a fully connected layer, a ReLU activation function and another fully connected layer connected in sequence.

[0014] Further, the specific process of training and parameterizing the network model by using the automatically weighted contrast loss is as follows:

[0015] 1) Similarity calculation: through data enhancement and the encoder module, two sets of embedding vector collections of the same batch of input images can be obtained, embedding vectors from the same image are defined as positive sample pairs, and embedding vectors from different images are defined as negative sample pairs; the cosine similarity between the two sets of embedding vector collections is calculated to obtain a similarity matrix;

[0016] 2) Gaussian weighting: the similarity relationship of the sample pairs is converted into a weight factor by using the Gaussian weighting technology, and then a weight matrix is obtained; the mathematical expression of the weight value in the weight matrix is as follows:

[0017]

[0018] wherein, w ij represents the weight value of the i-th row and the j-th column in the weight matrix W, s ij represents the similarity value between the enhanced view of the i-th input image and the enhanced view of the j-th input image; μ is the mean value, representing the center position of the Gaussian distribution curve, i.e., the central tendency of the data distribution; σ is the standard deviation, used to measure the dispersion degree of the data to the mean value;

[0019] 3) Dynamic adjustment: set a similarity threshold λ, select the samples with similarity greater than the threshold from the negative sample set and define them as difficult negative samples, and set the weight value φ corresponding to the difficult negative samples to a value small enough and infinitely close to 0 to obtain a weighted weight matrix; the mathematical expression of the dynamic adjustment of the weight value is as follows:

[0020] w i ′ j =w ij ·H(s ij -λ)+φ·(1-H(s ij -λ)),

[0021] wherein H(ξ) represents an indicator function, H(ξ) = 1 when ξ ≥ 0, and H(ξ) = 0 when ξ < 0; w i j represents the weight value of the i-th row and the j-th column in the adjusted weight matrix, corresponding to the weight value of the negative sample pair composed of the i-th input image and the j-th input image after weighting; w ij represents the weight value of the i-th row and the j-th column in the weight matrix W before adjustment, corresponding to the weight value of the negative sample pair composed of the i-th input image and the j-th input image before adjustment; φ represents the weight value of a difficult negative sample, which should be set to be close to 0; λ represents a similarity threshold value;

[0022] 4) An automatic weighting contrastive loss is proposed: the positive and negative sample terms in the contrastive loss are multiplied by their corresponding weighted weight values to obtain an automatic weighting contrastive loss. The calculation formula of the automatic weighting contrastive loss is as follows:

[0023]

[0024] wherein L AWC (i) represents the automatic weighting contrastive loss, represents the input image x i generated enhanced view the embedding vector obtained after mapping, represents the input image x i generated enhanced view the embedding vector obtained after mapping, sim(·,·) represents a cosine similarity, represents the feature of the i-th input image in the enhanced view Figure 1 or the enhanced view Figure 2 , represents the feature of the j-th input image in the enhanced view Figure 1 or the enhanced view Figure 2 , w i i represents the weight value of the positive sample pair of the i-th input image after weighting, w i j represents the weight value of the negative sample pair composed of the i-th input image and the j-th input image after weighting, τ represents a temperature coefficient, and exp(·) represents an exponential function with a constant e as the base number;

[0025] 5) The automatic weighting contrastive loss is used to train the feature learning stage of the automatic weight distribution-based contrastive hashing network model, to obtain fine-tuned network model parameters.

[0026] Further, in the hash learning stage, the optimized automatic weighting contrastive loss is represented as:

[0027] ​​​

[0028] wherein, represents the rewritten auto-weighted contrastive loss, represents the continuous hash code of the i-th input image under the enhanced view Figure 1 represents the continuous hash code of the i-th input image under the enhanced view represents the continuous hash code of the i-th input image under the enhanced view Figure 2 represents the continuous hash code of the i-th input image under the enhanced view represents the continuous hash code of the i-th input image under the enhanced view Figure 1 or the enhanced view Figure 2 represents the continuous hash code of the i-th input image under the enhanced view represents the continuous hash code of the j-th input image under the enhanced view Figure 1 or the enhanced view Figure 2 represents the continuous hash code of the j-th input image under the enhanced view

[0029] Further, in the hash learning stage, the continuous hash codes and are quantized into binary hash codes and b i 2 by a sign function, and the gradient disappearance problem of the network is avoided by a back propagation function, to obtain two sets of discrete binary hash code sets corresponding to the two sets of enhanced views; the sign function is represented as:

[0030]

[0031] wherein, b i represents the binary hash code, represents the continuous hash code of the i-th input image;

[0032] the back propagation function is represented as:

[0033]

[0034] wherein, b t+1 represents the binary hash code of the t+1-th gradient, b t represents the binary hash code of the t-th gradient, L represents the loss function, and θ represents the model parameter used for the t-th gradient descent iteration, and r represents the model learning rate used for the t-th gradient descent iteration;

[0035] Further, in the hash learning stage, the expression of the quantization loss in the quantization process is calculated according to the two sets of binary hash codes obtained and the embedding vectors obtained in the hash learning stage:

[0036]

[0037] wherein, and b i 2denotes the binary hash code obtained after mapping by the sign function, L Q denotes the quantization loss, and sim(·,·) denotes the cosine similarity.

[0038] Then the optimized auto-weighted contrastive loss and the quantization loss are added to obtain a joint total loss function for jointly optimizing the network model, and the calculation formula of the total loss function is:

[0039]

[0040] wherein, L total denotes the total loss function, is used for training the network model in the hash learning stage, N denotes the batch size, and i denotes the i-th original input image.

[0041] Further, the specific process of step S2 is:

[0042] First, the parameters of the auto-weighted contrastive hash network model trained by the two-stage strategy are loaded and frozen, and then the query image and the remote sensing dataset image are input into the trained auto-weighted contrastive hash network model to extract the discrete binary hash codes of the query image and the remote sensing dataset image; then the Hamming distance between the query image and the remote sensing dataset image is calculated through the discrete binary hash codes to measure the similarity between the query image and the remote sensing dataset image; the Hamming distance can quantify the difference between two hash codes, and the smaller the Hamming distance, the more similar the two images are; the remote sensing dataset image with the smallest Hamming distance will be ranked in the front, indicating that it is the most similar to the query image; finally, the remote sensing dataset image with the smallest Hamming distance is output, and the most similar remote sensing dataset image to the query image is obtained, and the retrieval result is obtained.

[0043] Compared with the prior art, the remote sensing image retrieval optimization method based on auto-weighted allocation and contrastive hash provided by the present application has the following advantages:

[0044] Firstly, the Gaussian weighting strategy is introduced in the proposed automatic weighted contrastive loss, and the application can automatically assign weights according to the similarity of the sample pair, solve the problem that the standard contrastive loss usually regards all sample pairs as equally important, and make the model further close the positive sample pair and push away the negative sample pair. Secondly, the dynamic adjustment strategy is introduced in the proposed automatic weighted contrastive loss, firstly, the difficulty negative sample is identified by setting the similarity threshold, and then the weight value of the difficulty negative sample is updated to a small enough value, which can effectively avoid the error definition of the images belonging to the same category as negative samples, thereby improving the accuracy of the positive and negative sample division, reducing the interference of the difficulty negative sample on the model training, and helping the model to better learn the significant similarity of the in-class images, thereby improving the discrimination precision of the model. Thirdly, the hash algorithm is introduced to effectively retain important information in high-dimensional features, improve the retrieval efficiency while maintaining high retrieval accuracy. At the same time, the back propagation function is defined to avoid the gradient vanishing problem.

[0045] The application proposes a remote sensing image retrieval optimization method based on automatic weight allocation and contrastive hashing, aiming to solve the problem of dependence of supervised hashing method on labeled images in high-resolution remote sensing image retrieval. In the feature learning stage, the automatic weighted contrastive loss can further close the positive sample pair and push away the negative sample pair by Gaussian weighting processing of the positive and negative sample pairs. Then the weight value of the difficulty negative sample in the negative sample set is dynamically adjusted, which cleverly reduces the adverse effect of the difficulty negative sample on the model, significantly increases the feature difference between the positive and negative sample pairs, and strengthens the discrimination of the model. In the hash learning stage, in order to prevent gradient disappearance, the sign function is used for forward propagation of quantization error, and the automatic weighted contrastive loss and quantization loss are introduced to jointly optimize the model, so that the model can generate high-quality hash code for remote sensing image retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to better describe the embodiments of the application, the following provides a brief description of the embodiments, which together with the specification is used to explain the principles of the application.

[0047] Figure 1 The two-stage flow chart of the contrastive hashing network model based on automatic weight allocation in the embodiment of the application.

[0048] Figure 2 The remote sensing image retrieval flow chart in the embodiment of the application.

[0049] Figure 3 The UCM data set category example graph in the embodiment of the application.

[0050] Figure 4 The AID data set category example graph in the embodiment of the application. DETAILED DESCRIPTION

[0051] In order to enable the above-mentioned objects, features and advantages of the present application to be more clearly understood, the present application will be further described below in conjunction with the accompanying drawings and specific embodiments. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways beyond the specific embodiments described herein, and the present application is not limited to the specific embodiments disclosed below.

[0052] The embodiment of the present application provides a remote sensing image retrieval optimization method based on automatic weight distribution and contrastive hashing, and specifically comprises the following steps:

[0053] S1: Construct a contrastive hashing network model based on automatic weight distribution, wherein the contrastive hashing network model based on automatic weight distribution mainly comprises a feature learning stage and a hash learning stage. First, the feature learning stage uses a data enhancement module and an encoder module to obtain features of different views. At the same time, the feature learning stage innovatively introduces a Gaussian weighting and a dynamic adjustment strategy to propose an automatic weighted contrastive loss for adjusting the network model, so that the weight matrix can express the similarity of different sample pairs and identify and reduce the interference of difficult negative sample pairs on model training. Finally, the hash learning stage introduces a hash algorithm into the network model, and combines the automatic weighted contrastive loss and a quantization loss to optimize the contrastive hashing network model based on automatic weight distribution.

[0054] The structure of the contrastive hashing network model based on automatic weight distribution constructed by the present application is as shown in Figure 1The feature learning stage includes a data augmentation module, an encoder module, and an automatic weighted contrastive loss. In the feature learning stage, N remote sensing images in each batch are processed by the data augmentation module to obtain different augmented views, and the encoder module is used to extract latent features from the augmented views. Then, the similarity between the features is calculated to obtain a similarity matrix S, in which the elements on the main diagonal are the similarities of positive sample pairs, and the other elements are the similarities of negative sample pairs. The similarity matrix S is weighted by a Gaussian function to obtain a weight matrix W, in which the elements on the main diagonal are the weight values of positive sample pairs, and the other elements are the weight values of negative sample pairs. When the similarity value of a negative sample pair reaches a threshold, it is defined as a hard negative sample. The weight values of hard negative samples in the weight matrix are dynamically adjusted to the same minimum value, and a weighted weight matrix is obtained (the hard negative samples are marked with black blocks in the matrix). Next, the weighted weight matrix is used to optimize the contrastive loss function, and an automatic weighted contrastive loss is designed. The automatic weighted contrastive loss is used to train and adjust the network model based on the automatic weight distribution of the contrastive hashing network, and the network model parameters are fine-tuned. The hash learning stage shares the data augmentation module and the encoder module with the feature learning stage, and also includes a hash layer and a joint loss backpropagation training network model. In the hash learning stage, the fine-tuned network model parameters obtained in the feature learning stage are loaded, and the images are input into the trained and adjusted network model for feature extraction to obtain optimized image features and embedding vectors. The hash layer is used to convert the optimized embedding vectors into continuous hash codes, and the continuous hash codes are used to calculate the optimized automatic weighted contrastive loss. Then, the continuous hash codes are quantized into binary hash codes, and the backpropagation function is used to avoid gradient vanishing in the network to obtain two sets of discrete binary hash codes. The quantization loss in the quantization process is calculated based on the two sets of binary hash codes, and the joint total loss function is calculated based on the optimized automatic weighted contrastive loss and the quantization loss. The network model is trained and learned using the total loss function to obtain the trained network model parameters. Finally, the trained network model is applied to remote sensing image retrieval.

[0055] S11: In the network model provided by the present application, the feature learning stage includes three components, namely a data augmentation module, an encoder module, and a training and adjustment using an automatic weighted contrastive loss.

[0056] S111: The data augmentation module generates two different augmented views for each of the N input images in the same batch in the training set. A set of rich data augmentation strategies is introduced to generate two different augmented views for each image And The input image x i The two generated augmented views defined as similar views, i.e. positive sample pairs, similarly, input images x j two generated augmented views also belong to similar views, as positive sample pairs. While and represent dissimilar view pairs, i.e. negative sample pairs, generated from different input images, where i≠j. The data augmentation strategy adopted in this embodiment includes random cropping, scaling, horizontal flipping, random color jittering, random grayscale and random Gaussian blur, aiming to increase the generalization ability of the network model.

[0057] S112: encoder module, the automatic weight allocation based contrastive hashing network model of the present application is provided with two parallel and weight sharing branches to jointly learn the representation of input image data. The core component of the two-stage network model is the encoder module. The encoder module in this embodiment not only integrates a powerful backbone network for basic feature extraction, but also adds a nonlinear projector, the purpose of which is to further map the extracted high-level features to an embedding space suitable for self-supervised learning. The backbone network f(·) usually uses a convolutional neural network (CNN), and in this paper, the ResNet50 model in the convolutional neural network category is selected as the backbone network. ResNet-50 is a deep residual network with strong modeling ability and good feature extraction ability. The backbone network f(·), i.e. the ResNet50 model, includes an input layer, an initial convolutional layer, a residual block, a global average pooling layer and a fully connected layer, which are used for feature extraction to obtain the feature representation of the image. The nonlinear projector g(·) (i.e. Projection) uses a multilayer perceptron (MLP) with one hidden layer to map the feature representation extracted by the backbone network f(·) to the space where the contrastive loss is applied. The nonlinear projector is composed of a fully connected layer, a ReLU activation function and another fully connected layer connected in turn.

[0058] each pair of augmented views and are input into the backbone network for feature extraction to obtain their respective feature representations and where N is the batch size. These feature representations are the results output by the backbone network f(·) via the average pooling layer, and the feature dimension is 2048. Next, the nonlinear projector g(·) maps the feature representations and to a low-dimensional embedding space to obtain embedding vectors and with a feature dimension of 256. This conversion process can be summarized as follows:

[0059]

[0060] wherein, represents an enhanced view the embedding vectors obtained after mapping, represents an enhanced view the embedding vectors obtained after mapping; the mapping process corresponding to f(·) is represents a backbone network mapping, and the mapping process corresponding to g(·) is represents a nonlinear projector mapping, H and W are the height and width of the input image, and C is the number of channels.

[0061] S113: training and parameter adjustment using an automatic weighted contrast loss. The core of the contrast loss is to minimize the distance between positive sample pairs and maximize the distance between positive and negative sample pairs. According to the definition of positive and negative samples, positive sample pairs usually have higher similarity. Therefore, in order to further optimize the loss function, this paper proposes an automatic weighted loss, which can highlight the learning of more valuable sample pairs and weaken the interference of difficult negative sample pairs on the network model, and maintain the stability of the network model learning. The specific process is as follows:

[0062] 1) Similarity calculation: for N input image samples in the same batch Two embedding vector sets of image features are obtained through the data enhancement module and the encoder module respectively and The cosine similarity of the two embedding vector sets is calculated to obtain a similarity matrix S={s ij}∈R N×N , R represents a real number set, and s ij represents the similarity value between the enhanced view of the i-th input image and the enhanced view of the j-th input image. .

[0063] 2) Gaussian weighting: the similarity relationship of sample pairs is converted into a weight factor through Gaussian weighting technology. Higher weight is given to positive sample pairs with high similarity, and lower weight is given to negative sample pairs with low similarity, thereby forming a decreasing weight distribution pattern. This mechanism plays a dual effect in the optimized network model of the present application. Not only does it strengthen the intimacy between positive sample pairs, but it also promotes their feature vector distance to be more closely, while increasing the discrimination between positive and negative sample pairs, pushing away the feature difference between them, and deepening the interval. Gaussian weighting is based on the function form of Gaussian distribution. In this embodiment, the similarity matrix S is weighted to obtain a weight matrix W through a Gaussian function. The mathematical expression of the weight value in the weight matrix W is as follows:

[0064]

[0065] wherein, w ij ​represents the weight value in the i-th row and j-th column of the weight matrix W. μ is the mean value, representing the center position of the Gaussian distribution curve, i.e. the central tendency of the data distribution. σ is the standard deviation, used to measure the dispersion degree of the data from the mean value.

[0066] 3) Dynamic adjustment: adjusting the input image x i The two generated enhanced views are defined as similar views, i.e. positive sample pairs. While and represent the enhanced views generated from different input images, which are defined as dissimilar view pairs, i.e. negative sample pairs, where i≠j. However, if different views of the same image are simply identified as positive samples, and all views of different images are identified as negative samples, the significant similarity between different images within the same class is easily ignored, resulting in reduced discrimination accuracy of the model. In this embodiment, the samples with the highest similarity ranking are selected from the negative sample set and defined as difficult negative samples, and the weight of the difficult negative samples is reduced to reduce the interference on the network model training. The higher the similarity, the higher the probability that the difficult negative sample is an image of the same class as the input image to be queried. Therefore, a similarity threshold λ is set, and samples with a similarity greater than the threshold are selected as difficult negative samples, and the weight value φ corresponding to the difficult negative samples is set to be small enough, i.e. to a value close to 0. The weighted weight matrix is obtained, so as to indirectly reduce the interference of the difficult negative samples on the model training. The mathematical expression for dynamically adjusting the weight value is as follows:

[0067] w i ′ j = w ij · H(s ij - λ) + φ · (1 - H(s ij - λ)) (3)

[0068] In the formula, H(ξ) represents an indicator function, H(ξ) = 1 when ξ≥0, and H(ξ) = 0 when ξ<0. w i ′ j represents the weight value in the i-th row and j-th column of the adjusted weighted weight matrix, corresponding to the weight value of the negative sample pair composed of the i-th input image and the j-th input image after weighting. w ij represents the weight value in the i-th row and j-th column of the weight matrix W before adjustment, corresponding to the weight value of the negative sample pair composed of the i-th input image and the j-th input image before adjustment. φ represents the weight value of the difficult negative sample, which should be set to be close to 0. λ represents the similarity threshold.

[0069] 4) Propose an Automatic Weighted Contrast Loss: This method multiplies the similarity of positive and negative sample terms in the contrast loss by their corresponding weighted values. This automatically adjusts the contribution of different samples to the network model's learning, further narrowing positive sample pairs and widening negative sample pairs. Simultaneously, it assigns lower weights to difficult-to-negative samples, reducing their interference with the network model's learning and helping the network model learn features from the samples more evenly. The Automatic Weighted Contrast Loss is defined as follows:

[0070]

[0071] Among them, L AWC (i) represents the automatically weighted contrastive loss, and sim(·,·) represents the cosine similarity. This indicates that the i-th input image is in the enhanced vision Figure 1 or enhance vision Figure 2 The following features This indicates that the j-th input image is in the augmented vision. Figure 1 or enhance vision Figure 2 The following features, w i ′ i w represents the weight value of the positive sample pair of the i-th input image after weighting. i ′ j τ represents the weight value of the negative sample pair formed by the i-th input image and the j-th input image after weighting, τ represents the temperature coefficient, and exp(·) represents the exponential function with the constant e as the base.

[0072] 5) Use automatic weighted contrast loss L AWC (i) Train the contrastive hash network model based on automatic weight allocation once to obtain fine-tuned network model parameters.

[0073] S12: In the network model provided by this invention, the hash learning stage includes four components: a data augmentation module, an encoder module, a hash layer, and a backpropagation training network model based on the joint loss of quantization loss and automatic weighted contrast loss. The data augmentation module and the encoder module are shared with the feature learning stage and directly share the information obtained by the data augmentation module and the encoder module in the feature learning stage.

[0074] S121: In the hash learning phase, it is first necessary to load and freeze the fine-tuned network model parameters obtained in the feature learning phase, and then use the trained and fine-tuned network model to re-acquire the input image from the input image. Two enhanced views and And to obtain the new embedding vectors obtained by training and fine-tuning the data augmentation module and encoder module. and

[0075]

[0076] S122: Then, the embedding vectors obtained in the hash learning stage are reduced in dimensionality by the hash layer. The hash layer is defined as follows:

[0077]

[0078] The hash layer fc(·) uses fully connected layers to generate hash codes. Fully connected layers can convert embedding vectors into hash codes. and embedding vector Dimensionality reduced to K-dimensional continuous hash representation and

[0079] S123: Rewrite the automatically weighted comparison loss using consecutive hash codes as follows:

[0080]

[0081] In the formula, This represents the rewritten automatic weighted comparison loss. This indicates that the i-th input image is in the enhanced vision Figure 1 or enhance vision Figure 2 The following consecutive hash codes, This indicates that the j-th input image is in the augmented vision. Figure 1 or enhance vision Figure 2 The following is a continuous hash code; w i ′ i This represents the weight value of the positive sample pair of the i-th input image after weighting.

[0082] The sequential hash code is quantized into a binary hash code using the sign function sgn(·), which can be represented as:

[0083]

[0084] In the formula, b i Represents binary hash code, This represents the consecutive hash code of the i-th input image.

[0085] At this time, after mapping the continuous hash code into discrete binary hash code, the traditional back propagation method is used to optimize the parameters of continuous numerical value, but it cannot directly process the discrete output of binary, which means that we need to solve the problem of how to make the network output discrete hash code in the process of gradient descent. Most previous work chooses to use continuous relaxation method for binary hash optimization, replacing the discrete sign function with smooth activation function (such as tanh or sigmoid). These relaxation schemes will introduce quantization error, resulting in suboptimal hash code. Some work strictly uses the sign function to propagate the quantization error forward, and then transmits the gradient to the previous layer network structure, effectively preventing gradient disappearance, and updating all bits together. The present invention uses the definition of back propagation function to avoid the problem of gradient disappearance in the network. The definition of back propagation function is as follows:

[0086]

[0087] Where b t+1 represents the binary hash code of the t+1th gradient, b t represents the binary hash code of the tth gradient, L represents the loss function, θ represents the model parameters used in the tth gradient descent iteration, and r represents the model learning rate used in the tth gradient descent iteration. Hash represents And hash represents The binary hash code obtained by mapping through the sign function is And the binary hash code b i 2 ;

[0088] Then according to the obtained binary hash code And the binary hash code b i 2 The embedding vector obtained in the hash learning stage is used to calculate the quantization loss in the quantization process, and the quantization loss is defined as:

[0089]

[0090] Where L Q represents the quantization loss, and sim(·,·) represents the cosine similarity.

[0091] S124: Combine the rewritten automatic weighted contrastive loss and the quantization loss to obtain the total loss function:

[0092]

[0093] Where L total represents the total loss function, which is used for training the network model in the hash learning stage, N represents the batch size, and i represents the ith original input image.

[0094] S125: utilize the total loss function L total The automatic weight distribution based contrastive hashing network model is trained and learned to obtain trained network model parameters.

[0095] S2: Finally, the automatic weight distribution based contrastive hashing network model proposed in the application is applied to a remote sensing image retrieval task. The two most critical problems in remote sensing image retrieval are feature extraction and similarity measurement. The former is dedicated to mining key image information into high-quality features, and the latter measures the similarity between features to judge the difference between images. The application constructs a high-performance hashing network model through a carefully designed two-stage training strategy. The network model can quantize high-dimensional real value features into low-dimensional discrete hash codes to improve the retrieval efficiency of remote sensing images.

[0096] The remote sensing image retrieval process is shown in Figure 2 The automatic weight distribution based contrastive hashing network model parameters trained by the two-stage strategy are loaded and frozen, and the image to be queried and the remote sensing dataset images are input into the trained automatic weight distribution based contrastive hashing network model to extract the discrete binary hash codes of the image to be queried and the remote sensing dataset images. The Hamming distance between the image to be queried and the remote sensing dataset images is calculated through the discrete binary hash codes to measure the similarity between the image to be queried and the remote sensing dataset images. The Hamming distance can quantify the difference between two hash codes, and the smaller the Hamming distance, the more similar the two images. The Hamming distance is sorted from small to large, and the remote sensing dataset image with the smallest Hamming distance is placed in the front, indicating that it is the most similar to the image to be queried. Finally, the remote sensing dataset image with the smallest Hamming distance is output, and the most similar remote sensing dataset image to the image to be queried is obtained, and the retrieval result is obtained.

[0097] The embodiments of the application are evaluated on two general remote sensing datasets. The UCM dataset is a land use image dataset for research purposes, which contains 2100 images, including 21 scenes such as agriculture, airplane, baseball diamond, etc., 100 pictures for each type, and each picture is 256*256 pixels in size. Its example image is shown in Figure 3 The AID dataset collects high-resolution aerial remote sensing images from different regions, times and perspectives, and the dataset contains 10000 images, with inconsistent number of images in each category, including more than 30 categories such as city, farmland, forest and industrial area, and each image is 600*600 pixels in size. Its example image is shown in Figure 4 .

[0098] The method is used for remote sensing image retrieval by using some existing hash algorithms. Six existing competitive or classic unsupervised deep hash methods are re-implemented according to the open source code provided by the original author, including BiHalf algorithm, CIBHash algorithm, SDC algorithm, SDC_SimCLR algorithm, GreedyHash algorithm and DCSH algorithm. At the same time, the CIBHash algorithm, the SDC_SimCLR algorithm and the DCSH algorithm are all unsupervised hash algorithms based on contrast learning structure. The experimental results are shown in the following tables.

[0099] In terms of qualitative retrieval results, Tables 1 and 2 show the mAP (mean average precision) retrieval results of the present application compared with other unsupervised hash methods on the UCM and AID datasets. On the two datasets, we compared the retrieval performance of 16bit, 32bit, 64bit and 128bit hash codes respectively. Most hash methods obtain higher average precision with increasing bit number, which shows that the length of hash code has a great influence on the retrieval performance. Longer hash code can better preserve image feature information and improve retrieval performance. The present application shows higher retrieval accuracy on the UCM and AID datasets, and the average precision is much higher than that of other methods. Especially in 64bit and 128bit, the average precision of the present application reaches 80.46% and 77.80% respectively on the UCM dataset, and reaches 84.27% and 84.62% respectively on the AID dataset. It shows that the present application has effectiveness and stability in the task of remote sensing image hash retrieval.

[0100] Table 1 Comparison of average precision (mAP) retrieval results of different hash methods on UCM dataset

[0101]

[0102] Table 2 Comparison of average precision (mAP) retrieval results of different hash methods on AID dataset

[0103]

[0104] Table 3 and Table 4 show the ANMRR retrieval results comparison of different hash methods on UCM and AID datasets. According to the experimental results, on UCM and AID datasets, the method of the present application performs well at all bit numbers, with lower ANMRR values (average normalized modified retrieval rank), which indicates that the method of the present application has better performance in image retrieval tasks and can effectively maintain the order of retrieval results. In contrast, other hash methods perform relatively poorly, especially at shorter hash bit numbers, with higher ANMRR values, indicating a decline in retrieval performance. Therefore, the experimental results show that the method of the present application has stable performance on different datasets and bit numbers and can be used as a reliable image retrieval method.

[0105] Table 3 ANMRR retrieval results comparison of different hash methods on UCM dataset

[0106]

[0107] Table 4 ANMRR retrieval results comparison of different hash methods on AID dataset

[0108]

[0109] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An optimized method for remote sensing image retrieval based on automatic weight assignment and contrastive hashing, characterized in that, The method comprises the following steps: S1: constructing an automatic weight distribution based contrastive hashing network model, which comprises a feature learning stage and a hash learning stage; S11: the feature learning stage comprises a data enhancement module, an encoder module, and an automatic weighted contrastive loss; the data enhancement module is used to obtain remote sensing image data and obtain enhanced views of each input image, and the encoder module is used to extract deep partitioned image features from the enhanced views; The automatic weighted contrastive loss is calculated by dynamically adjusting the weight factor, and is used to train and parameterize the network model, so as to obtain fine-tuned network model parameters; S12: the hash learning stage shares the data enhancement module and the encoder module with the feature learning stage, and further comprises a hash layer and a joint loss; the hash learning stage first loads the fine-tuned network model parameters obtained in the feature learning stage, inputs the image into the trained and parameterized network model for feature extraction, and obtains optimized image features and embedding vectors; the hash layer converts the optimized embedding vectors into continuous hash codes, and calculates the optimized automatic weighted contrastive loss using the continuous hash codes; then, the continuous hash codes are quantized into binary hash codes to obtain two sets of discrete binary hash codes; the quantization loss in the quantization process is calculated according to the two sets of binary hash codes, and the joint total loss function is calculated by the optimized automatic weighted contrastive loss and the quantization loss; the network model is trained and learned using the total loss function, and trained network model parameters are obtained; S2: applying the trained network model to remote sensing image retrieval, querying the most similar remote sensing dataset image to the image to be queried from the remote sensing dataset through the network model, and outputting the retrieval result. 2.The method of claim 1, wherein, Each remote sensing image data generates two enhanced views with different perspectives after the data enhancement module, and the data enhancement module enhances the image data by random cropping, scaling, horizontal flipping, random color jittering, random grayscale, and random Gaussian blur.

3. The method of claim 2, wherein, The encoder module comprises a backbone network and a nonlinear projector, the backbone network is a ResNet50 model, and the backbone network, i.e., the ResNet50 model, comprises an input layer, an initial convolutional layer, a residual block, a global average pooling layer, and a fully connected layer, and is used for feature extraction to obtain high-dimensional feature representation of the image; the nonlinear projector uses a multilayer perceptron with one hidden layer to map the feature representation extracted by the backbone network to a space where the contrastive loss is applied, and converts to obtain low-dimensional feature representation of the image; the nonlinear projector is composed of a fully connected layer, a ReLU activation function, and another fully connected layer connected in sequence.

4. The method of claim 3, wherein, The specific process of training and parameterizing the network model by using the automatic weighted contrastive loss is as follows: through the data enhancement and the encoder module, two sets of embedding vector sets of the same batch of input images can be obtained, the embedding vectors from the same image are defined as positive sample pairs, and the embedding vectors from different images are defined as negative sample pairs; the cosine similarity between the two sets of embedding vectors is calculated to obtain a similarity matrix; The similarity relationship of the sample pairs is converted into weight factors by using a Gaussian weighting technique, and then a weight matrix is obtained; a similarity threshold is set Samples with a similarity greater than the threshold value are selected from the negative sample set and defined as difficult negative samples, and the weight values corresponding to the difficult negative samples are The weight matrix is obtained by setting the threshold value to be small enough and infinitely close to 0; the positive and negative sample items are multiplied by their corresponding weighted weight values to obtain an automatically weighted contrast loss; the automatically weighted contrast loss is used to train the feature learning stage of the contrast hashing network model based on automatic weight distribution, and the fine-tuning network model parameters are obtained.

5. The method of claim 4, wherein, The hash layer uses a full connection layer to reduce the embedding vector output by the encoder module into a K-dimensional continuous hash code.

6. The method of claim 5, wherein, In the hash learning stage, the continuous hash code is quantized into a binary hash code using a sign function sgn(·).

7. The method of claim 6, wherein, The specific steps of calculating the total loss function after jointing include: splicing and fusing the discrete binary hash codes corresponding to the two enhanced views; calculating the quantization loss between the low-dimensional embedding vector output by the encoder module and the binary hash code; at the same time, calculating the optimized automatically weighted contrast loss using the continuous hash code; adding the quantization loss and the optimized automatically weighted loss to obtain the total loss function after jointing, which is used to train and learn the network model in the hash learning stage.

8. The method of claim 7, wherein, The specific process of step S2 is: first, load and freeze the parameters of the contrast hash network model based on automatic weight allocation trained by the two-stage strategy, then input the to-be-queried image and the remote sensing dataset image into the trained contrast hash network model based on automatic weight allocation, and extract the discrete binary hash code of the to-be-queried image and the remote sensing dataset image; Then, the Hamming distance between the to-be-queried image and the remote sensing dataset image is calculated through the discrete binary hash code, and the similarity between the to-be-queried image and the remote sensing dataset image is measured; the Hamming distance can quantify the difference between two hash codes, and the smaller the Hamming distance, the more similar the two images are; the Hamming distance is sorted from small to large, and the remote sensing dataset image with the smallest Hamming distance will be ranked first, indicating that it is the most similar to the to-be-queried image; finally, the remote sensing dataset image with the smallest Hamming distance is output, and the most similar remote sensing dataset image to the to-be-queried image is obtained, and the retrieval result is obtained.

Images