Vector quantization-based low-quality remote sensing image deep hash retrieval method, system and equipment and medium

Abstract

The invention discloses a low-quality remote sensing image deep hash retrieval method, system and device based on vector quantization and a medium, and the method comprises the steps: firstly obtaining high-quality and low-quality remote sensing images as input, respectively extracting the features of the high-quality and low-quality remote sensing images through a deep convolutional neural network, and inputting the features into a vector quantization module; the method comprises the following steps of: quantifying the image data into an independent discrete space, generating quantized features, generating a hash code for image retrieval through a deep hash network, finally obtaining feature representation, and constraining the feature representation by applying a loss function which comprises Pairwise Loss, reconstruction loss and cross entropy loss. The collaborative learning module is used for ensuring semantic information retention, feature distance constraint and collaborative learning of an encoder, a codebook and a decoder; the system, the equipment and the medium are used for implementing the method. According to the method, the low-quality remote sensing image retrieval precision is improved, the storage space occupation and the calculation overhead are reduced, the robustness and the generalization ability of the model are improved, and the relatively high retrieval performance is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval and image processing technology, and in particular to a method, system, device and medium for deep hash retrieval of low-quality remote sensing images based on vector quantization technology, aiming to improve the performance of low-quality images in remote sensing image retrieval tasks. Background Art

[0002] Deep learning has shown significant advantages in computer vision tasks, but most current visual tasks often rely on high-quality images for training and testing. However, in practical applications, image acquisition is often affected by a variety of factors, such as sensor noise, camera motion blur, and atmospheric interference, resulting in image quality degradation. These low-quality images pose a challenge to the performance and generalization ability of the model, because deep learning models have difficulty in accurately extracting key features from low-quality images.

[0003] In existing research, the performance of low-quality images is mainly improved through feature distillation methods, that is, by allowing deep neural networks to extract features similar to high-quality images from low-quality images, improve the feature performance of low-quality images and apply them to downstream tasks. In addition, some studies use image restoration methods to first restore low-quality images to high-quality images, and then perform feature extraction and subsequent task processing. These methods have also achieved certain results.

[0004] In the field of remote sensing image retrieval, most existing research focuses on the retrieval of high-quality remote sensing images, and less attention is paid to low-quality remote sensing images. However, in practical applications, remote sensing images are often affected by a variety of factors, such as weather, environment, and equipment noise, which lead to the degradation of remote sensing image quality and thus affect the retrieval performance.

[0005] The Chinese patent application with publication number CN 118312636 A discloses a method for marine remote sensing ship image retrieval based on self-attention hashing. The method uses the self-attention mechanism to generate hash codes, thereby improving the accuracy and efficiency of image retrieval. However, since the method only focuses on remote sensing image retrieval under ideal conditions, it fails to consider the impact of complex environments (such as weather changes, sea conditions, etc.) on remote sensing images, and has the disadvantage of being unable to overcome the challenges of image retrieval in complex environments.

[0006] Liu Chao et al. published "Deep hash learning for remote sensing image retrieval" [J] (C. Liu, J. Ma, X. Tang, F. Liu, X. Zhang, and L. Jiao. Deep hash learning for remote sensing image retrieval [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, Vol. 59 (Issue 4): 3420-3443). This paper proposes a remote sensing image retrieval method based on deep hash learning, which uses deep neural networks for feature extraction and generates compact hash codes to improve retrieval efficiency. However, since this method fails to fully consider the quantization error of low-quality image features in the hash code generation process, the retrieval accuracy of low-quality images decreases, and it has the disadvantage of being unable to maintain consistency between low-quality images and high-quality images.

[0007] Song Wei et al. published "Asymmetric hash code learning for remote sensing image retrieval" [J] (W. Song, Z. Gao, R. Dian, P. Ghamisi, Y. Zhang, and J. A. Benediktsson. Asymmetric hash code learning for remote sensing image retrieval [J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, Vol. 60: 1-14). The paper proposed an asymmetric hash code learning method to improve the accuracy of remote sensing image retrieval by generating different hash codes for different modal data. However, since this method is insensitive to modal differences when processing low-quality remote sensing images, the retrieval performance of low-quality images is reduced, and it has the disadvantage of not being able to fully utilize multimodal information for accurate retrieval. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a deep hash retrieval method, system, device and medium for low-quality remote sensing images based on vector quantization. By introducing vector quantization into the deep hash retrieval task of low-quality remote sensing images, the essential characteristics of low-quality remote sensing images are learned, and a codebook of high-quality remote sensing images and low-quality remote sensing images is constructed. The three loss functions of pairwise loss, cross entropy loss and reconstruction loss are used to realize the mapping of low-quality remote sensing image features to high-quality remote sensing image features, thereby improving the retrieval performance of low-quality remote sensing images.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A deep hash retrieval method for low-quality remote sensing images based on vector quantization includes the following steps:

[0011] Step 1: Image feature extraction: obtain high-quality remote sensing images and low-quality remote sensing images as input, and extract the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks;

[0012] Step 2: Vector quantization: Input the features extracted in step 1 into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features;

[0013] Step 3: Hash code generation: The quantized features generated in step 2 are used through a deep hash network to generate a hash code for image retrieval;

[0014] Step 4: Application of loss function: After feature extraction, feature quantization and hash code generation in steps 1 to 3, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss, which are used to ensure semantic information retention, feature distance constraints and collaborative learning of encoder, codebook and decoder.

[0015] The specific steps of step 1 are as follows:

[0016] High-quality remote sensing images are represented by X = {x1, x2, …, x n} and the corresponding low-quality remote sensing image is expressed as At the same time, it is input into the backbone network AlexNet, and the image features are extracted through the backbone network AlexNet. After feature extraction, the dimension is reduced through two layers of fully connected layers to generate a real-valued hash code. The real-valued hash code of a high-quality remote sensing image is represented by h = {h1,h2,…,h n}, the real-valued hash code of the low-quality remote sensing image is represented as Among them, h and All belong to R K, K is the number of bits of the real-valued hash code, and the formula for generating the real-valued hash code is as follows:

[0017] h i =Φ(x i ;θ),i=1,2,…,N

[0018]

[0019] Among them, θ represents the weight parameters of the backbone network AlexNet, two fully connected layers and the hash layer, and Φ represents a mapping function represented by θ, which includes convolution, pooling and fully connected operations.

[0020] The specific steps of step 2 are as follows:

[0021] Furthermore, in order to obtain the quantized image features, the vector quantization steps are as follows:

[0022] The two learnable codebooks E∈R m×K Added at the end of the feature extraction network, it is used to quantify the features of low-quality remote sensing images and high-quality remote sensing images, respectively, where m is the number of codewords in the codebook, e∈R K is the codeword, that is, the codebook E∈R m ×K The real-valued hash codes of high-quality remote sensing images and low-quality remote sensing images are vector quantized through the corresponding codebook to obtain quantized features. For any real-valued hash code h of a high-quality remote sensing image, its standard quantization process is expressed as:

[0023] s=E(h)=e q

[0024]

[0025] Where E(h) represents discretizing h into a codeword in the codebook, e q represents the qth codeword in the codebook, q is the codeword index with the smallest Euclidean distance to the real-valued hash code h in the codebook E, and then the first L codewords with the closest Euclidean distance to the real-valued hash code are selected for weighted summation to supplement the information of the quantitative features and ultimately improve its representation ability. Specifically, for any real-valued hash code h of a remote sensing image, the first L codewords with the closest Euclidean distance to it are {e1,e2,…,e L}, and the distance from the real-valued hash code h of the high-quality remote sensing image is {d1,d2,…,d L}, and calculate the weights {β1,β2,…,β L}, the final formula of the quantized vector z is:

[0026]

[0027] After vector quantization, the quantitative features of high-quality remote sensing images are expressed as Z = {z1,z2,…,z n}, the quantitative features of low-quality remote sensing images are expressed as

[0028] The specific steps of step 3 are as follows:

[0029] Furthermore, in order to obtain a hash code that can be used for retrieval, the hash code generation steps are as follows:

[0030] In order to reduce the quantization error in the process of converting real-valued hash codes into hash codes, after the real-valued hash codes are generated, the value of each dimension of the real-valued hash codes is limited to between -1 and 1 through the activation function tanh(·). The specific formula is:

[0031]

[0032] Real-valued hash code z for high-quality remote sensing images i , its normalized hash code v i It is expressed as:

[0033] v i =tanh(z i ),i=1,2,…,N

[0034] Similarly, for the real-valued hash codes of low-quality remote sensing images Its normalized hash code It is expressed as:

[0035]

[0036] In order to assist the learning of the codebook in the deep hashing network, the quantized features are constrained by the cross entropy loss through the classification layer to inject semantic information. The specific formula is:

[0037] p i =fc(v i ),i=1,2,…,N

[0038]

[0039] Among them, fc(·) represents the classification layer, p i represents the classification vector of the i-th high-quality remote sensing image, Represents the classification vector of the i-th low-quality remote sensing image;

[0040] The process of generating a hash code is to pass the vector p i and Binarization is performed to generate the final high-quality remote sensing image hash code b iand hash codes for low-quality remote sensing images The specific formula is:

[0041]

[0042] b i =sign(v i ),i=1,2,…,N

[0043]

[0044] Among them, b i represents the hash code of the i-th high-quality remote sensing image, v i is its corresponding normalized real-valued hash code; similarly, represents the hash code of the i-th low-quality remote sensing image, is the normalized real-valued hash code of its corresponding low-quality remote sensing image. The real-valued hash code is converted into binary form by the sign(·) function to generate the final hash code for retrieval.

[0045] The specific steps of step 4 are as follows:

[0046] In order to ensure the preservation of semantic information of remote sensing image feature representation, the constraint of feature distance, and the collaborative learning of encoder, codebook and decoder, the loss function application steps are as follows:

[0047] The Pairwise Loss function is used to constrain the feature space learning of high-quality remote sensing images and low-quality remote sensing images, ensuring that the features of the same type of images are closer and the features of different types of images are farther apart. Specifically, Pairwise Loss is used in the following three situations:

[0048] Case 1: Embedding space constraints of high-quality remote sensing images: It is used to constrain the feature vectors of high-quality remote sensing images so that the feature distances between high-quality remote sensing images of the same type are closer and the feature distances between high-quality remote sensing images of different types are farther. The loss function formula is as follows:

[0049]

[0050] Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images, v i and v j are the feature vectors of low-quality remote sensing images;

[0051] Case 2: Embedding space constraints of low-quality remote sensing images: It is used to constrain the features of low-quality remote sensing images so that the feature distances between low-quality remote sensing images of the same type are closer, and the feature distances between low-quality remote sensing images of different types are farther. The loss function formula is as follows:

[0052]

[0053] Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images. and are the feature vectors of low-quality remote sensing images;

[0054] Case 3: Pairing constraints of high-quality remote sensing image features and low-quality remote sensing image features: It is used to bring high-quality remote sensing image features and low-quality remote sensing image features of the same category closer to each other, and keep image features of different categories away. The loss function formula is as follows:

[0055]

[0056] Among them, φ ij Represents the similarity indicator matrix of high-quality remote sensing images and low-quality remote sensing images. When φ ij =1, it means that the i-th high-quality remote sensing image and the j-th low-quality remote sensing image belong to the same category; when φ ij =0, it means they belong to different categories;

[0057] The overall Pairwise Loss loss function can be expressed as:

[0058] L pairwise =L pairwise-high +L pairwise-low +L pairwise-hl

[0059] The cross entropy loss function is used to guide the learning of the feature extraction network and the codebook to ensure that the quantized features of high-quality remote sensing images and low-quality remote sensing images retain their semantic information. Specifically, the cross entropy loss function is used to optimize the classification performance of high-quality remote sensing images and low-quality remote sensing images. The formula is as follows:

[0060]

[0061] Among them, y ic is the true label of the i-th high-quality remote sensing image for category c, p ic is the predicted probability of category c for high-quality remote sensing images; is the true label of the i-th low-quality remote sensing image for category c, is the predicted probability of low-quality remote sensing images for category c; C is the number of categories;

[0062] The reconstruction loss function is used to guide the collaborative learning of the feature extraction network and the codebook. Since the argmin operation in vector quantization is not differentiable, the straight-through estimator (STE) method is used to bypass the codebook and pass the gradients brought by the pairwise loss and cross entropy loss behind the codebook to the front feature extraction network, thereby realizing the parameter update of the feature extraction network. The specific loss function formula is as follows:

[0063] L commitment =|h-sg(e q )| 2

[0064] Here, sg(·) refers to the stop gradient operation, which is used to prevent the gradient from propagating in the codebook. Although the direct estimation method can realize gradient transmission, the codebook cannot directly receive the gradient information from the pairwise loss and cross entropy loss, which limits the codebook learning.

[0065] In order to allow the codebook space to participate in learning normally, a dictionary learning method is adopted to optimize the learning process of the codebook by calculating the L2 error between the features output by the feature extraction network and the corresponding quantized codewords. The specific loss function formula is as follows:

[0066] L vq =|sg(h)-e q | 2

[0067] Among them, h is the output of the feature extraction network, e q is the corresponding codeword;

[0068] The overall optimization problem of the model optimizes the learnable parameters in the network framework by combining all loss functions. The overall loss function formula is as follows:

[0069]

[0070] Among them, Θ represents all the learnable parameters in the network framework of this chapter.

[0071] A deep hash retrieval system for low-quality remote sensing images based on vector quantization, comprising:

[0072] Image feature extraction module: It obtains high-quality remote sensing images and low-quality remote sensing images as input, and extracts the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks;

[0073] Vector quantization module: Input the features into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features;

[0074] Hash code generation module: generates hash codes for image retrieval through deep hashing network.

[0075] Loss function application module: After feature extraction, feature quantization and hash code generation, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss to ensure semantic information retention, feature distance constraints and collaborative learning of encoder, codebook and decoder.

[0076] A low-quality remote sensing image deep hash retrieval device based on vector quantization, comprising:

[0077] Memory: used for storing a computer program for implementing the deep hash retrieval method for low-quality remote sensing images based on vector quantization;

[0078] Processor: used to implement the deep hash retrieval method for low-quality remote sensing images based on vector quantization when executing the computer program.

[0079] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a deep hash retrieval method for low-quality remote sensing images based on vector quantization are implemented.

[0080] Compared with the prior art, the beneficial effects of the present invention are:

[0081] 1. The present invention maps the features of low-quality remote sensing images into discrete space through vector quantization, thereby improving the retrieval accuracy of low-quality remote sensing images and effectively reducing quantization errors.

[0082] 2. The present invention encodes quantitative features through a deep hash network to generate compact hash codes, thereby greatly reducing storage space occupancy and computational overhead during image retrieval.

[0083] 3. The present invention adopts a combination of pairwise loss, cross entropy loss and reconstruction loss, which effectively ensures the feature consistency between low-quality remote sensing images and high-quality remote sensing images, and improves the robustness and generalization ability of the model.

[0084] 4. The present invention adopts a combination of vector quantization and deep hash learning, especially for low-quality remote sensing images, and can still maintain a high retrieval performance in complex environments (such as weather, noise and other conditions), overcoming the shortcomings of the existing technology in complex environments.

[0085] In summary, the present invention improves the retrieval accuracy of low-quality remote sensing images, reduces storage space occupancy and computational overhead, improves the robustness and generalization ability of the model, and ensures higher retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a schematic diagram of the network structure of the present invention.

[0087] Figure 2 It is a schematic diagram of the standard vector quantization process of the present invention.

[0088] Figure 3 It is a schematic diagram of the direct estimation process of the present invention.

[0089] Figure 4 This is an example diagram of a low-quality remote sensing image of the present invention.

[0090] Figure 5 This is a retrieval effect diagram of a query example of the AID (fog) data set under 64-bit hash coding of the present invention.

[0091] Figure 6 This is a retrieval effect diagram of a query example of the AID (gaussian_noise) data set under 64-bit hash coding of the present invention.

[0092] Figure 7 This is a retrieval effect diagram of a query example of the AID (motion_blur) data set under 64-bit hash coding of the present invention. DETAILED DESCRIPTION

[0093] The present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0094] This paper proposes a deep hash retrieval method for low-quality remote sensing images based on vector quantization. Figure 1 As shown in the figure, by combining vector quantization, deep hashing network and multiple loss functions, the feature extraction and retrieval performance of high-quality remote sensing images and low-quality remote sensing images are optimized. This method mainly solves the performance bottleneck of low-quality remote sensing images in remote sensing image retrieval.

[0095] A deep hash retrieval method for low-quality remote sensing images based on vector quantization includes the following steps:

[0096] Step 1: Image feature extraction: obtain high-quality remote sensing images and low-quality remote sensing images as input, and extract the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks;

[0097] Step 2: Vector quantization: Input the features extracted in step 1 into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features;

[0098] Step 3: Hash code generation: The quantized features generated in step 2 are used through a deep hash network to generate a hash code for image retrieval;

[0099] Step 4: Application of loss function: After feature extraction, feature quantization and hash code generation in steps 1 to 3, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss, which are used to ensure semantic information retention, feature distance constraint and collaborative learning of encoder, codebook and decoder.

[0100] The specific steps of step 1 are as follows:

[0101] High-quality remote sensing images are represented by X = {x1, x2, …, x n} and the corresponding low-quality remote sensing image is expressed as At the same time, it is input into the backbone network AlexNet, and the image features are extracted through the backbone network AlexNet. After feature extraction, the dimension is reduced through two layers of fully connected layers to generate a real-valued hash code. The real-valued hash code of a high-quality remote sensing image is represented by h = {h1,h2,…,h n}, the real-valued hash code of the low-quality remote sensing image is represented as Among them, h and All belong to R K , K is the number of bits of the real-valued hash code, and the formula for generating the real-valued hash code is as follows:

[0102] h i =Φ(x i ;θ),i=1,2,…,N

[0103]

[0104] Among them, θ represents the weight parameters of the backbone network AlexNet, two fully connected layers and the hash layer, and Φ represents a mapping function represented by θ, which includes convolution, pooling and fully connected operations.

[0105] like Figure 2 As shown, the specific steps of step 2 are as follows:

[0106] Furthermore, in order to obtain the quantized image features, the vector quantization steps are as follows:

[0107] The two learnable codebooks E∈R m×KAdded at the end of the feature extraction network, it is used to quantify the features of low-quality remote sensing images and high-quality remote sensing images, respectively, where m is the number of codewords in the codebook, e∈R K is the codeword, that is, the codebook E∈R m ×K The real-valued hash codes of high-quality remote sensing images and low-quality remote sensing images are vector quantized through the corresponding codebook to obtain quantized features. For any real-valued hash code h of a high-quality remote sensing image, its standard quantization process is expressed as:

[0108] s=E(h)=e q

[0109]

[0110] Where E(h) represents discretizing h into a codeword in the codebook, and e q represents the qth codeword in the codebook, q is the codeword index with the smallest Euclidean distance to the real-valued hash code h in the codebook E, and then the first L codewords with the closest Euclidean distance to the real-valued hash code are selected for weighted summation to supplement the information of the quantitative features and ultimately improve its representation ability. Specifically, for any real-valued hash code h of a remote sensing image, the first L codewords with the closest Euclidean distance to it are {e1,e2,…,e L}, and the distance from the real-valued hash code h of the high-quality remote sensing image is {d1,d2,…,d L}, and calculate the weights {β1,β2,…,β L}, the final formula of the quantized vector z is:

[0111]

[0112] After vector quantization, the quantitative features of high-quality remote sensing images are expressed as Z = {z1,z2,…,z n}, the quantitative features of low-quality remote sensing images are expressed as

[0113] The specific steps of step 3 are as follows:

[0114] Furthermore, in order to obtain a hash code that can be used for retrieval, the hash code generation steps are as follows:

[0115] In order to reduce the quantization error in the process of converting real-valued hash codes into hash codes, after the real-valued hash codes are generated, the value of each dimension of the real-valued hash codes is limited to between -1 and 1 through the activation function tanh(·). The specific formula is:

[0116]

[0117] Real-valued hash code z for high-quality remote sensing imagesi , its normalized hash code v i It is expressed as:

[0118] v i =tanh(z i ),i=1,2,…,N

[0119] Similarly, for the real-valued hash codes of low-quality remote sensing images Its normalized hash code It is expressed as:

[0120]

[0121] In order to assist the learning of the codebook in the deep hashing network, the quantized features are constrained by the cross entropy loss through the classification layer to inject semantic information. The specific formula is:

[0122] p i =fc(v i ),i=1,2,…,N

[0123]

[0124] Among them, fc(·) represents the classification layer, p i represents the classification vector of the i-th high-quality remote sensing image, Represents the classification vector of the i-th low-quality remote sensing image;

[0125] The process of generating a hash code is to pass the vector p i and Binarization is performed to generate the final high-quality remote sensing image hash code b i and hash codes for low-quality remote sensing images The specific formula is:

[0126]

[0127] b i =sign(v i ),i=1,2,…,N

[0128]

[0129] Among them, b i represents the hash code of the i-th high-quality remote sensing image, v i is its corresponding normalized real-valued hash code; similarly, represents the hash code of the i-th low-quality remote sensing image, is the normalized real-valued hash code of its corresponding low-quality remote sensing image. The real-valued hash code is converted into binary form by the sign(·) function to generate the final hash code for retrieval.

[0130] The specific steps of step 4 are as follows:

[0131] In order to ensure the preservation of semantic information of image feature representation, the constraint of feature distance, and the collaborative learning of encoder, codebook and decoder, the loss function application steps are as follows:

[0132] The Pairwise Loss function is used to constrain the feature space learning of high-quality remote sensing images and low-quality remote sensing images, ensuring that the features of the same type of images are closer and the features of different types of images are farther apart. Specifically, Pairwise Loss is used in the following three situations:

[0133] Case 1: Embedding space constraints of high-quality remote sensing images: It is used to constrain the feature vectors of high-quality remote sensing images so that the feature distances between high-quality remote sensing images of the same type are closer and the feature distances between high-quality remote sensing images of different types are farther. The loss function formula is as follows:

[0134]

[0135] Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images, v i and v j are the feature vectors of low-quality remote sensing images;

[0136] Case 2: Embedding space constraints of low-quality remote sensing images: It is used to constrain the features of low-quality remote sensing images so that the feature distances between low-quality remote sensing images of the same type are closer, and the feature distances between low-quality remote sensing images of different types are farther. The loss function formula is as follows:

[0137]

[0138] Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images. and are the feature vectors of low-quality remote sensing images;

[0139] Case 3: Pairing constraints of high-quality remote sensing image features and low-quality remote sensing image features: It is used to bring high-quality remote sensing image features and low-quality remote sensing image features of the same category closer to each other, and keep image features of different categories away. The loss function formula is as follows:

[0140]

[0141] Among them, φ ij Represents the similarity indicator matrix of high-quality remote sensing images and low-quality remote sensing images. When φ ij =1, it means that the i-th high-quality remote sensing image and the j-th low-quality remote sensing image belong to the same category; when φ ij =0, it means they belong to different categories;

[0142] The overall Pairwise Loss loss function can be expressed as:

[0143] L pairwise =L pairwise-high +L pairwise-low +L pairwise-hl

[0144] The cross entropy loss function is used to guide the learning of the feature extraction network and the codebook to ensure that the quantized features of high-quality remote sensing images and low-quality remote sensing images retain their semantic information. Specifically, the cross entropy loss function is used to optimize the classification performance of high-quality remote sensing images and low-quality remote sensing images. The formula is as follows:

[0145]

[0146] Among them, y ic is the true label of the i-th high-quality remote sensing image for category c, p ic is the predicted probability of category c for high-quality remote sensing images; is the true label of the i-th low-quality remote sensing image for category c, is the predicted probability of low-quality remote sensing images for category c; C is the number of categories;

[0147] The reconstruction loss function is used to guide the collaborative learning of the feature extraction network and the codebook, such as Figure 3 As shown in the figure, since the arg min operation in vector quantization cannot be differentiated, the straight-through estimator (STE) method is used to bypass the codebook with the gradients brought by the pairwise loss and cross entropy loss behind the codebook, and pass them to the feature extraction network in front, so as to realize the parameter update of the feature extraction network. The specific loss function formula is as follows:

[0148] L commitment =|h-sg(e q )| 2

[0149] Here, sg(·) refers to the stop gradient operation, which is used to prevent the gradient from propagating in the codebook. Although the direct estimation method can realize gradient transmission, the codebook cannot directly receive the gradient information from the pairwise loss and cross entropy loss, which limits the codebook learning.

[0150] In order to allow the codebook space to participate in learning normally, a dictionary learning method is adopted to optimize the learning process of the codebook by calculating the L2 error between the features output by the feature extraction network and the corresponding quantized codewords. The specific loss function formula is as follows:

[0151] L vq =|sg(h)-e q | 2

[0152] Among them, h is the output of the feature extraction network, e q is the corresponding codeword;

[0153] The overall optimization problem of the model optimizes the learnable parameters in the network framework by combining all loss functions. The overall loss function formula is as follows:

[0154]

[0155] Among them, Θ represents all the learnable parameters in the network framework of this chapter.

[0156] A deep hash retrieval system for low-quality remote sensing images based on vector quantization, comprising:

[0157] Image feature extraction module: by obtaining high-quality remote sensing images and low-quality remote sensing images as input, and extracting the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks, it is used to implement step 1 of a deep hash retrieval method for low-quality remote sensing images based on vector quantization;

[0158] Vector quantization module: input the features into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features for implementing step 2 of a deep hash retrieval method for low-quality remote sensing images based on vector quantization;

[0159] Hash code generation module: generates hash codes for image retrieval through deep hash networks, which is used to implement step 3 of a deep hash retrieval method for low-quality remote sensing images based on vector quantization;

[0160] Loss function application module: After feature extraction, feature quantization and hash code generation, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss, which are used to ensure semantic information retention, feature distance constraints and collaborative learning of encoders, codebooks and decoders, and to implement step 4 of a deep hash retrieval method for low-quality remote sensing images based on vector quantization.

[0161] A low-quality remote sensing image deep hash retrieval device based on vector quantization, comprising:

[0162] Memory: used for storing a computer program for implementing the deep hash retrieval method for low-quality remote sensing images based on vector quantization;

[0163] Processor: used to implement the deep hash retrieval method for low-quality remote sensing images based on vector quantization when executing the computer program.

[0164] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the deep hash retrieval method for low-quality remote sensing images based on vector quantization.

[0165] In order to verify the effectiveness of the proposed method in the low-quality remote sensing image retrieval task, the method of the present invention simulates three degradation conditions of fog, Gaussian noise and motion blur on the AID dataset. Figure 4 As shown. Subsequent comparative experiments and ablation experiments are conducted on these three datasets. The AID dataset contains 30 categories of remote sensing images, a total of 10,000 images, each category contains 220 to 420 images, and the resolution is 600×600.

[0166] The present invention uses AlexNet as the backbone network in the network framework for feature extraction. In order to adapt to the hash retrieval task, the last classification layer of the AlexNet network is replaced by a hash layer. The output dimension of the hash layer is K, that is, the number of bits of the binary hash code. The entire network framework is trained using the Adam optimizer, and the learning rate and weight decay parameters in the optimizer are set to 1×10 -4 and 5×10 -4. In the vector quantization operation, the number of selected codewords L is set to 10, and the number of codewords m in the codebook is set to 5000. The codebook is used to quantize the features of high-quality and low-quality remote sensing images. The resolution of all input remote sensing images is adjusted to 224×224 before entering the network. Each type of scene image in the AID dataset is divided into 70% as a training set and 30% as a test set. All codes are written in Python 3.8 and implemented based on the PyTorch framework. The experiments are run on the hardware environment of Intel Xeon Gold 5218R CPU and NVIDIA GeForce RTX 3080GPU.

[0167] In order to verify the effectiveness of the present invention in low-quality remote sensing image retrieval tasks, a comparative experiment with existing methods was conducted. The compared methods include the following:

[0168] (1) Scratch: directly use low-quality remote sensing images to train the model from scratch as a baseline method.

[0169] (2) Finetune: First use high-quality remote sensing images for pre-training, and then use low-quality remote sensing images for fine-tuning.

[0170] (3) FAH: A remote sensing image retrieval method that uses high-quality remote sensing images and low-quality remote sensing images for simultaneous training;

[0171] (4) AHCL: Another remote sensing image retrieval method that uses high-quality remote sensing images and low-quality remote sensing images for joint training.

[0172] The effects of these methods are measured by Mean Average Precision (MAP), which is the average precision of the retrieval method in all queries, and Precision@50 is used to measure the accuracy of the retrieval method. The higher the value, the better the retrieval effect.

[0173] Table 1 Different hash retrieval methods in different datasets

[0174]

[0175] The experiments were carried out under three degradation conditions (fog, Gaussian noise, and motion blur) of the AID dataset, and the results are shown in Table 1.

[0176] In the AID (fog) dataset, as shown in the figure, the proposed method performs best in all hash code bit counts. The Scratch method performs poorly in low-quality remote sensing images, with MAP values ​​of 78.59 and 79.86 for 32-bit and 64-bit hash codes, respectively. The Finetune method improves the effect by fine-tuning after pre-training, reaching 86.03 for 64-bit hash codes. In contrast, the FAH and AHCL methods are trained by combining high-quality and low-quality remote sensing images, with MAP values ​​of 91.86 and 92.02 for 64-bit hash codes, respectively; similarly, Figure 5 As shown in the figure, compared with other methods, this method has the highest value in the Precision@50 indicator.

[0177] In the AID (gaussian_noise) dataset, the Scratch method was significantly affected, and the MAP of the 32-bit hash code was 77.10. The performance of the Finetune method was significantly improved, and the MAP reached 87.90 for the 64-bit hash code. The FAH and AHCL methods performed better for the 64-bit hash code, reaching 90.88 and 91.99 respectively. The method of the present invention achieved the best MAP value of 92.39 for the 64-bit hash code, which was 0.40% higher than that of AHCL. Similarly, Figure 6 As shown, compared with other methods, the method of the present invention has the highest value in the Precision@50 index.

[0178] In the AID (motion_blur) dataset, the MAP of all methods is improved. The present invention achieves the best results of 92.47 and 92.70 under the conditions of 128-bit and 256-bit hash codes, respectively, which is 1.10% and 1.16% higher than the FAH method. Figure 7 As shown, compared with other methods, the value of the present invention on the Precision@50 index is the highest.

Claims

1. A deep hash retrieval method for low-quality remote sensing images based on vector quantization, characterized in that: The following steps are involved: Step 1: Image feature extraction: obtain high-quality remote sensing images and low-quality remote sensing images as input, and extract the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks; Step 2: Vector quantization: Input the features extracted in step 1 into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features; Step 3: Hash code generation: The quantized features generated in step 2 are used through a deep hash network to generate a hash code for image retrieval; Step 4: Application of loss function: After feature extraction, feature quantization and hash code generation in steps 1 to 3, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss, which are used to ensure semantic information retention, feature distance constraints and collaborative learning of encoder, codebook and decoder.

2. According to claim 1, a deep hash retrieval method for low-quality remote sensing images based on vector quantization is characterized in that: The specific steps of step 1 are as follows: High-quality remote sensing images are represented by X = {x1, x2, …, x n } and the corresponding low-quality remote sensing image is expressed as At the same time, it is input into the backbone network AlexNet, and the image features are extracted through the backbone network AlexNet. After feature extraction, the dimension is reduced through two layers of fully connected layers to generate a real-valued hash code. The real-valued hash code of a high-quality remote sensing image is represented by h = {h1,h2,…,h n }, the real-valued hash code of the low-quality remote sensing image is represented as Among them, h and All belong to R K , K is the number of bits of the real-valued hash code, and the formula for generating the real-valued hash code is as follows: h i =Φ(x i ;θ),i=1,2,…,N Among them, θ represents the weight parameters of the backbone network AlexNet, two fully connected layers and the hash layer, and Φ represents a mapping function represented by θ, which includes convolution, pooling and fully connected operations.

3. According to the method of claim 1, the method is characterized in that: The specific steps of step 2 are as follows: Furthermore, in order to obtain the quantized remote sensing image features, the vector quantization steps are as follows: The two learnable codebooks E∈R m×K Added at the end of the feature extraction network, it is used to quantify the features of low-quality remote sensing images and high-quality remote sensing images, respectively, where m is the number of codewords in the codebook, e∈R K is the codeword, that is, the codebook E∈R m×K The real-valued hash codes of high-quality remote sensing images and low-quality remote sensing images are vector quantized through the corresponding codebook to obtain quantized features. For any real-valued hash code h of a high-quality remote sensing image, its standard quantization process is expressed as: s=E(h)=e q Where E(h) represents discretizing h into a codeword in the codebook, e q represents the qth codeword in the codebook, q is the codeword index with the smallest Euclidean distance to the real-valued hash code h in the codebook E, and then the first L codewords with the closest Euclidean distance to the real-valued hash code are selected for weighted summation to supplement the information of the quantitative features and ultimately improve its representation ability. Specifically, for any real-valued hash code h of a remote sensing image, the first L codewords with the closest Euclidean distance to it are {e1,e2,…,e L }, and the distance from the real-valued hash code h of the high-quality remote sensing image is {d1,d2,…,d L }, and calculate the weights {β1,β2,…,β L }, the final formula of the quantized vector z is: After vector quantization, the quantitative features of high-quality remote sensing images are expressed as Z = {z1,z2,…,z n }, the quantitative features of low-quality remote sensing images are expressed as 4. According to the method of claim 1, the method is characterized in that: The specific steps of step 3 are as follows: Furthermore, in order to obtain a hash code that can be used for retrieval, the hash code generation steps are as follows: In order to reduce the quantization error in the process of converting real-valued hash codes into hash codes, after the real-valued hash codes are generated, the value of each dimension of the real-valued hash codes is limited to between -1 and 1 through the activation function tanh(·). The specific formula is: Real-valued hash code z for high-quality remote sensing images i , its normalized hash code v i It is expressed as: v i =tanh(x i ),i=1,2,…,N Similarly, for the real-valued hash codes of low-quality remote sensing images Its normalized hash code It is expressed as: In order to assist the learning of the codebook in the deep hashing network, the quantized features are constrained by the cross entropy loss through the classification layer to inject semantic information. The specific formula is: p i =fc(v i ),i=1,2,…,N Among them, fc(·) represents the classification layer, p i represents the classification vector of the i-th high-quality remote sensing image, Represents the classification vector of the i-th low-quality remote sensing image; The process of generating a hash code is to pass the vector p i and Binarization is performed to generate the final high-quality remote sensing image hash code b i and hash codes for low-quality remote sensing images The specific formula is: b i =sign(v i ),i=1,2,…,N Among them, b i represents the hash code of the i-th high-quality remote sensing image, v i is its corresponding normalized real-valued hash code; similarly, represents the hash code of the i-th low-quality remote sensing image, is the normalized real-valued hash code of its corresponding low-quality remote sensing image. The real-valued hash code is converted into binary form by the sign(·) function to generate the final hash code for retrieval.

5. According to the method of claim 1, the method is characterized in that: The specific steps of step 4 are as follows: In order to ensure the preservation of semantic information of remote sensing image feature representation, the constraint of feature distance, and the collaborative learning of encoder, codebook and decoder, the loss function application steps are as follows: The Pairwise Loss function is used to constrain the feature space learning of high-quality remote sensing images and low-quality remote sensing images, ensuring that the features of the same type of images are closer and the features of different types of images are farther apart. Specifically, Pairwise Loss is used in the following three situations: Case 1: Embedding space constraints of high-quality remote sensing images: It is used to constrain the feature vectors of high-quality remote sensing images so that the feature distances between high-quality remote sensing images of the same type are closer and the feature distances between high-quality remote sensing images of different types are farther. The loss function formula is as follows: Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images, v i and v j are the feature vectors of low-quality remote sensing images; Case 2: Embedding space constraints of low-quality remote sensing images: It is used to constrain the features of low-quality remote sensing images so that the feature distances between low-quality remote sensing images of the same type are closer, and the feature distances between low-quality remote sensing images of different types are farther. The loss function formula is as follows: Where n represents the number of remote sensing image samples, i and j represent the indexes of remote sensing image samples, m represents the distance threshold, and θ ij is the similarity indicator matrix, which is defined similarly to the case of high-quality remote sensing images. and are the feature vectors of low-quality remote sensing images; Case 3: Pairing constraints of high-quality remote sensing image features and low-quality remote sensing image features: It is used to bring high-quality remote sensing image features and low-quality remote sensing image features of the same category closer to each other, and keep image features of different categories away. The loss function formula is as follows: Among them, φ ij Represents the similarity indicator matrix of high-quality remote sensing images and low-quality remote sensing images. When φ ij =1, it means that the i-th high-quality remote sensing image and the j-th low-quality remote sensing image belong to the same category; when φ ij =0, it means they belong to different categories; The overall Pairwise Loss loss function can be expressed as: L pairwise =L pairwise-high +L pairwise-low +L pairwise-hl The cross entropy loss function is used to guide the learning of the feature extraction network and the codebook to ensure that the quantized features of high-quality remote sensing images and low-quality remote sensing images retain their semantic information. Specifically, the cross entropy loss function is used to optimize the classification performance of high-quality remote sensing images and low-quality remote sensing images. The formula is as follows: Among them, y ic is the true label of the i-th high-quality remote sensing image for category c, p ic is the predicted probability of category c for high-quality remote sensing images; is the true label of the i-th low-quality remote sensing image for category c, is the predicted probability of low-quality remote sensing images for category c; C is the number of categories; The reconstruction loss function is used to guide the collaborative learning of the feature extraction network and the codebook. Since the arg min operation in vector quantization cannot be differentiated, the straight-through estimation (STE) method is adopted to bypass the codebook with the gradients brought by the pairwise loss and cross entropy loss behind the codebook and pass them to the feature extraction network in front, thereby realizing the parameter update of the feature extraction network. The specific loss function formula is as follows: L commitment =|h-sg(e q )| 2 Here, sg(·) refers to the stop gradient operation, which is used to prevent the gradient from propagating in the codebook. Although the direct estimation method can realize gradient transmission, the codebook cannot directly receive the gradient information from the pairwise loss and cross entropy loss, which limits the codebook learning. In order to allow the codebook space to participate in learning normally, a dictionary learning method is adopted to optimize the learning process of the codebook by calculating the L2 error between the features output by the feature extraction network and the corresponding quantized codewords. The specific loss function formula is as follows: L vq =|sg(h)-e q | 2 Among them, h is the output of the feature extraction network, e q is the corresponding codeword; The overall optimization problem of the model optimizes the learnable parameters in the network framework by combining all loss functions. The overall loss function formula is as follows: Among them, Θ represents all the learnable parameters in the network framework of this chapter.

6. A retrieval system for a low-quality remote sensing image deep hash retrieval method based on vector quantization according to any one of claims 1 to 5, characterized in that: include: Image feature extraction module: It obtains high-quality remote sensing images and low-quality remote sensing images as input, and extracts the features of high-quality remote sensing images and low-quality remote sensing images respectively through deep convolutional neural networks; Vector quantization module: Input the features into the vector quantization module, quantize them into independent discrete spaces, and generate quantized features; Hash code generation module: generates hash codes for image retrieval through deep hashing network. Loss function application module: After feature extraction, feature quantization and hash code generation, feature representation is obtained, and the feature representation is constrained by applying loss functions. The loss functions include pairwise loss, reconstruction loss and cross entropy loss to ensure semantic information retention, feature distance constraints and collaborative learning of encoder, codebook and decoder.

7. A deep hash retrieval device for low-quality remote sensing images based on vector quantization, characterized in that: include: Memory: used to store a computer program for implementing a deep hash retrieval method for low-quality remote sensing images based on vector quantization as described in any one of claims 1 to 5; Processor: used to implement a deep hash retrieval method for low-quality remote sensing images based on vector quantization as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a deep hash retrieval method for low-quality remote sensing images based on vector quantization as described in any one of claims 1 to 5 are implemented.

Images