A satellite cloud image multi-label hashing retrieval method based on interpretable deep network

By using the multi-label hash retrieval method of interpretable deep network in satellite cloud maps, global and local features are extracted and rich semantic hash codes are generated, which solves the problem that semantic information is difficult to reveal in satellite cloud map retrieval, and realizes the complex cloud map accurate search and description.

CN117725252BActive Publication Date: 2024-12-06NINGBO UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311526961.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-16
Publication Date
2024-12-06
Estimated Expiration
2043-11-16

AI Technical Summary

Technical Problem

Existing satellite cloud map search technology is difficult to accurately reveal the meteorological semantic information of cloud maps, resulting in poor similarity search results, especially in complex scenarios where complex categories in images cannot be distinguished.

Method used

The multi-label hash retrieval method of interpretable deep network is adopted to extract fusion features through global and local feature learning modules, and a multi-label supervision mechanism is introduced to generate hash codes with rich semantics for precise retrieval of satellite cloud maps.

Benefits of technology

It improves the accuracy and interpretability of satellite cloud map retrieval, enhances the research and application capabilities of meteorological workers, and realizes accurate description and similarity search of complex cloud maps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117725252B_ABST
    Figure CN117725252B_ABST
Patent Text Reader

Abstract

The invention discloses a satellite cloud image multi-label hash retrieval method of an interpretable deep network, which relates to the field of satellite cloud image multi-label retrieval. A target cloud image is input into a pre-trained interpretable deep network to obtain a target hash code; the target hash code and the hash codes of each historical cloud image are used to perform similarity measurement to retrieve a historical cloud image similar to the target cloud image; the interpretable deep network comprises a feature learning module and a hash learning module, the feature learning module comprises a global feature learning module and a local feature learning module, the local feature learning module comprises an attention branch network and a suppression module, the attention branch network improves the effectiveness of feature extraction, the suppression module can gradually discover other complementary areas to give each local feature unique semantic information, so that the retrieval has a certain interpretability, and the multi-label supervision mechanism introduces the hash learning module to better describe the complex semantic content of the satellite cloud image, improve the retrieval efficiency, and realize the accurate retrieval of the satellite cloud image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of satellite cloud image multi-label retrieval, and in particular to a satellite cloud image multi-label hash retrieval method of an interpretable deep network. Background Art

[0002] Clouds play an important role in the weather system. Cloud type, cloud phase and cloud height all have a significant impact on the generation and development of the weather system. Understanding the weather by observing clouds is one of the important means for people to predict the weather. Satellite images are a powerful tool for detecting clouds and weather systems. Satellite images can be used to understand different weather conditions, assess their intensity and future development trends, and provide an all-weather basis for weather forecasts and disaster weather forecasts.

[0003] Content-based image retrieval (CBIR) is an important branch of image retrieval, which can search for images similar to a certain satellite cloud image in the database. At present, CBIR mainly adopts two methods. One is to extract the feature information of the cloud image by traditional feature extraction methods and perform retrieval based on it. The feature extraction method is the gray level co-occurrence matrix method or the grid-based inscribed circle method. However, the content of satellite cloud images is complex, and the similarity between various types of images is high. The use of manual features cannot accurately reveal the meteorological semantic information contained in the cloud image, resulting in poor similarity retrieval effect. The second method is to refer to the processing method of remote sensing images, relying on convolutional neural networks to extract image features and then realize similarity retrieval. However, this method usually relies on single-label retrieval, which means that only the most significant labels or features will be learned by the model. For simple scenes, such as forest scenes, the above method can obtain good retrieval results, but for complex scenes, such as satellite cloud images, the above single-label method cannot distinguish complex categories in the image, ignoring the rich and complex information contained in the image.

[0004] Therefore, how to design an effective solution to realize similarity retrieval of satellite cloud images is a problem that needs to be solved urgently. Summary of the invention

[0005] The problem solved by the present invention is: to provide a satellite cloud image multi-label hash retrieval method of an interpretable deep network, wherein the extracted fusion features include rich semantic information of the target cloud image, and the target hash code subsequently generated on this basis also includes rich semantics and is interpretable, thereby improving the credibility of the entire interpretable deep network and realizing accurate retrieval of satellite cloud images.

[0006] To solve the above problems, the present invention provides a satellite cloud image multi-label hash retrieval method based on an interpretable deep network, comprising:

[0007] The obtained target cloud map is input as an input item into a pre-trained interpretable deep network to obtain a target hash code of the target cloud map, wherein the interpretable deep network includes a feature learning module for generating an interpretable fusion feature of the target cloud map and a hash learning module for generating the target hash code, the feature learning module includes a global feature learning module and a local feature learning module, the global feature learning module is a unit for learning a single global semantic feature representing the target cloud map, and the local feature learning module is a unit for learning local semantic features of multiple cloud types in different regions of the target cloud map, and then the output of the global feature learning module is spliced ​​and combined with the output of the local feature learning module to obtain the fusion feature; the local feature learning module includes multiple local branches, each of which includes an attention branch network and a suppression module, to obtain multiple local features for expressing local semantic information of multiple cloud types in different regions, so as to generate the target hash code using the fusion feature; a multi-label supervision mechanism is introduced in the hash learning module;

[0008] A similarity measurement is performed based on the target hash code and the hash codes of each historical cloud map in the historical cloud map database, and a historical cloud map similar to the target cloud map is retrieved.

[0009] The beneficial effects of the present invention are as follows: the fusion features extracted in the present application include rich semantic information of the target cloud image, and the target hash code subsequently generated on this basis also includes rich semantics and is interpretable, thereby improving the credibility of the entire interpretable deep network, realizing accurate retrieval of satellite cloud images, and providing great help for meteorological workers to carry out further research, which is beneficial to practical applications.

[0010] Furthermore, the hash learning module in the interpretable deep network generates the target hash code according to the fusion feature, including:

[0011] Inputting the fused feature as an input item into a pre-trained hash code learning module to obtain a class hash code corresponding to the fused feature;

[0012] Determine a target hash code of the target cloud image according to the class hash code and a first preset relationship;

[0013] The first preset relationship is:

[0014]

[0015] Wherein, d is the class hash code, and b is the target hash code.

[0016] In this scheme, the fused features include rich semantic information, and the hash code generated based on them also includes rich semantic information, which makes the interpretable deep network interpretable and improves the credibility of the output results of the network.

[0017] Furthermore, the step of generating the fusion features of the target cloud map by the feature learning module in the interpretable deep network includes:

[0018] Inputting the target cloud map into the backbone network to obtain a deep feature map;

[0019] Determine the global features representing the global semantic information of the target cloud map according to the deep feature map and the preset global feature extraction strategy;

[0020] Based on the deep feature map and the preset local feature extraction strategy, determine the local features representing the local semantic information of multiple cloud types in different areas of the target cloud map;

[0021] The global feature is concatenated with the local feature to obtain a fused feature.

[0022] Further, according to the deep feature map and the preset global feature extraction strategy, the global features representing the global semantic information of the target cloud map are determined, including:

[0023] Performing an average pooling operation on the deep feature map;

[0024] The result after the average pooling operation is input into the fully connected layer for feature dimensionality reduction to obtain global features that characterize the global semantic information of the target cloud image.

[0025] In this scheme, the global features are determined simply, accurately and reliably through the average pooling operation and feature dimension reduction of the fully connected layer, which is conducive to the determination of the subsequent fusion features.

[0026] Further, based on the deep feature map and the preset local feature extraction strategy, local features representing local semantic information of cloud types in different regions in the target cloud map are determined, including:

[0027] Convolving the deep feature map to obtain a first sub-map;

[0028] Processing the first sub-graph according to a preset attention strategy to obtain a first attention graph representing a mining result of the most distinctive region;

[0029] Generate a first local feature map according to the first attention map and the first sub-map;

[0030] A first local feature representing local semantic information of the cloud type under the first local feature map is determined, so as to splice the global feature with the first local feature to obtain a fusion feature corresponding to the target cloud map.

[0031] In this solution, the design of the preset local feature extraction strategy ensures that the most distinctive area can be mined so as to extract its first local feature and add it to the fusion feature, thereby enriching the semantics of the fusion feature.

[0032] Further, the first sub-graph is processed according to a preset attention strategy to obtain a first attention graph representing the mining result of the most distinctive region, including:

[0033] activating the first sub-image to determine its corresponding activation feature image;

[0034] The convolution operation and the aggregation normalization operation are sequentially performed on the activated feature map, and the execution result is processed using the Sigmoid activation function to obtain the first attention map.

[0035] In this scheme, the design of the preset attention strategy incorporates the consideration of the attention mechanism, which can focus more on the most distinctive areas.

[0036] Further, after the first sub-graph is processed according to a preset attention strategy to obtain a first attention graph representing the mining result of the most distinctive region, the method further includes:

[0037] S21: set o=1;

[0038] S22: determining a second pixel point based on a first attention map corresponding to the first sub-image and a second preset relationship to obtain a suppression map corresponding to the second pixel point;

[0039] The second preset relationship is:

[0040]

[0041] Among them, μ k′ is the pixel value of the kth second pixel in the suppression map, μ k is the pixel value of the kth pixel in the first attention map, a is the preset hyperparameter, μ mean is the average pixel value of all pixels in the first attention map, μ std is the standard deviation of the pixel values ​​of all pixels in the first attention map;

[0042] S23: determining an oth second sub-image according to the suppression image and the first sub-image;

[0043] S24: Processing the oth second sub-graph according to the preset attention strategy to obtain an oth second attention graph, so as to generate an oth second local feature graph according to the oth second attention graph and the oth second sub-graph, and further determining the oth second local feature corresponding to the oth second local feature graph;

[0044] S25: Let o=o+1;

[0045] S26: Determine whether o reaches a preset branch number, where the preset branch number is an integer not less than 2; if not, proceed to S27; if yes, proceed to S29;

[0046] S27: determining a new second pixel point according to the o-1th second attention map and the second preset relationship to obtain a new suppression map corresponding to the new second pixel point;

[0047] S28: Generate the oth second sub-graph according to the new suppression graph and the o-1th second sub-graph; and return to S24;

[0048] S29: splicing the global feature, the first local feature and the obtained o-1 second local features to obtain a fusion feature corresponding to the target cloud image.

[0049] In this scheme, by generating multiple second local features, the mining of the remaining specific object areas except the most distinctive areas is supplemented, so that the formed fusion features have rich semantic information, and the final retrieval results also have a certain degree of interpretability, which better describes the complex semantic content of the satellite cloud image.

[0050] Furthermore, the interpretable deep network is pre-trained based on a cloud map training set, the cloud map training set includes N training cloud maps, each of the training cloud maps corresponds to at least one real image classification label, the total number of categories of real image classification labels corresponding to all training cloud maps is C, and N and C are both integers not less than 1; the training steps of the interpretable deep network include:

[0051] Inputting the training cloud graph into an interpretable deep network to be trained to obtain a hash code-like feature;

[0052] Inputting the class hash code feature into the classification layer to obtain the corresponding image classification label prediction value;

[0053] Training the interpretable deep network using the class hash code features, the image classification label prediction value and a preset loss function;

[0054] The preset loss function is:

[0055] L total =λLcls +ηL q +νL b

[0056] L total is the training error, L cls is the multi-label classification loss, λ is the loss corresponding to L cls The corresponding first penalty parameter, L q To quantify the loss, η is the same as L q The corresponding second penalty parameter, L b is the bit balance loss, ν is related to L b The corresponding third penalty parameter;

[0057] Among them, the L cls Determined based on a third preset relationship, the third preset relationship is:

[0058]

[0059] is the i-th predicted value in the image classification label prediction value corresponding to the n-th training cloud map, ω i is the weight value, is the i-th true value in the real image classification label corresponding to the n-th training cloud image, and σ is the Sigmoid activation function;

[0060] The L q Determined based on a fourth preset relationship, the fourth preset relationship is:

[0061]

[0062] K is the preset hash code length, d n is the class hash code feature corresponding to the nth training cloud image, and e is a K-dimensional vector with a value of 1;

[0063] The L b Determined based on the fifth preset relationship, the fifth preset relationship is:

[0064]

[0065] mean(d n ) represents the d n The average of the median values.

[0066] In this scheme, multi-label supervision is introduced into the interpretable deep network, which can generate more compact hash codes and improve the efficiency of similarity retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1A flowchart of a satellite cloud image multi-label hash retrieval method of an interpretable deep network provided by the present invention;

[0068] Figure 2 A schematic diagram of implementing a preset local feature extraction strategy provided by the present invention;

[0069] Figure 3 A schematic diagram of information transmission of an interpretable deep network provided by the present invention;

[0070] Description of reference numerals:

[0071] F1-first sub-map, Fm-activated feature map, M1-first attention map, LFM1-first local feature map, GF-global feature, LF1-first local feature, LF2-first second local feature, LF3-second second local feature, F21-first second sub-map, F22-second second sub-map, M2-first second attention map. DETAILED DESCRIPTION

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0073] Please refer to Figure 1 , Figure 1 A flowchart of a satellite cloud image multi-label hash retrieval method of an interpretable deep network provided by the present invention.

[0074] The satellite cloud image multi-label hash retrieval method of the interpretable deep network includes:

[0075] S11: Input the acquired target cloud map as an input item into a pre-trained interpretable deep network to obtain a target hash code of the target cloud map, wherein the interpretable deep network includes a feature learning module for generating a target cloud map with interpretable fusion features and a hash learning module for generating a target hash code, the feature learning module includes a global feature learning module and a local feature learning module, the global feature learning module is a unit for learning a single global semantic feature representing the target cloud map, and the local feature learning module is a unit for learning local semantic features of multiple cloud types in different regions in the target cloud map, and then the output of the global feature learning module is spliced ​​and combined with the output of the local feature learning module to obtain a fusion feature; the local feature learning module includes multiple local branches, each of which includes an attention branch network and an inhibition module, and multiple local features for expressing local semantic information of multiple cloud types in different regions are obtained, so as to generate a target hash code using the fusion feature; a multi-label supervision mechanism is introduced in the hash learning module;

[0076] S12: Perform similarity measurement based on the target hash code and the hash codes of each historical cloud map in the historical cloud map database, and retrieve the historical cloud map similar to the target cloud map.

[0077] Specifically, considering that different areas in satellite cloud images often contain different cloud types, an interpretable deep network is designed. The network is pre-trained based on a multi-label cloud image training set, which overcomes the neglect of rich semantic information in the cloud image when relying on a single semantic label for network training; the feature learning module in the interpretable deep network can extract global features and multiple local features, and then obtain fused features, and in the process of extracting each local feature, the attention branch network and the inhibition module are applied. The layer-by-layer inhibition is conducive to the discovery of multiple different complementary regions, so that the global feature and each local feature have specific semantic information; the hash learning module in the interpretable deep network introduces a multi-label supervision mechanism, which can generate a compact target hash code to represent the content of the target cloud image, reducing the time and space costs, and it also has rich semantic information, relying on the target hash code and the hash code of each historical cloud image for similarity retrieval. Specifically, the retrieval is performed by calculating the Hamming distance through a simple XOR operation, which is conducive to improving the retrieval efficiency and has interpretability.

[0078] In addition, the method provided in this application was extensively experimented on a public satellite cloud image dataset, namely LSCIDMR-V2. The experimental results showed that the mAP on LSCIDMR-V2 reached 92.27%, which shows that the method has achieved excellent performance.

[0079] In summary, the present application provides a satellite cloud image multi-label hash retrieval method of an interpretable deep network, in which the extracted fusion features include rich semantic information of the target cloud image, and the target hash code subsequently generated on this basis also includes rich semantics and is interpretable, thereby improving the credibility of the entire interpretable deep network, enabling meteorologists to trust the retrieval results, and realizing accurate retrieval of satellite cloud images, which is beneficial to practical applications.

[0080] As a preferred embodiment, the hash learning module in the interpretable deep network generates a target hash code according to the fusion features, including:

[0081] Input the fused features as input items into the pre-trained hash code learning module to obtain the class hash code corresponding to the fused features;

[0082] Determine a target hash code of the target cloud image according to the class hash code and the first preset relationship;

[0083] The first preset relationship is:

[0084]

[0085] Among them, d is the class hash code and b is the target hash code.

[0086] Specifically, a compact target hash code can be generated for subsequent retrieval in the above manner, and the target hash code also includes rich semantic information, which makes the interpretable deep network interpretable and improves the credibility of the output results of the network.

[0087] As a preferred embodiment, the step of generating the fusion features of the target cloud map by the feature learning module in the interpretable deep network includes:

[0088] Input the target cloud map into the backbone network to obtain a deep feature map;

[0089] Determine the global features representing the global semantic information of the target cloud image based on the deep feature map and the preset global feature extraction strategy;

[0090] Based on the deep feature map and the preset local feature extraction strategy, the local features representing the local semantic information of multiple cloud types in different regions of the target cloud map are determined;

[0091] The global features are concatenated with the local features to obtain the fused features.

[0092] Specifically, the backbone network includes but is not limited to ResNet50, and the target cloud map can be extracted through ResNet50 to obtain a deep feature map. Figure 3 , Figure 3 The module for generating the global feature GF is called the global feature learning module, and the module for generating the local features (including the first local feature LF1, the first second local feature LF2 and the second second local feature LF3) is called the local feature learning module.

[0093] As a preferred embodiment, according to the deep feature map and the preset global feature extraction strategy, the global features representing the global semantic information of the target cloud map are determined, including:

[0094] Perform average pooling operation on deep feature maps;

[0095] The result after the average pooling operation is input into the fully connected layer for feature dimensionality reduction to obtain the global features that represent the global semantic information of the target cloud image.

[0096] Specifically, by passing the deep feature map through the average pooling operation and the fully connected layer in sequence, the global features can be determined simply, accurately and reliably, which is conducive to the determination of the subsequent fusion features.

[0097] As a preferred embodiment, based on the deep feature map and the preset local feature extraction strategy, local features representing the local semantic information of cloud types in different regions in the target cloud map are determined, including:

[0098] After convolving the deep feature map, the first sub-map F1 is obtained;

[0099] Processing the first sub-image F1 according to a preset attention strategy to obtain a first attention image M1 representing the mining result of the most distinctive region;

[0100] Generate a first local feature map LFM1 according to the first attention map M1 and the first sub-map F1;

[0101] The first local feature representing the local semantic information of the cloud type under the first local feature map LFM1 is determined, so as to splice the global feature with the first local feature, and then obtain the fusion feature corresponding to the target cloud map.

[0102] Specifically, considering that satellite cloud images often include multiple cloud types, it is necessary to capture the features of these local areas in order to generate more meaningful fusion features and hash codes. For this reason, the design mechanism of the preset local feature extraction strategy takes into account the attention mechanism, so as to focus more on the foreground of the image and highlight the salient parts, thereby better capturing the features of the local areas.

[0103] To expand on this, the first sub-image F1 is obtained by convolving the deep feature map with 1×1, please refer to Figure 2 , Figure 2 Schematic diagram of the implementation of a preset local feature extraction strategy of the present invention. The first sub-image F1 is processed according to the preset attention strategy to obtain the first attention map M1. The first sub-image F1 and the first attention map M1 are subjected to the Hadamard product between the elements to obtain the first local feature map LFM1. Figure 3 The process of generating the first attention map M1 is marked as the attention mechanism.

[0104] A specific method for determining the first local feature of the local semantic information of the cloud type under the first local feature map LFM1 may be: performing an average pooling operation on the first local feature map LFM1, and inputting the result after the average pooling operation into a fully connected layer for feature dimensionality reduction, so as to obtain the first local feature representing the local semantic information of the cloud type under the first local feature map LFM1, such as Figure 3 shown.

[0105] As a preferred embodiment, the first sub-image is processed according to a preset attention strategy to obtain a first attention image representing the mining result of the most distinctive area, including:

[0106] Activate the first sub-image F1 to determine its corresponding activation feature image Fm;

[0107] The convolution operation and the aggregation normalization operation are performed on the activated feature map Fm in sequence, and the execution result is processed using the Sigmoid activation function to obtain the first attention map M1.

[0108] For details, please refer to Figure 2 , the step of activating the first sub-image F1 to determine its corresponding activation feature map Fm can be: passing the first sub-image F1 through 3×3 convolution, batch normalization layer, 1×1 convolution and batch normalization layer in sequence, and then activating it with the Relu activation function to obtain the activation feature map Fm, passing the activation feature map Fm through 1×1 convolution and batch normalization layer in sequence, and then processing it with the Sigmoid activation function to obtain the first attention map M1.

[0109] As a preferred embodiment, after processing the first sub-image according to the preset attention strategy to obtain the first attention image representing the mining result of the most distinctive area, the method further includes:

[0110] S21: set o=1;

[0111] S22: determining a second pixel point based on a first attention map corresponding to the first sub-image and a second preset relationship to obtain a suppression map corresponding to the second pixel point;

[0112] The second preset relationship is:

[0113]

[0114] Among them, μ k′ is the pixel value of the kth second pixel in the suppression image, μ k is the pixel value of the kth pixel in the first attention map, a is the preset hyperparameter, μ mean is the average pixel value of all pixels in the first attention map, μ std is the standard deviation of the pixel values ​​of all pixels in the first attention map;

[0115] S23: determining the oth second sub-image according to the suppression image and the first sub-image;

[0116] S24: Processing the oth second sub-image according to a preset attention strategy to obtain an oth second attention map, so as to generate an oth second local feature map according to the oth second attention map and the oth second sub-image, and further determining the oth second local feature corresponding to the oth second local feature map;

[0117] S25: Let o=o+1;

[0118] S26: Determine whether o reaches a preset branch number, where the preset branch number is an integer not less than 2; if not, proceed to S27; if yes, proceed to S29;

[0119] S27: determining a new second pixel point according to the o-1th second attention map and the second preset relationship to obtain a new suppression map corresponding to the new second pixel point;

[0120] S28: Generate the oth second sub-graph according to the new suppression graph and the o-1th second sub-graph; and return to S24;

[0121] S29: Concatenate the global feature, the first local feature and the obtained o-1 second local features to obtain a fusion feature corresponding to the target cloud image.

[0122] Specifically, further considering that although the most distinctive area can be mined through the mining of the first local feature, the remaining object specific areas in the complementary area of ​​the cloud map are sometimes ignored. That is to say, there is a contextual relationship between the various cloud areas in the satellite cloud map, which means that if the most distinctive area is simply erased, it may affect the mining of the remaining specific object areas. For this reason, it is necessary to enhance other specific object areas while suppressing the most distinctive area before. In detail, after obtaining the first attention map, the above method is used to determine multiple second local features.

[0123] It should be noted that the preset hyperparameter a here is a parameter used to distinguish the degree of suppression ratio of significant areas and the degree of enhancement ratio of other activated areas. The higher a is, the higher the degree of suppression and enhancement is. Through this suppression-enhancement-based operation, the most distinctive areas in the previous stage will be partially suppressed. At the same time, those other complementary areas that are activated will be enhanced accordingly, maintaining the relationship between the activated areas in the previous stage and the activated areas generated in the next stage.

[0124] It should also be noted that the specific value of the preset number of branches is set according to the training effect and is not particularly limited here, including but not limited to being set to 3; in S23, the method for determining the oth second sub-graph according to the inhibition map and the first sub-graph can be: performing Hadamard product between elements of the inhibition map and the first sub-graph; in step S24, the method for determining the oth second local feature corresponding to the oth second local feature map can be: performing an average pooling operation on the oth second local feature map, and inputting the result after the average pooling operation into the fully connected layer for feature dimensionality reduction, so as to obtain the oth second local feature representing the local semantic information of the cloud type under the oth second local feature map.

[0125] Please refer to Figure 3, where the preset number of branches = 3 is used as an example for explanation. It can be seen that the fusion feature at this time includes the global feature GF, the first local feature LF1, and a total of 2 second local features (the first second local feature LF2 and the second second local feature LF3). Figure 3 In FIG. 1 , the process of generating the first second subgraph F21 is marked as a suppression module.

[0126] As a preferred embodiment, the interpretable deep network is pre-trained based on a cloud map training set, the cloud map training set includes N training cloud maps, each training cloud map corresponds to at least one real image classification label, the total number of categories of real image classification labels corresponding to all training cloud maps is C, and N and C are both integers not less than 1; the training steps of the interpretable deep network include:

[0127] Input the training cloud image into the interpretable deep network to be trained to obtain hash code-like features;

[0128] Input the class hash code features into the classification layer to obtain the corresponding image classification label prediction value;

[0129] Use class hash code features, image classification label prediction values ​​and preset loss functions to train interpretable deep networks;

[0130] The default loss function is:

[0131] L total =λL cls +ηL q +νL b

[0132] L total is the training error, L cls is the multi-label classification loss, λ is the loss of L cls The corresponding first penalty parameter, L q To quantify the loss, η is related to L q The corresponding second penalty parameter, L b is the potential balance loss, ν is related to L b The corresponding third penalty parameter;

[0133] Among them, L cls Determined based on the third preset relationship, the third preset relationship is:

[0134]

[0135] is the i-th predicted value in the image classification label prediction value corresponding to the n-th training cloud map, ω i is the weight value, is the i-th true value in the real image classification label corresponding to the n-th training cloud image, and σ is the sigmoid activation function;

[0136] L q Determined based on the fourth preset relationship, the fourth preset relationship is:

[0137]

[0138] K is the preset hash code length, d n is the class hash code feature corresponding to the nth training cloud image, and e is a K-dimensional vector with a value of 1;

[0139] L b Determined based on the fifth preset relationship, the fifth preset relationship is:

[0140]

[0141] mean(d n ) means d n The average of the median values.

[0142] It should be noted that there are multiple training cloud maps in the cloud map training set. These training cloud maps are divided into multiple batches and input into the interpretable deep network for training (each batch includes N training cloud maps), preferably 64 batches can be set; in addition, unlike general natural images, satellite cloud maps are multispectral images, and their different channels will reflect different physical properties. In this application, the band images of the 1st, 2nd, 3rd and 5th channels of the LSCIDMR-V2 data set are selected as training cloud maps to obtain the input of the interpretable deep network during training, wherein the 1st, 2nd and 3rd channels are visible light bands, providing aerosol physical properties, and the 5th channel is a near-infrared band, providing cloud physical parameter data; in addition, considering that the image resolution of the training cloud maps input into the interpretable deep network may be different, in order to maintain more details of the training cloud maps and enhance the robustness of batch processing, these training cloud maps can be subjected to a series of preprocessing operations, such as uniformly adjusting the size of the cloud maps to 256×256 and normalizing the input data.

[0143] It can be understood that any training cloud map corresponds to at least one real image classification label, so as to use the multi-label annotated training cloud map and the above-mentioned preset loss function to complete the training of the interpretable deep network and achieve the desired training results, wherein the label vector corresponding to any training cloud map is {z1,z2,...,z C}, z i∈{0,1}, where 1 indicates that the training cloud image has the i-th real image classification label, and 0 indicates that the training cloud image does not have the i-th real image classification label. The first penalty parameter λ, the second penalty parameter η, and the third penalty parameter ν can be used to balance the multi-label classification loss, quantization loss, and bit balance loss. The preferred setting of the values ​​of each parameter in this solution is provided as follows: the first penalty parameter λ=0.5, the second penalty parameter η=0.5, and the third penalty parameter ν=0.0002.

[0144] In addition, L q As the quantization loss, it is used to measure the quantization error generated when the fusion features are converted into binary training hash codes; L b As a bit balance loss, it is used to ensure that each training hash code has a 50% probability of being 0 or 1. In the design, all possible real image classification labels included in the training cloud map are regarded as a binary classification problem, that is, a yes or no problem, and the multi-label classification loss L is designed. cls , where σ is the Sigmoid activation function, which is the input The specific operations performed are as follows, which are intended to Mapped to the (0,1) interval:

[0145]

[0146] It can be seen that in this application, a multi-label cloud map training set is used to train an interpretable deep network. Each real image classification label contains the overall semantic information of the cloud map, which can be used as a semantic clue to assist the interpretable deep network, improve the accuracy of the generated hash code, and make the hash code also have rich semantics. After the interpretable deep network training is completed, it can be used to determine the hash code of each historical cloud map in the cloud map data set. After determining the target hash code of the target cloud map, the Hamming distance between the target hash code and the hash code of each historical cloud map can be calculated and sorted, thereby realizing similarity retrieval and finding the historical cloud map that is most similar to the target cloud map.

[0147] Although the disclosure is disclosed as above, the protection scope of the disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the disclosure, and these changes and modifications will fall within the protection scope of the present invention.

[0148] It should also be noted that, in this specification, relational terms such as first, second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not imply any actual relationship or order between these entities or operations.

Claims

1. A satellite cloud image multi-label hash retrieval method based on an interpretable deep network, characterized in that: include: The obtained target cloud map is input as an input item into a pre-trained interpretable deep network to obtain a target hash code of the target cloud map, wherein the interpretable deep network includes a feature learning module for generating an interpretable fusion feature of the target cloud map and a hash learning module for generating the target hash code, the feature learning module includes a global feature learning module and a local feature learning module, the global feature learning module is a unit for learning a single global semantic feature representing the target cloud map, and the local feature learning module is a unit for learning local semantic features of multiple cloud types in different regions of the target cloud map, and then the output of the global feature learning module is spliced ​​and combined with the output of the local feature learning module to obtain the fusion feature; the local feature learning module includes multiple local branches, each of which includes an attention branch network and a suppression module, to obtain multiple local features for expressing local semantic information of multiple cloud types in different regions, so as to generate the target hash code using the fusion feature; a multi-label supervision mechanism is introduced in the hash learning module; Perform similarity measurement based on the target hash code and the hash codes of each historical cloud map in the historical cloud map database, and retrieve a historical cloud map similar to the target cloud map; The steps for generating the fusion features of the target cloud map by the feature learning module in the interpretable deep network are as follows: inputting the target cloud map into the backbone network to obtain a deep feature map; determining the global features representing the global semantic information of the target cloud map according to the deep feature map and the preset global feature extraction strategy; determining the local features representing the local semantic information of multiple cloud types in different regions in the target cloud map based on the deep feature map and the preset local feature extraction strategy; splicing the global features with the local features to obtain fusion features; The specific process of determining the local features representing the local semantic information of cloud types in different regions in the target cloud map based on the deep feature map and the preset local feature extraction strategy includes: convolving the deep feature map to obtain a first sub-map; processing the first sub-map according to the preset attention strategy to obtain a first attention map representing the mining results of the most distinctive regions; generating a first local feature map according to the first attention map and the first sub-map; determining the first local features representing the local semantic information of cloud types under the first local feature map, so as to splice the global features with the first local features, and then obtain the fusion features corresponding to the target cloud map; After the first sub-graph is processed according to the preset attention strategy to obtain the first attention graph representing the mining result of the most distinctive region, the method further includes: S21: set o=1; S22: determining a second pixel point based on a first attention map corresponding to the first sub-image and a second preset relationship to obtain a suppression map corresponding to the second pixel point; The second preset relationship is: Among them, μ k′ is the pixel value of the kth second pixel in the suppression map, μ k is the pixel value of the kth pixel in the first attention map, a is the preset hyperparameter, μ mean is the average pixel value of all pixels in the first attention map, μ std is the standard deviation of the pixel values ​​of all pixels in the first attention map; S23: determining an oth second sub-image according to the suppression image and the first sub-image; S24: Processing the oth second sub-graph according to the preset attention strategy to obtain an oth second attention graph, so as to generate an oth second local feature graph according to the oth second attention graph and the oth second sub-graph, and further determining the oth second local feature corresponding to the oth second local feature graph; S25: Let o=o+1; S26: Determine whether o reaches a preset branch number, where the preset branch number is an integer not less than 2; if not, proceed to S27; if yes, proceed to S29; S27: determining a new second pixel point according to the o-1th second attention map and the second preset relationship to obtain a new suppression map corresponding to the new second pixel point; S28: Generate the oth second sub-graph according to the new suppression graph and the o-1th second sub-graph; and return to S24; S29: splicing the global feature, the first local feature and the obtained o-1 second local features to obtain a fusion feature corresponding to the target cloud image.

2. The satellite cloud image multi-label hash retrieval method of the interpretable deep network as claimed in claim 1, characterized in that: The hash learning module in the interpretable deep network generates the target hash code according to the fusion feature, including: Inputting the fused feature as an input item into a pre-trained hash code learning module to obtain a class hash code corresponding to the fused feature; Determine a target hash code of the target cloud image according to the class hash code and a first preset relationship; The first preset relationship is: Wherein, d is the class hash code, and b is the target hash code.

3. The satellite cloud image multi-label hash retrieval method of the interpretable deep network as claimed in claim 1, characterized in that: According to the deep feature map and the preset global feature extraction strategy, a global feature representing the global semantic information of the target cloud map is determined, including: Performing an average pooling operation on the deep feature map; The result after the average pooling operation is input into the fully connected layer for feature dimensionality reduction to obtain global features that characterize the global semantic information of the target cloud image.

4. The satellite cloud image multi-label hash retrieval method of the interpretable deep network as claimed in claim 1, characterized in that: The first sub-graph is processed according to a preset attention strategy to obtain a first attention graph representing the mining result of the most distinctive region, including: activating the first sub-image to determine its corresponding activation feature image; The convolution operation and the aggregation normalization operation are sequentially performed on the activated feature map, and the execution result is processed using the Sigmoid activation function to obtain the first attention map.

5. The satellite cloud image multi-label hash retrieval method of the interpretable deep network according to any one of claims 1 to 4, characterized in that: The interpretable deep network is pre-trained based on a cloud map training set, wherein the cloud map training set includes N training cloud maps, each of which corresponds to at least one real image classification label, and the total number of categories of real image classification labels corresponding to all training cloud maps is C, where N and C are both integers not less than 1; The training steps of the interpretable deep network include: Inputting the training cloud graph into an interpretable deep network to be trained to obtain a hash code-like feature; Inputting the class hash code feature into the classification layer to obtain the corresponding image classification label prediction value; Training the interpretable deep network using the class hash code features, the image classification label prediction value and a preset loss function; The preset loss function is: L total =λL cls +ηL q +νL b L total is the training error, L cls is the multi-label classification loss, λ is the loss corresponding to L cls The corresponding first penalty parameter, L q To quantify the loss, η is the same as L q The corresponding second penalty parameter, L b is the bit balance loss, ν is related to L b The corresponding third penalty parameter; Among them, the L cls Determined based on a third preset relationship, the third preset relationship is: is the i-th predicted value in the image classification label prediction value corresponding to the n-th training cloud map, ω i is the weight value, is the i-th true value in the real image classification label corresponding to the n-th training cloud image, and σ is the Sigmoid activation function; The L q Determined based on a fourth preset relationship, the fourth preset relationship is: K is the preset hash code length, d n is the class hash code feature corresponding to the nth training cloud image, and e is a K-dimensional vector with a value of 1; The L b Determined based on the fifth preset relationship, the fifth preset relationship is: mean(d n ) represents the d n The average of the median values.

Citation Information

Patent Citations

  • Point cloud multi-mode-based contrast learning pre-training method and device and medium

    CN116152608A

  • Hash image retrieval method and network based on fine-grained similarity relation comparative learning

    CN116955675A

  • Image retrieval method, image retrieval apparatus, image retrieval device and medium

    US20200334287A1

  • Image processing method, image matching method, device and storage medium

    US20210099310A1

  • Method, computer device, and storage medium, for feature fusion model training and sample retrieval

    US20230401833A1