Multimodal remote sensing image fusion classification model construction method based on adversarial learning, classification method and related device
Through the adversarial learning network and self-attention mechanism, the fusion features of multimodal remote sensing images are extracted, and a more accurate and high-level classification results are achieved.
Patent Information
- Application Number
- CN202411073288.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
AI Technical Summary
The existing remote sensing image classification method based on deep learning has shortcomings in multimodal fusion feature extraction, resulting in insufficient multimodal interactive feature extraction and insufficient multi-source heterogeneous information fusion.
Adopting a multimodal remote sensing image fusion classification model based on adversarial learning, the first-level, second-level and third-level modal fusion characteristics are extracted through adversarial learning network and self-attention mechanism, and analyses are used to introduce adversarial loss and cross entropy loss function optimization model to achieve more compact and accurate multimodal fusion feature extraction.
Achieve more accurate multimodal fusion feature extraction, fully explore potential complementary information between multimodal remote sensing images, capture long-term dependence between cross-modal data, and improve the accuracy of classification results and the level of semantic information. .
Smart Images

Figure CN118941871A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image recognition, and is a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, a classification method and related devices. Background Art
[0002] With the rapid development of remote sensing earth observation technology, more and more complementary remote sensing images from different sources can be used for land cover classification. For example, hyperspectral images HSI contain rich spectral features and can be used to distinguish different types of land objects, but HSI has low spatial resolution and is easily affected by the atmosphere. Using only HSI for remote sensing image classification is difficult to meet the needs of a large number of applications that require spatial resolution and radiation information sensitivity. Unlike HSI, LiDAR modal data contains the distance and three-dimensional information of the target surface, has a higher spatial resolution, and can make up for the limitations of HSI in harsh environments. Therefore, the combined use of HSI and LiDAR data can make full use of their complementarity to achieve more reliable and accurate land cover classification.
[0003] In order to make full use of the complementary information advantages of HSI and LiDAR modal data for intelligent interpretation of remote sensing images, many excellent fusion classification algorithms have been proposed, which are mainly divided into traditional methods and deep learning methods. The traditional method mainly adopts the collaborative classification method of feature engineering, uses manual feature extraction technology to extract features from multimodal remote sensing data, and then stacks the extracted features and completes the classification using a classifier; the deep learning method can automatically learn complex intrinsic features from remote sensing images. Although the CNN-based method has achieved good performance in the HSI and LiDAR fusion classification task, it still has the problems of insufficient multimodal interactive feature extraction and insufficient fusion of multi-source heterogeneous information. Summary of the invention
[0004] The present invention provides a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, a classification method and related devices, which overcome the shortcomings of the above-mentioned prior art and can effectively solve the problem of insufficient multimodal fusion feature extraction in the existing remote sensing image classification method based on deep learning.
[0005] One of the technical solutions of the present invention is achieved by the following measures: a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, comprising:
[0006] Obtain a historical multimodal remote sensing image dataset, select M sample pairs in each category of the historical multimodal remote sensing image dataset as a training dataset, and the remaining sample pairs as a test dataset, wherein the historical multimodal remote sensing image dataset includes multiple category groups, each category group includes several sample pairs, and each sample pair includes an HSI modal image sample and a LiDAR modal image sample of size 11×11;
[0007] Using adversarial learning network and self-attention mechanism, the first-level modal fusion features, second-level modal fusion features and third-level modal fusion features of each sample in the training data set are extracted;
[0008] The discriminator and classifier are used to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively. The adversarial loss function and the cross entropy loss function are introduced according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss.
[0009] The discriminator and classifier are optimized using adversarial loss and cross entropy loss, and the training is repeated based on the training data set. After the training termination conditions are met, the corresponding multimodal remote sensing image fusion classification model is output;
[0010] The multimodal remote sensing image fusion classification model is tested using a test data set, and in response to meeting the test evaluation conditions, the multimodal remote sensing image fusion classification model is output.
[0011] The following are further optimizations and / or improvements to the above technical solutions:
[0012] The above method uses adversarial learning network and self-attention mechanism to extract the primary modal fusion features, secondary modal fusion features and tertiary modal fusion features of each sample in the training data set, including:
[0013] The HSI modality image samples and the LiDAR modality image samples are input into the HSI encoder and the LiDAR encoder respectively, wherein the HSI encoder and the LiDAR encoder have the same structure and both include 3 convolution blocks;
[0014] The output features of the convolution blocks at the same position in the HSI encoder and the LiDAR encoder are input into a cross-modal interactive information extraction module based on the self-attention mechanism for feature fusion to obtain the first-level modality fusion features, the second-level modality fusion features and the third-level modality fusion features.
[0015] The above uses the discriminator and classifier to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively. According to the analysis results, the adversarial loss function and the cross entropy loss function are introduced to obtain the corresponding adversarial loss and cross entropy loss, including:
[0016] The three-level modality fusion features are input into the HSI decoder and the LiDAR decoder respectively, and the pseudo HSI modality image and the pseudo LiDAR modality image are output. The HSI decoder and the LiDAR decoder have the same structure, both of which include 3 convolution blocks.
[0017] The pseudo HSI modal image and the original HSI modal image, the pseudo LiDAR modal image and the original LiDAR modal image are respectively input into a discriminator to obtain the corresponding identification probability, and the corresponding adversarial loss is obtained by using the adversarial loss function;
[0018] The convolution block is used to keep the number of channels of the first-level modal fusion feature and the second-level modal fusion feature consistent with the number of channels of the third-level modal fusion feature, and the first-level modal fusion feature and the second-level modal fusion feature are added and concatenated with the third-level modal fusion feature to obtain the fusion feature;
[0019] The three convolutional block groups, the global average pooling layer, and the fully connected layer are used in sequence to perform nonlinear transformation, size transformation, and dimension transformation on the fused features;
[0020] The softmax function is used to calculate the processed fusion features to obtain the classification probability, and the cross entropy loss function is used to obtain the corresponding cross entropy loss.
[0021] The above HSI modality image samples include 64 bands, and the LiDAR modality image samples include 2 bands.
[0022] The second technical solution of the present invention is achieved by the following measures: a multimodal remote sensing image fusion classification method based on adversarial learning, characterized by comprising:
[0023] Obtaining HSI modality images and LiDAR modality images of remote sensing images to be classified;
[0024] The HSI modality image and the LiDAR modality image are input into a multimodal remote sensing image fusion classification model to obtain corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by the multimodal remote sensing image fusion classification model construction method of the adversarial learning.
[0025] The third technical solution of the present invention is achieved by the following measures: a device for constructing a multimodal remote sensing image fusion classification model for adversarial learning, comprising:
[0026] A training data set acquisition unit is used to acquire a historical multimodal remote sensing image data set, and select M sample pairs in each category of the historical multimodal remote sensing image data set as a training data set, and the remaining sample pairs are used as a test data set, wherein the historical multimodal remote sensing image data set includes multiple category groups, each category group includes a number of sample pairs, and each sample pair includes an HSI modal image sample and a LiDAR modal image sample of a size of 11×11;
[0027] The feature extraction unit uses the adversarial learning network and self-attention mechanism to extract the primary modality fusion features, secondary modality fusion features and tertiary modality fusion features of each sample in the training data set;
[0028] The classification analysis unit uses the discriminator and the classifier to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features, and introduces the adversarial loss function and the cross entropy loss function according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss;
[0029] Update the training unit, use adversarial loss and cross entropy loss to optimize the discriminator and classifier, repeat the training based on the training data set, and output the corresponding multimodal remote sensing image fusion classification model after the training termination condition is met;
[0030] The testing unit tests the multimodal remote sensing image fusion classification model using a test data set, and outputs the multimodal remote sensing image fusion classification model in response to meeting the test evaluation conditions.
[0031] The above classification and analysis units include:
[0032] In the first classification analysis module, the three-level modality fusion features are input into the HSI decoder and the LiDAR decoder respectively, and the pseudo HSI modality image and the pseudo LiDAR modality image are output. The HSI decoder and the LiDAR decoder have the same structure and both include 3 convolution blocks. The pseudo HSI modality image and the original HSI modality image, the pseudo LiDAR modality image and the original LiDAR modality image are input into a discriminator respectively to obtain the corresponding identification probability, and the corresponding adversarial loss is obtained by using the adversarial loss function.
[0033] In the second classification analysis module, the number of channels of the first-level modal fusion features and the second-level modal fusion features are kept consistent with the number of channels of the third-level modal fusion features by using convolution blocks, and the first-level modal fusion features and the second-level modal fusion features are added and concatenated with the third-level modal fusion features respectively to obtain fusion features; three convolution block groups, global average pooling layers, and fully connected layers are used in turn to perform nonlinear transformation, size transformation, and dimension transformation on the fusion features; the softmax function is used to calculate the processed fusion features to obtain the classification probability, and the cross entropy loss function is used to obtain the corresponding cross entropy loss.
[0034] The fourth technical solution of the present invention is achieved by the following measures: a multimodal remote sensing image fusion classification method based on adversarial learning, characterized in that it includes:
[0035] A data acquisition unit, which obtains an HSI modality image and a LiDAR modality image of a remote sensing image to be classified;
[0036] A classification unit inputs the HSI modality image and the LiDAR modality image into a multimodal remote sensing image fusion classification model to obtain corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by the method as described in any one of claims 1 to 4.
[0037] The beneficial effects of the present invention include:
[0038] First, the present invention proposes a multi-level interactive fusion network based on adversarial learning. The network is constructed based on GAN to extract multimodal fusion features. Through adversarial training between the generator and the discriminator, more compact and accurate multimodal fusion features are obtained.
[0039] Second, the present invention designs a cross-modal interactive information extraction structure based on the self-attention mechanism, which fully mines the potential complementary information between multimodal remote sensing images, can capture the long-term dependencies between cross-modal data, and realize the mutual enhancement and complementary utilization of information of different modalities.
[0040] Third, the present invention proposes a multimodal feature fusion classification method, which makes full use of the complementarity between low-level, mid-level and high-level features learned by the neural network, obtains higher-level multimodal fusion semantic information, and thus obtains better classification results. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Attached Figure 1 The figure is a flow chart of the model building method of the present invention.
[0042] Attached Figure 2 It is a schematic diagram of the flow of the multimodal fusion feature extraction method of the present invention.
[0043] Attached Figure 3 It is a schematic diagram of the model structure of the present invention.
[0044] Attached Figure 4 It is a schematic diagram of the multimodal fusion feature extraction structure of the present invention.
[0045] Attached Figure 5 It is a schematic diagram of the multimodal fusion feature analysis structure of the present invention.
[0046] Attached Figure 6 It is a schematic diagram of the feature fusion classification structure of the present invention.
[0047] Attached Figure 7 It is the HSI modality remote sensing image used in the simulation in Example 3 of the present invention.
[0048] Attached Figure 8 It is the LiDAR modality remote sensing image used in the simulation in Example 3 of the present invention.
[0049] Attached Fig. 9 This is a map of land feature distribution used in the simulation in Example 3 of the present invention.
[0050] Attached Fig.10 It is a schematic diagram comparing the simulation effects of various simulation classification methods in Example 3 of the present invention.
[0051] Attached Fig.11 The figure is a flow chart of the classification method of the present invention.
[0052] Attached Fig.12 It is a schematic diagram of the structure of the model building device of the present invention.
[0053] Attached Fig.13 It is a schematic diagram of the structure of the classification device of the present invention. DETAILED DESCRIPTION
[0054] The present invention is not limited by the following embodiments, and specific implementation methods can be determined based on the technical solution of the present invention and actual conditions.
[0055] The present invention will be further described below in conjunction with embodiments and drawings:
[0056] Embodiment 1: As attached Figure 1 As shown, the embodiment of the present invention discloses a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, comprising:
[0057] Step S110, obtaining a historical multimodal remote sensing image dataset, selecting M sample pairs in each category of the historical multimodal remote sensing image dataset as a training dataset, and the remaining sample pairs as a test dataset, wherein the historical multimodal remote sensing image dataset includes multiple category groups, each category group includes a number of sample pairs, and each sample pair includes an HSI modality image sample and a LiDAR modality image sample of a size of 11×11;
[0058] Step S120, extracting the primary modality fusion features, the secondary modality fusion features and the tertiary modality fusion features of each sample in the training data set by using the adversarial learning network and the self-attention mechanism;
[0059] Step S130, using a discriminator and a classifier to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features, respectively, and introducing an adversarial loss function and a cross entropy loss function according to the analysis results to obtain corresponding adversarial losses and cross entropy losses;
[0060] Step S140, using adversarial loss and cross entropy loss to optimize the discriminator and classifier, repeating the training based on the training data set, and outputting the corresponding multimodal remote sensing image fusion classification model after the training termination condition is met;
[0061] Step S150, testing the multimodal remote sensing image fusion classification model using a test data set, and outputting the multimodal remote sensing image fusion classification model in response to meeting the test evaluation conditions.
[0062] The embodiment of the present invention discloses a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning. The method extracts modal fusion features based on an adversarial learning network and a self-attention mechanism, and constructs a multimodal remote sensing image fusion classification model by extracting more accurate multimodal fusion features through adversarial training between a generator and a discriminator. The obtained multimodal remote sensing image fusion classification model can more accurately classify multimodal remote sensing images.
[0063] Embodiment 2: The embodiment of the present invention discloses a method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, comprising:
[0064] Step S210, obtaining a historical multimodal remote sensing image dataset, selecting M sample pairs in each category of the historical multimodal remote sensing image dataset as a training dataset, and the remaining sample pairs as a test dataset, wherein the historical multimodal remote sensing image dataset includes multiple category groups, each category group includes several sample pairs, and each sample pair includes an HSI modality image sample and a LiDAR modality image sample of size 11×11.
[0065] In the above steps, the HSI modality image samples include 64 bands, and the LiDAR modality image samples include 2 bands.
[0066] Step S220, using the adversarial learning network and self-attention mechanism, extract the primary modal fusion features, secondary modal fusion features and tertiary modal fusion features of each sample in the training data set.
[0067] The above steps, as shown in the attached Figure 2 As shown, specifically including:
[0068] Step S221, inputting the HSI modality image samples and the LiDAR modality image samples into the HSI encoder and the LiDAR encoder respectively, wherein the HSI encoder and the LiDAR encoder have the same structure and both include 3 convolution blocks;
[0069] As attached Figure 3 As shown in the figure, the HSI encoder consists of three convolution blocks, each of which contains a convolution layer, a batch normalization layer, and an activation function layer. The convolution kernel size of the convolution layer is 3×3, and the number of convolution kernels is 32, 64, and 128 respectively. The input sample size of the HSI encoder is 11×11×64, and the output feature map size is 11×11×128.
[0070] As attached Figure 3 As shown in the figure, the LiDAR encoder consists of 3 convolution blocks, each of which contains a convolution layer, a batch normalization layer, and an activation function layer. The convolution kernel size of the convolution layer is 3×3, and the number of convolution kernels is 32, 64, and 128 respectively. The input sample size of the LiDAR encoder is 11×11×2.
[0071] The above-mentioned HSI encoder and LiDAR encoder constitute the Generator. It should also be noted that a weight sharing strategy is adopted in the last two layers of the encoder to learn highly similar semantic features in the two modalities while reducing the number of network parameters.
[0072] In step S222, the output features of the convolution blocks at the same position in the HSI encoder and the LiDAR encoder are input into a cross-modal interactive information extraction module based on a self-attention mechanism for feature fusion to obtain primary modality fusion features, secondary modality fusion features, and tertiary modality fusion features.
[0073] The above steps are as follows: Figure 4 As shown, the input is the output features of the convolutional blocks at the same position in the HSI encoder and LiDAR encoder. and Its size is h×w×c k , where c k is the number of channels of the convolution operation in the kth convolution block. When performing self-attention calculation, and The corresponding three feature maps K, Q, and V are obtained by copying respectively. For the two modal branches, K and Q come from the same features of themselves, and V comes from the features of the other modal data.
[0074] For the HSI branch, K H and Q H From the HSI feature, V L From the LiDAR data features, first K H and Q H The similarity is measured by dot multiplication to obtain the attention weight with global context information, which is then normalized and combined with V LMultiply to get the cross-attention features:
[0075]
[0076] The size of the attention is reshaped to h×w×c through the reshape operation k , and use residual connection and input features Add the enhanced feature FA H :
[0077]
[0078] For the LiDAR branch, K L and Q L From the HSI feature, V H From the LiDAR data features, first K L and Q L The similarity is measured by dot multiplication to obtain the attention weight with global context information, which is then normalized and combined with V H Multiply to get the cross-attention features:
[0079]
[0080] The size of the attention is reshaped to h×w×c through the reshape operation k , and use residual connection and input features Add the enhanced feature FA L :
[0081]
[0082] The fusion feature with cross-modal global dependency information is obtained by linear addition.
[0083]
[0084] The network structure corresponding to the above process (such as the attached Figure 4 as shown) embedded in the attached Figure 3 The location of CMIIE. Figure 3 From left to right in the figure, CMIIE outputs the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively.
[0085] In step S230, the discriminator and the classifier are used to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively, and an adversarial loss function and a cross entropy loss function are introduced according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss.
[0086] The above steps, as shown in the attached Figure 5 As shown, specifically including:
[0087] Step S231, inputting the three-level modality fusion features into the HSI decoder and the LiDAR decoder respectively, and outputting a pseudo HSI modality image and a pseudo LiDAR modality image, wherein the HSI decoder and the LiDAR decoder have the same structure, both including 3 convolution blocks;
[0088] As attached Figure 3 As shown in the figure, the HSI encoder consists of three convolution blocks, each of which contains a convolution layer, a batch normalization layer, and an activation function layer. The convolution kernel size of the convolution layer is 3×3, and the number of convolution kernels is 64, 32, and 64 respectively. The input size of the HSI decoder is 11×11×128, and the output size is 11×11×64.
[0089] As attached Figure 3 As shown in the figure, the LiDAR encoder consists of 3 convolution blocks, each of which contains a convolution layer, a batch normalization layer, and an activation function layer. The convolution kernel size of the convolution layer is 3×3, and the number of convolution kernels is 64, 32, and 2 respectively. The input size of the HSI decoder is 11×11×128, and the output size is 11×11×2.
[0090] Furthermore, the first two layers of the decoder adopt a weight sharing strategy to learn highly similar semantic features in the two modalities while reducing the number of network parameters.
[0091] Step S232, inputting the pseudo HSI modal image and the original HSI modal image, the pseudo LiDAR modal image and the original LiDAR modal image into a discriminator respectively, obtaining corresponding identification probabilities, and obtaining corresponding adversarial losses using an adversarial loss function;
[0092] As attached Figure 3 As shown in the figure, the discriminator of the HSI branch contains three convolution blocks, each of which contains a convolution layer, batch normalization, activation function layer, maximum pooling and Dropout operations in sequence. It is followed by a global average pooling layer, a fully connected layer and a sigmoid activation function layer. The input data of the discriminator is the pseudo HSI modality image and the original HSI modality image, with a size of 11×11×64, and the output is the identification probability.
[0093] As attached Figure 3As shown in the figure, the discriminator of the LiDAR branch contains three convolution blocks, each of which contains a convolution layer, batch normalization, activation function layer, maximum pooling and Dropout operations in sequence. It is followed by a global average pooling layer, a fully connected layer and a sigmoid activation function layer. The input data of the discriminator is the pseudo LiDAR modality image and the original LiDAR modality image, with a size of 11×11×64, and the output is the identification probability.
[0094] Step S233, using a convolution block to keep the number of channels of the primary modal fusion feature and the secondary modal fusion feature consistent with the number of channels of the tertiary modal fusion feature, and respectively adding and concatenating the primary modal fusion feature and the secondary modal fusion feature with the tertiary modal fusion feature to obtain a fusion feature;
[0095] As attached Figure 6 As shown in the figure, the first-level modal fusion feature, the second-level modal fusion feature and the third-level modal fusion feature are set as Since the number of feature channels at the three levels is different, in order to keep the spatial size of the feature map unchanged, we use a convolution operation with a convolution kernel size of 1×1 to change and The number of channels is The number of channels is the same, which is convenient for the subsequent linear operations, and then Add and Afterwards and The concatenate operation is used for splicing, and the resulting feature dimension size is 11×11×384.
[0096] Step S234, using three convolution block groups, a global average pooling layer, and a fully connected layer in sequence to perform nonlinear transformation, size transformation, and dimension transformation on the fused features;
[0097] As attached Figure 6 As shown in the figure, three 1×1 convolution blocks are used to help the network learn more complex fusion feature representations through nonlinear transformations. The number of convolution kernels in the convolution operation is 256, 128, and 64. Global average pooling is used to reduce the size of the feature map to 1×1, and a fully connected layer is used to reduce the feature dimension to C, where C represents the number of object categories in each dataset.
[0098] Step S235, using the softmax function to calculate the processed fusion features to obtain the classification probability, and using the cross entropy loss function to obtain the corresponding cross entropy loss.
[0099] As attached Figure 6 As shown, the classification probability P is obtained through the softmax function.
[0100]
[0101] in, It is the feature expression obtained after three 1×1 convolution blocks, fc(·) is the fully connected operation, and pool(·) is the global average pooling.
[0102] Step S240, using adversarial loss and cross entropy loss to optimize the discriminator and classifier, repeating the training based on the training data set, and outputting the corresponding multimodal remote sensing image fusion classification model after the training termination condition is met.
[0103] In the above steps, the adversarial loss expression is as follows:
[0104]
[0105] Among them, D h (·) and D1(·) are the discriminator learning of HSI branch and LiDAR branch respectively, G h (·) and G l (·) Generator learning for HSI branch and LiDAR branch respectively.
[0106] In the above steps, the expression of cross entropy loss is as follows:
[0107]
[0108] Among them, N represents the batch size, C is the number of categories, and y ic Is the true label of sample i a One-hot encoding of category c, p ic is the predicted probability that sample i belongs to category c.
[0109] In summary, the joint loss formula is: L = L ce +λL ad , λ is the weight coefficient.
[0110] Step S250, using a test data set to test the multimodal remote sensing image fusion classification model, and in response to meeting the test evaluation conditions, outputting the multimodal remote sensing image fusion classification model.
[0111] Example 3: MUFFL data acquired by CASI-1500 sensor at Gulf Park, University of Southern Mississippi in November 2010, attached Figure 7 is the HSI mode remote sensing image used in the simulation, Figure 8 is the LiDAR modality remote sensing image used in the simulation, Fig. 9 This is a map of the terrain distribution used in the simulation.
[0112] The simulation platform is set as: 13th Gen Intel(R)Core(TM)i7-13700KF CPU, 32GB of memory, NVIDIA GeForce RTX 4080 graphics card and Windows 10 operating system, Pytorch deep learning platform, Python language.
[0113] The remote sensing image classification is simulated, and the results are shown in the attached Fig.10 As shown, Fig.10 (i) is a simulation result diagram of remote sensing image classification using the model constructed by the present invention, Fig.10 (a) to (h) are the results of simulating remote sensing images using existing technologies, where the existing technologies are 10(a) SVM, 10(b) CDCNN, 10(c) TBCNN, 10(d) EndNet, 10(e) FusAtNet, 10(f) S2Enet, 10(g) AMSSE, and 10(h) CALC.
[0114] From the attached Fig.10 It can be seen that the simulation result map of the present invention presents more realistic and finer details, and the classification result is closer to the ground object distribution label map than other methods.
[0115] The quantitative comparison results of the simulation of remote sensing image classification using the model constructed by the present invention are shown in Table 1. It can be seen that the present invention is superior to other methods in three common performance indicators, and the recognition rate of each category is relatively balanced. Specifically, the overall accuracy OA, average accuracy AA and Kappa value obtained by the simulation of the present invention are 93.47%, 93.81% and 91.29% respectively, which is 2.89% higher than the suboptimal model AMSSE in terms of OA index and 12.46% higher than the SVM method.
[0116] Embodiment 4: As attached Fig.11 As shown, the embodiment of the present invention discloses a multimodal remote sensing image fusion classification method based on adversarial learning, comprising:
[0117] Step S410, obtaining an HSI modality image and a LiDAR modality image of a remote sensing image to be classified;
[0118] Step S420, inputting the HSI modal image and the LiDAR modal image into a multimodal remote sensing image fusion classification model to obtain corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by a multimodal remote sensing image fusion classification model construction method based on adversarial learning.
[0119] Embodiment 5: As attached Fig.12As shown, the embodiment of the present invention discloses a device for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, comprising:
[0120] A training data set acquisition unit is used to acquire a historical multimodal remote sensing image data set, and select M sample pairs in each category of the historical multimodal remote sensing image data set as a training data set, and the remaining sample pairs are used as a test data set, wherein the historical multimodal remote sensing image data set includes multiple category groups, each category group includes a number of sample pairs, and each sample pair includes an HSI modal image sample and a LiDAR modal image sample of a size of 11×11;
[0121] The feature extraction unit uses the adversarial learning network and self-attention mechanism to extract the primary modality fusion features, secondary modality fusion features and tertiary modality fusion features of each sample in the training data set;
[0122] The classification analysis unit uses the discriminator and the classifier to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features, and introduces the adversarial loss function and the cross entropy loss function according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss;
[0123] Update the training unit, use adversarial loss and cross entropy loss to optimize the discriminator and classifier, repeat the training based on the training data set, and output the corresponding multimodal remote sensing image fusion classification model after the training termination condition is met;
[0124] The testing unit tests the multimodal remote sensing image fusion classification model using a test data set, and outputs the multimodal remote sensing image fusion classification model in response to meeting the test evaluation conditions.
[0125] The classification analysis units include:
[0126] In the first classification analysis module, the three-level modality fusion features are input into the HSI decoder and the LiDAR decoder respectively, and the pseudo HSI modality image and the pseudo LiDAR modality image are output. The HSI decoder and the LiDAR decoder have the same structure and both include 3 convolution blocks. The pseudo HSI modality image and the original HSI modality image, the pseudo LiDAR modality image and the original LiDAR modality image are input into a discriminator respectively to obtain the corresponding identification probability, and the corresponding adversarial loss is obtained by using the adversarial loss function.
[0127] In the second classification analysis module, the number of channels of the first-level modal fusion features and the second-level modal fusion features are kept consistent with the number of channels of the third-level modal fusion features by using convolution blocks, and the first-level modal fusion features and the second-level modal fusion features are added and concatenated with the third-level modal fusion features respectively to obtain fusion features; three convolution block groups, global average pooling layers, and fully connected layers are used in turn to perform nonlinear transformation, size transformation, and dimension transformation on the fusion features; the softmax function is used to calculate the processed fusion features to obtain the classification probability, and the cross entropy loss function is used to obtain the corresponding cross entropy loss.
[0128] Embodiment 6: As attached Fig.13 As shown, the embodiment of the present invention discloses a multimodal remote sensing image fusion classification method based on adversarial learning, comprising:
[0129] A data acquisition unit, which obtains an HSI modality image and a LiDAR modality image of a remote sensing image to be classified;
[0130] The classification unit inputs the HSI modal image and the LiDAR modal image into the multimodal remote sensing image fusion classification model to obtain the corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by a multimodal remote sensing image fusion classification model construction method based on adversarial learning.
[0131] Embodiment 7: The embodiment of the present invention discloses a storage medium, on which a computer program that can be read by a computer is stored, and the computer program is configured to execute a multimodal remote sensing image fusion classification method based on adversarial learning when running.
[0132] The above storage medium may include, but is not limited to: a USB flash drive, a read-only memory, a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.
[0133] Embodiment 8: The embodiment of the present invention discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement a multimodal remote sensing image fusion classification method based on adversarial learning.
[0134] The processor may be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. It may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The memory may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory, a mobile hard disk, a magnetic disk or an optical disk.
[0135] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0138] The above technical features constitute the best embodiment of the present invention, which has strong adaptability and best implementation effect. Non-essential technical features can be added or reduced according to actual needs to meet the requirements of different situations.
Claims
1. A method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning, characterized in that: include: Obtain a historical multimodal remote sensing image dataset, select M sample pairs in each category of the historical multimodal remote sensing image dataset as a training dataset, and the remaining sample pairs as a test dataset, wherein the historical multimodal remote sensing image dataset includes multiple category groups, each category group includes several sample pairs, and each sample pair includes an HSI modal image sample and a LiDAR modal image sample of size 11×11; Using adversarial learning network and self-attention mechanism, the first-level modal fusion features, second-level modal fusion features and third-level modal fusion features of each sample in the training data set are extracted; The discriminator and classifier are used to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively. The adversarial loss function and the cross entropy loss function are introduced according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss. The discriminator and classifier are optimized using adversarial loss and cross entropy loss, and the training is repeated based on the training data set. After the training termination conditions are met, the corresponding multimodal remote sensing image fusion classification model is output; The multimodal remote sensing image fusion classification model is tested using a test data set, and in response to meeting the test evaluation conditions, the multimodal remote sensing image fusion classification model is output.
2. The method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning according to claim 1, characterized in that: The method uses an adversarial learning network and a self-attention mechanism to extract the primary modality fusion features, the secondary modality fusion features, and the tertiary modality fusion features of each sample in the training data set, including: The HSI modality image samples and the LiDAR modality image samples are input into the HSI encoder and the LiDAR encoder respectively, wherein the HSI encoder and the LiDAR encoder have the same structure and both include 3 convolution blocks; The output features of the convolution blocks at the same position in the HSI encoder and the LiDAR encoder are input into a cross-modal interactive information extraction module based on the self-attention mechanism for feature fusion to obtain the first-level modality fusion features, the second-level modality fusion features and the third-level modality fusion features.
3. The method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning according to claim 1 or 2, characterized in that: The discriminator and the classifier are used to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features respectively, and the adversarial loss function and the cross entropy loss function are introduced according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss, including: The three-level modality fusion features are input into the HSI decoder and the LiDAR decoder respectively, and the pseudo HSI modality image and the pseudo LiDAR modality image are output. The HSI decoder and the LiDAR decoder have the same structure, both of which include 3 convolution blocks. The pseudo HSI modal image and the original HSI modal image, the pseudo LiDAR modal image and the original LiDAR modal image are respectively input into a discriminator to obtain the corresponding identification probability, and the corresponding adversarial loss is obtained using the adversarial loss function; The convolution block is used to keep the number of channels of the first-level modal fusion feature and the second-level modal fusion feature consistent with the number of channels of the third-level modal fusion feature, and the first-level modal fusion feature and the second-level modal fusion feature are added and concatenated with the third-level modal fusion feature to obtain the fusion feature; The three convolutional block groups, the global average pooling layer, and the fully connected layer are used in sequence to perform nonlinear transformation, size transformation, and dimension transformation on the fused features; The softmax function is used to calculate the processed fusion features to obtain the classification probability, and the cross entropy loss function is used to obtain the corresponding cross entropy loss.
4. The method for constructing a multimodal remote sensing image fusion classification model based on adversarial learning according to any one of claims 1 to 3, characterized in that: The HSI modality image samples include 64 bands, and the LiDAR modality image samples include 2 bands.
5. A multimodal remote sensing image fusion classification method based on adversarial learning, characterized in that: include: Obtaining HSI modality images and LiDAR modality images of remote sensing images to be classified; The HSI modality image and the LiDAR modality image are input into a multimodal remote sensing image fusion classification model to obtain corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by the method as described in any one of claims 1 to 4.
6. A device for constructing a multimodal remote sensing image fusion classification model based on adversarial learning using the method described in any one of claims 1 to 4, characterized in that: include: A training data set acquisition unit is used to acquire a historical multimodal remote sensing image data set, and select M sample pairs in each category of the historical multimodal remote sensing image data set as a training data set, and the remaining sample pairs are used as a test data set, wherein the historical multimodal remote sensing image data set includes multiple category groups, each category group includes a number of sample pairs, and each sample pair includes an HSI modal image sample and a LiDAR modal image sample of a size of 11×11; The feature extraction unit uses the adversarial learning network and self-attention mechanism to extract the primary modality fusion features, secondary modality fusion features and tertiary modality fusion features of each sample in the training data set; The classification analysis unit uses the discriminator and the classifier to analyze the first-level modal fusion features, the second-level modal fusion features, and the third-level modal fusion features, and introduces the adversarial loss function and the cross entropy loss function according to the analysis results to obtain the corresponding adversarial loss and cross entropy loss; Update the training unit, use adversarial loss and cross entropy loss to optimize the discriminator and classifier, repeat the training based on the training data set, and output the corresponding multimodal remote sensing image fusion classification model after the training termination condition is met; The testing unit tests the multimodal remote sensing image fusion classification model using a test data set, and outputs the multimodal remote sensing image fusion classification model in response to meeting the test evaluation conditions.
7. The device for constructing a multimodal remote sensing image fusion classification model based on adversarial learning according to claim 6, characterized in that: The classification analysis unit comprises: In the first classification analysis module, the three-level modality fusion features are input into the HSI decoder and the LiDAR decoder respectively, and the pseudo HSI modality image and the pseudo LiDAR modality image are output. The HSI decoder and the LiDAR decoder have the same structure and both include 3 convolution blocks. The pseudo HSI modality image and the original HSI modality image, the pseudo LiDAR modality image and the original LiDAR modality image are input into a discriminator respectively to obtain the corresponding identification probability, and the corresponding adversarial loss is obtained by using the adversarial loss function. In the second classification analysis module, the number of channels of the first-level modal fusion features and the second-level modal fusion features are kept consistent with the number of channels of the third-level modal fusion features by using convolution blocks, and the first-level modal fusion features and the second-level modal fusion features are added and concatenated with the third-level modal fusion features respectively to obtain fusion features; three convolution block groups, global average pooling layers, and fully connected layers are used in turn to perform nonlinear transformation, size transformation, and dimension transformation on the fusion features; the softmax function is used to calculate the processed fusion features to obtain the classification probability, and the cross entropy loss function is used to obtain the corresponding cross entropy loss.
8. A multimodal remote sensing image fusion classification method based on adversarial learning, characterized in that: include: A data acquisition unit, which obtains an HSI modality image and a LiDAR modality image of a remote sensing image to be classified; A classification unit inputs the HSI modality image and the LiDAR modality image into a multimodal remote sensing image fusion classification model to obtain corresponding classification results, wherein the multimodal remote sensing image fusion classification model is obtained by the method as described in any one of claims 1 to 4.
9. A storage medium, characterized in that: The storage medium stores a computer program that can be read by a computer, and the computer program is configured to execute the multimodal remote sensing image fusion classification method based on adversarial learning as claimed in claim 6 when running.
10. An electronic device, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the multimodal remote sensing image fusion classification method based on adversarial learning as claimed in claim 6.
Citation Information
Cited By
Cervical image classification method and system based on wavelet transform multi-modal fusion
CN120997633A
