Multi-modal remote sensing image ground object classification method based on modal information constraint

Abstract

The invention provides a modal information constraint-based multi-modal remote sensing image ground object classification method, which comprises the following steps of: firstly, adaptively learning a fusion weight of modal sharing and modal difference characteristics by adopting an MDF decoder, and adaptively controlling the contribution of modal sharing and modal proprietary information from a multi-scale angle so as to reserve the modal proprietary information; then, an MSIE decoder is adopted to guide modal difference features and multi-modal features to carry out a multi-modal auxiliary classification task so as to enhance optical and SAR proprietary information beneficial to classification; therefore, compared with other multi-mode remote sensing image ground feature classification methods, the method is higher in recognition precision, can effectively relieve the ground feature recognition performance bottleneck in a complex scene, and provides effective support for multi-mode remote sensing image interpretation.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a multi-modal remote sensing image object classification method based on modal information constraints. Background Art

[0002] Remote sensing image object classification is a fundamental task in earth observation, and its main goal is to assign each pixel in the remote sensing image to a predefined object category. This technology has played an important role in urban planning, environmental protection, precision agriculture, etc. Optical images usually contain rich spectral and texture information, but are easily affected by climate and lighting conditions, making it difficult to accurately classify objects. In contrast, synthetic aperture radar (SAR) images can provide structural and scattering features, enhancing the feature diversity of objects. Recently, many studies have focused on multimodal remote sensing image object classification using optical and SAR data to overcome performance bottlenecks.

[0003] In the classification of multimodal remote sensing images, it is crucial to integrate the complementary information of each modality. Among them, exploring the relationship between modality sharing and modality-specific information helps to integrate the complementary information of modalities, thereby achieving a comprehensive interpretation. In earth observation, both optical images and SAR images contain both modality sharing information and modality-specific information. Among them, modality sharing information represents the common characteristics of the two modalities, which helps to bridge the differences between them, such as the shape and position of the objects; modality-specific information represents the unique attributes of each modality that are crucial for accurately describing the objects, such as spectral and scattering characteristics. At present, multimodal remote sensing image fusion methods are mainly based on deep feature-level fusion. These methods usually use fusion operators such as splicing and cross-attention for optical and SAR features, and obtain fusion features by learning fusion weights, thereby indirectly regulating the contribution of modality sharing and modality-specific information, and achieving good classification results.

[0004] However, in challenging scenarios such as complex object distribution in optical images or speckle noise in SAR images, the performance of these existing multimodal remote sensing image fusion methods may have performance bottlenecks. Both optical features and SAR features contain both modality-shared and modality-specific information. Existing methods directly learn the fusion weights of optical and SAR features, which makes it difficult to balance the contribution of modality-shared and modality-specific information, resulting in the loss of modality-specific information. In order to improve the classification performance of multimodal remote sensing image object classification methods in challenging scenarios such as complex object distribution or noisy conditions, the above key issues need to be addressed. Summary of the invention

[0005] In order to solve the problem of loss of modality-specific information in existing multimodal remote sensing image object classification methods, the present invention provides a multimodal remote sensing image object classification method based on modality information constraints, which enhances the representation ability of fusion features and has higher object classification accuracy.

[0006] A multimodal remote sensing image object classification method based on modal information constraint adopts a multimodal classification model to classify objects in a scene to be tested, wherein the multimodal classification model includes an OPT encoder, a SAR encoder, an MDF decoder, an MSIE-OPT decoder and an MSIE-SAR decoder, and the training method of the multimodal classification model is:

[0007] Use OPT encoder to obtain optical image X opt Optical multimodal features at five different scales

[0008]

[0009] Use SAR encoder to obtain SAR image X sar SAR multimodal characteristics at five different scales Among them, the optical image X opt With SAR image X sar To obtain the same scene;

[0010] Multimodal features and Project to the same dimension to obtain the corresponding multimodal projection features and

[0011] Using MDF decoder fusion and Get the modal fusion classification result At the same time, the MDF decoder is used to obtain and The difference between them, we get the modal difference feature

[0012] Using MSIE-OPT decoder to fuse optical multimodal projection features and modal difference characteristics Get the optical fusion classification results

[0013] Using MSIE-SAR decoder to fuse SAR multi-modal projection features and modal difference characteristics Get SAR fusion classification results

[0014] Classification results based on modality fusion Optical fusion classification results SAR fusion classification results Construct loss function L;

[0015] The multimodal classification model is supervised and trained based on the loss function L until a multimodal classification model is obtained in which the loss function L is less than a set value.

[0016] Furthermore, the MDF decoder includes five layers of networks, and except for the first layer of networks which only include the MDF-VSS module and the VSS module, the remaining layers of networks include the MDF-VSS module, the VSS module, the sampling module, the summation module and the residual module;

[0017] The modality fusion classification result The method to obtain is:

[0018] Optical multimodal projection features at all scales and SAR multi-modal projection features The MDF decoder is input into each layer of the network in the order from small scale to large scale; the MDF-VSS module of each layer of the network is used to obtain the modal sharing features of the optical multi-modal projection features and SAR multi-modal projection features received by itself. It is also used to obtain the modal difference characteristics of the optical multi-modal projection characteristics and SAR multi-modal projection characteristics received by itself Then the MDF-VSS module of each layer of the network is also used to fuse the modal shared features obtained by itself Difference between modal features Get the initial fusion features

[0019] The VSS modules of each layer of the network are used to extract the initial fusion features output by the MDF-VSS module of the same layer of the network. Context information, get context features

[0020] The sampling module of the second layer network is used to extract context features from the first layer network. Perform double upsampling; the summation module of the second layer network is used to sum the context features output by the VSS module of the second layer network The residual module of the second layer network is used to fuse the summed features output by the summation module of the network in this layer to obtain the secondary fusion features.

[0021] The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the summation modules of the 3rd to 5th layers of the network are used to sum the context features output by the VSS module of the same layer of the network as themselves. The summed features are added to the secondary fusion features after double upsampling from the previous layer of the network to obtain the summed features; the residual modules of the 3rd to 5th layers of the network are used to fuse the summed features output by the summation modules of the current layers of the network to obtain the secondary fusion features; among them, the secondary fusion features output by the residual module of the 5th layer of the network are the final modal fusion classification results

[0022]

[0023] Furthermore, the modality sharing feature of the i-th layer network The method to obtain is:

[0024]

[0025] in, is the optical multimodal feature of the i-th scale Optical multimodal projection features after projection to the set dimension, is the SAR multimodal feature of the i-th scale SAR multimodal projection features after being projected to the set dimension, VSS(·) indicates that the VSS module with shared parameters is used to perform context feature extraction, and Concat(·) is the concatenation operator;

[0026] Modal difference features of the i-th layer network The method to obtain is:

[0027]

[0028] Initial fusion features of the i-th layer network The method to obtain is:

[0029]

[0030] Among them, Conv(·) represents the convolution module of 3×3 convolution-batch normalization-ReLU, and Res(·) represents the feature fusion operation using the residual module.

[0031] Furthermore, the MSIE-OPT decoder includes a five-layer network, and except that the first layer network only includes a first summation module and a first residual module, the remaining layers of the network include a first summation module, a first residual module, a sampling module, a second summation module, and a second residual module;

[0032] The optical fusion classification results The method to obtain is:

[0033] The optical multimodal features of each scale After projecting to the set dimension, the optical multimodal projection features are obtained

[0034] Optical multimodal projection features at all scales and modal difference characteristics The networks of each layer of the MDF-OPT decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the optical multimodal projection features and modal difference features received by itself to obtain the first sum value feature; the residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial optical fusion feature;

[0035] The sampling module of the second-layer network is used to perform two-fold upsampling on the initial optical fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial optical fusion features output by the first residual module of the second-layer network to the initial optical fusion features from the first-layer network and after two-fold upsampling to obtain a second sum feature; the second residual module is used to perform feature fusion on the second sum feature of the current layer network to obtain a secondary fusion feature;

[0036] The sampling modules of the 3rd to 5th layers of the network are respectively used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are respectively used to add the initial optical fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; among which, the secondary fusion feature output by the second residual module of the 5th layer of the network is the final optical fusion classification result

[0037] Furthermore, the MSIE-SAR decoder includes a five-layer network, and except that the first layer network only includes a first summation module and a first residual module, the remaining layers of the network include a first summation module, a first residual module, a sampling module, a second summation module and a second residual module;

[0038] The SAR fusion classification results The method to obtain is:

[0039] The SAR multimodal features of each scale After projecting to the set dimension, the SAR multimodal projection features are obtained

[0040] SAR multimodal projection characteristics at various scales and modal difference characteristics The networks of each layer of the MDF-SAR decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the SAR multi-modal projection features and modal difference features received by itself to obtain the first sum value feature; the first residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial SAR fusion feature;

[0041] The sampling module of the second-layer network is used to perform two-fold upsampling on the initial SAR fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial SAR fusion features output by the first residual module of the second-layer network to the initial SAR fusion features from the first-layer network after two-fold upsampling to obtain the second sum feature; the second residual module is used to perform feature fusion on the second sum feature of the current layer network to obtain the secondary fusion feature;

[0042] The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are used to add the initial SAR fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; among them, the secondary fusion feature output by the second residual module of the 5th layer of the network is the final SAR fusion classification result

[0043] Furthermore, the loss function L is calculated as:

[0044] L=L fuse +λ(L opt +L sar )

[0045] Among them, L fuse is the cross entropy loss function corresponding to the MDF decoder, L opt is the cross entropy loss function corresponding to the MSIE-OPT decoder, L sar is the cross entropy loss function corresponding to the MSIE-SAR decoder, and λ is the set weight.

[0046] Furthermore, the cross entropy loss function L corresponding to the MDF decoder is fuse The calculation method is:

[0047]

[0048] Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

[0049] Furthermore, the cross entropy loss function L corresponding to the MSIE-OPT decoder is opt The calculation method is:

[0050]

[0051] Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

[0052] Furthermore, the cross entropy loss function L corresponding to the MSIE-SAR decoder is sar The calculation method is:

[0053]

[0054] Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

[0055] Beneficial effects:

[0056] The present invention provides a multimodal remote sensing image object classification method based on modal information constraints. Firstly, an MDF decoder is used to adaptively learn the fusion weights of modal sharing and modal difference features, and the contribution of modal sharing and modal-specific information is adaptively controlled from a multi-scale perspective, so as to retain the modal-specific information; then, an MSIE decoder is used to guide the modal difference features and multimodal features to carry out a multimodal auxiliary classification task, so as to enhance the optical and SAR-specific information that is conducive to classification; therefore, compared with other multimodal remote sensing image object classification methods, the present invention has higher recognition accuracy, can effectively alleviate the bottleneck of object recognition performance in complex scenes, and provide effective support for multimodal remote sensing image interpretation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A principle block diagram of the multimodal classification model provided by the present invention;

[0058] Figure 2 The principle block diagram of the MDF-VSS module provided by the present invention;

[0059] Figure 3 A functional block diagram of the MDF decoder provided by the present invention;

[0060] Figure 4 The principle block diagram of the MSIE-OPT decoder provided by the present invention;

[0061] Figure 5 This is a principle block diagram of the MSIE-SAR decoder provided by the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0063] A multimodal remote sensing image object classification method based on modal information constraints uses a multimodal classification model to classify objects in the scene to be tested, where Figure 1 As shown, the multimodal classification model includes an OPT encoder, a SAR encoder, an MDF decoder, an MSIE-OPT decoder and an MSIE-SAR decoder, and the training method of the multimodal classification model is:

[0064] S1: Use OPT encoder to obtain optical image X opt Optical multimodal features at five different scales

[0065]

[0066] S2: Use SAR encoder to obtain SAR image X sar SAR multimodal characteristics at five different scales Among them, the optical image X opt With SAR image X sar To obtain the same scene;

[0067] It should be noted that at this time, if the multimodal features are projected to the same dimension, the corresponding multimodal projection features can be obtained and

[0068] S3: Using MDF decoder to fuse multimodal projection features and Get the modal fusion classification result At the same time, the MDF decoder is used to obtain and The difference between them, we get the modal difference feature

[0069] Specifically, Figure 3As shown, the MDF decoder includes five layers of networks, and except that the first layer of networks only includes the MDF-VSS module and the VSS module, the remaining layers of networks include the MDF-VSS module, the VSS module, the sampling module, the summation module and the residual module;

[0070] The modality fusion feature The method to obtain is:

[0071] Optical multimodal projection features at all scales and SAR multi-modal projection features The MDF decoder is input into each layer of the network in the order from small scale to large scale; the MDF-VSS module of each layer of the network is used to obtain the modal sharing features of the optical multi-modal projection features and SAR multi-modal projection features received by itself. It is also used to obtain the modal difference characteristics of the optical multi-modal projection characteristics and SAR multi-modal projection characteristics received by itself Then the MDF-VSS module of each layer of the network is also used to fuse the modal shared features obtained by itself Difference between modal features Get the initial fusion features

[0072] like Figure 2 As shown, the working principle of the MDF-VSS module of the i-th layer network is as follows:

[0073] First, for each scale, the multimodal projection features are obtained and Use the VSS modules with shared parameters to map them to the common embedding space to obtain the optical shared features and SAR shared features. Then, concatenate the two projected features in the dimension to obtain the modal shared features of the i-th layer network.

[0074]

[0075] in, is the optical multimodal feature of the i-th scale Optical multimodal projection features after projection to the set dimension, is the SAR multimodal feature of the i-th scale SAR multimodal projection features after being projected to the set dimension. VSS(·) means using shared parameters, that is, VSS modules with exactly the same parameters are used for context feature extraction. Concat(·) is the concatenation operator.

[0076] Then, for each layer, the multimodal projection features obtained and Use subtraction to calculate the difference features of multimodal features, and use the VSS module to model the difference features, extract the contextual information of the difference features for complex objects, and finally obtain the modal difference features of the i-th layer network.

[0077]

[0078] Finally, the modal shared features obtained above are Through the convolution module with the structure of 3×3 convolution-batch normalization-ReLU and modality difference features Add and improve the feature representation ability through the residual module to obtain the initial fusion feature of the i-th layer network

[0079]

[0080] Among them, Conv(·) represents the convolution module of 3×3 convolution-batch normalization-ReLU, and Res(·) represents the feature fusion operation using the residual module.

[0081] The VSS modules of each layer of the network are used to extract the initial fusion features output by the MDF-VSS module of the same layer of the network. Context information, get context features

[0082] The sampling module of the second layer network is used to extract context features from the first layer network. Perform double upsampling in the form of bilinear interpolation; the summation module of the second layer network is used to sum the context features output by the VSS module of the second layer network The residual module of the second layer network is used to fuse the summed features output by the summation module of the network in this layer to obtain the secondary fusion features.

[0083] The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the summation modules of the 3rd to 5th layers of the network are used to sum the context features output by the VSS module of the same layer of the network as themselves. The summed features are added to the secondary fusion features after double upsampling from the previous layer of the network to obtain the summed features; the residual modules of the 3rd to 5th layers of the network are used to fuse the summed features output by the summation modules of the current layers of the network to obtain the secondary fusion features; among them, the secondary fusion features output by the residual module of the 5th layer of the network are the final modal fusion classification results

[0084]

[0085] It can be seen that in the MDF decoder, the fusion features of each layer The VSS modules are used to extract context information from complex objects. These features are then bilinearly interpolated and added to adjacent higher-resolution features. Finally, they are fused through the residual module to output the fused classification results.

[0086] S4: Fusion of optical multimodal projection features using MSIE-OPT decoder and modal difference characteristics Get the optical fusion classification results

[0087] like Figure 4 As shown, the MSIE-OPT decoder includes a five-layer network, and except that the first layer network only includes a first summation module and a first residual module, the remaining layers of the network include a first summation module, a first residual module, a sampling module, a second summation module and a second residual module.

[0088] The optical fusion classification results The method to obtain is:

[0089] The optical multimodal features of each scale After projecting to the set dimension, the optical multimodal projection features are obtained

[0090] Optical multimodal projection features at all scales and modal difference characteristics The networks of each layer of the MDF-OPT decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the optical multimodal projection features and modal difference features received by itself to obtain the first sum value feature; the first residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial optical fusion feature;

[0091] The sampling module of the second-layer network is used to perform two-fold upsampling on the initial optical fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial optical fusion features output by the first residual module of the second-layer network to the initial optical fusion features from the first-layer network after two-fold upsampling to obtain a second sum feature; the second residual module is used to perform feature fusion on the second sum feature of this layer of network to obtain a secondary fusion feature.

[0092] The sampling modules of the 3rd to 5th layers of the network are respectively used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are respectively used to add the initial optical fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; the secondary fusion feature output by the second residual module of the 5th layer of the network is the final optical fusion classification result

[0093] It can be seen that in the MSIE-OPT decoder, each layer of optical projection features And the modal difference characteristics of each layer is sent to the MSIE-OPT decoder; first, each layer of modal difference features The corresponding optical projection characteristics Add; secondly, enhance the difference features through the first residual module SAR-specific information that is beneficial to the classification task; then these features are bilinearly interpolated and added to the adjacent higher-resolution features, and finally fused through the second residual module; finally, the fused classification result is output

[0094] S5: Using MSIE-SAR decoder to fuse SAR multi-modal projection features Modal difference characteristics Get SAR fusion classification results

[0095] like Figure 5 As shown, the MSIE-SAR decoder includes five layers of networks, and except that the first layer of the network only includes the first summation module and the first residual module, the remaining layers of the network include the first summation module, the first residual module, the sampling module, the second summation module and the second residual module.

[0096] The SAR fusion classification results The method to obtain is:

[0097] The SAR multimodal features of each scale After projecting to the set dimension, the SAR multimodal projection features are obtained

[0098] SAR multimodal projection characteristics at various scales and modal difference characteristics The networks of each layer of the MDF-SAR decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the projected SAR multimodal features and modal difference features received by itself to obtain the first sum value feature; the first residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial SAR fusion feature;

[0099] The sampling module of the second-layer network is used to double-upsample the initial SAR fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial SAR fusion features output by the first residual module of the second-layer network and the initial SAR fusion features from the first-layer network after double upsampling to obtain the second sum feature; the second residual module is used to perform feature fusion on the second sum feature of this layer of network to obtain the secondary fusion feature.

[0100] The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are used to add the initial SAR fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; among them, the secondary fusion feature output by the second residual module of the 5th layer of the network is the final SAR fusion classification result

[0101] It can be seen that in the MDF-SAR decoder, each layer of SAR projection features And the modal difference characteristics of each layer is sent to the MSIE-SAR decoder; first, each layer of modal difference features Corresponding SAR projection features Add; secondly, enhance the difference features through the first residual module The optical-specific information that is beneficial to the classification task is obtained; these features are then bilinearly interpolated and added to the adjacent higher-resolution features, and finally fused through the second residual module; the fused classification result is finally output

[0102] S6: Classification results based on modality fusion Optical fusion classification results SAR fusion classification results Construct loss function L;

[0103] Furthermore, the loss function L is calculated as:

[0104] L=L fuse +λ(L opt +L sar )

[0105] Among them, L fuse is the cross entropy loss function corresponding to the MDF decoder, L opt is the cross entropy loss function corresponding to the MSIE-OPT decoder, L sar is the cross entropy loss function corresponding to the MSIE-SAR decoder, λ is a hyperparameter for balancing the contribution of the MSIE decoder, and its empirical value is set to 0.005.

[0106] The cross entropy loss function L corresponding to the MDF decoder fuse The calculation method is:

[0107]

[0108] The cross entropy loss function L corresponding to the MSIE-OPT decoder opt The calculation method is:

[0109]

[0110] The cross entropy loss function L corresponding to the MSIE-SAR decoder sar The calculation method is:

[0111]

[0112] Among them, CE(·) is the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

[0113] S7: Perform supervised training on the multimodal classification model based on the loss function L until a multimodal classification model is obtained in which the loss function L is less than a set value.

[0114] It should be noted that when the multimodal classification model is supervised, in each iteration, if the loss function L is not less than the set value, the value of the weight set in the loss function is changed and Figure 1 The weight parameters of each layer of the network in the OPT encoder, SAR encoder, MDF decoder, MSIE-OPT decoder and MSIE-SAR decoder are used to update the multimodal classification model, and then iterate again until the loss function L meets the requirements and the training is terminated.

[0115] Taking the actual application scenario of multimodal remote sensing image object classification as an example, a variety of existing multimodal remote sensing image object classification methods are compared on multiple public remote sensing data sets. As shown in Table 1 and Table 2, it can be seen that the present invention has better accuracy.

[0116] Table 1 Performance comparison of various existing multimodal remote sensing image object classification methods and embodiments on the DFC20 dataset

[0117]

[0118]

[0119] Table 2 Performance comparison of various existing multimodal remote sensing image object classification methods and embodiments on the WHU-OPT-SAR dataset

[0120]

[0121] In summary, the present invention proposes a multimodal remote sensing image object classification method based on modality-specific information retention and enhancement, and realizes adaptive control of modality sharing and contribution of modality-specific information through multimodal feature decomposition and fusion (MDF)-visual state space (VSS) module, and introduces MDF decoder based on this module to realize adaptive control of multi-scale modality sharing and contribution of modality-specific information, thereby retaining modality-specific information that is beneficial to classification tasks. In addition, by introducing modality-specific information enhancement (MSIE) decoder, auxiliary classification tasks guided by modality difference features are carried out to enhance modality-specific information that is beneficial to classification. It can be seen that the present invention enhances the representation ability of fusion features; after a large number of experiments, the results show that the present invention has stronger performance than other multimodal remote sensing image object classification methods; the method proposed by the present invention has higher recognition accuracy than other multimodal remote sensing image object classification methods, which can effectively alleviate the bottleneck of object recognition performance in complex scenes and provide effective support for multimodal remote sensing image interpretation.

[0122] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal remote sensing image object classification method based on modal information constraints, characterized in that: A multimodal classification model is used to classify objects in the scene to be tested, wherein the multimodal classification model includes an OPT encoder, a SAR encoder, an MDF decoder, an MSIE-OPT decoder and an MSIE-SAR decoder, and the training method of the multimodal classification model is: Use OPT encoder to obtain optical image X opt Optical multimodal features at five different scales Use SAR encoder to obtain SAR image X sar SAR multimodal characteristics at five different scales Among them, the optical image X opt With SAR image X sar To obtain the same scene; Multimodal features and Project to the same dimension to obtain the corresponding multimodal projection features and Using MDF decoder fusion and Get the modal fusion classification result At the same time, the MDF decoder is used to obtain and The difference between them, we get the modal difference feature Using MSIE-OPT decoder to fuse optical multimodal projection features and modal difference characteristics Get the optical fusion classification results Using MSIE-SAR decoder to fuse SAR multi-modal projection features and modal difference characteristics Get SAR fusion classification results Classification results based on modality fusion Optical fusion classification results SAR fusion classification results Construct loss function L; The multimodal classification model is supervised and trained based on the loss function L until a multimodal classification model is obtained in which the loss function L is less than a set value.

2. The multimodal remote sensing image object classification method based on modal information constraint as claimed in claim 1, characterized in that: The MDF decoder includes five layers of networks, and except for the first layer of networks which only include the MDF-VSS module and the VSS module, the remaining layers of networks include the MDF-VSS module, the VSS module, the sampling module, the summation module and the residual module; The modality fusion classification result The method to obtain is: Optical multimodal projection features at all scales and SAR multi-modal projection features The MDF decoder is input into each layer of the network in the order from small scale to large scale; the MDF-VSS module of each layer of the network is used to obtain the modal sharing features of the optical multi-modal projection features and SAR multi-modal projection features received by itself. It is also used to obtain the modal difference characteristics of the optical multi-modal projection characteristics and SAR multi-modal projection characteristics received by itself Then the MDF-VSS module of each layer of the network is also used to fuse the modal shared features obtained by itself Difference between modal features Get the initial fusion features The VSS modules of each layer of the network are used to extract the initial fusion features output by the MDF-VSS module of the same layer of the network. Context information, get context features The sampling module of the second layer network is used to extract context features from the first layer network. Perform double upsampling; the summation module of the second layer network is used to sum the context features output by the VSS module of the second layer network The residual module of the second layer network is used to fuse the summed features output by the summation module of the network in this layer to obtain the secondary fusion features. The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the summation modules of the 3rd to 5th layers of the network are used to sum the context features output by the VSS module of the same layer of the network as themselves. The summed features are added to the secondary fusion features after double upsampling from the previous layer of the network to obtain the summed features; the residual modules of the 3rd to 5th layers of the network are used to fuse the summed features output by the summation modules of the current layers of the network to obtain the secondary fusion features; among them, the secondary fusion features output by the residual module of the 5th layer of the network are the final modal fusion classification results 3. The multimodal remote sensing image object classification method based on modal information constraint as claimed in claim 2, characterized in that: Modality sharing features of the i-th layer network The method to obtain is: in, is the optical multimodal feature of the i-th scale Optical multimodal projection features after projection to the set dimension, is the SAR multimodal feature of the i-th scale SAR multimodal projection features after being projected to the set dimension, VSS(·) indicates that the VSS module with shared parameters is used to perform context feature extraction, and Concat(·) is the concatenation operator; Modal difference features of the i-th layer network The method to obtain is: Initial fusion features of the i-th layer network The method to obtain is: Among them, Conv(·) represents the convolution module of 3×3 convolution-batch normalization-ReLU, and Res(·) represents the feature fusion operation using the residual module.

4. The multimodal remote sensing image object classification method based on modal information constraint as claimed in claim 1, characterized in that: The MSIE-OPT decoder includes a five-layer network, and except that the first layer network only includes a first summation module and a first residual module, the remaining layers of the network include a first summation module, a first residual module, a sampling module, a second summation module and a second residual module; The optical fusion classification results The method to obtain is: The optical multimodal features of each scale After projecting to the set dimension, the optical multimodal projection features are obtained Optical multimodal projection features at all scales and modal difference characteristics The networks of each layer of the MDF-OPT decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the optical multimodal projection features and modal difference features received by itself to obtain the first sum value feature; the residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial optical fusion feature; The sampling module of the second-layer network is used to perform two-fold upsampling on the initial optical fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial optical fusion features output by the first residual module of the second-layer network to the initial optical fusion features from the first-layer network and after two-fold upsampling to obtain a second sum feature; the second residual module is used to perform feature fusion on the second sum feature of the current layer network to obtain a secondary fusion feature; The sampling modules of the 3rd to 5th layers of the network are respectively used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are respectively used to add the initial optical fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; among which, the secondary fusion feature output by the second residual module of the 5th layer of the network is the final optical fusion classification result 5. The multimodal remote sensing image object classification method based on modal information constraint according to claim 1, characterized in that: The MSIE-SAR decoder includes a five-layer network, and except that the first layer network only includes a first summation module and a first residual module, the remaining layers of the network include a first summation module, a first residual module, a sampling module, a second summation module and a second residual module; The SAR fusion classification results The method to obtain is: The SAR multimodal features of each scale After projecting to the set dimension, the SAR multimodal projection features are obtained SAR multimodal projection characteristics at various scales and modal difference characteristics The networks of each layer of the MDF-SAR decoder are input in the order from small scale to large scale; wherein the first summation module of each layer of the network is used to add the SAR multi-modal projection features and modal difference features received by itself to obtain the first sum value feature; the first residual module of each layer of the network is used to perform feature fusion on the first sum value feature output by the first summation module of the same layer of the network to obtain the initial SAR fusion feature; The sampling module of the second-layer network is used to perform two-fold upsampling on the initial SAR fusion features from the first-layer network; the second summation module of the second-layer network is used to add the initial SAR fusion features output by the first residual module of the second-layer network to the initial SAR fusion features from the first-layer network after two-fold upsampling to obtain the second sum feature; the second residual module is used to perform feature fusion on the second sum feature of the current layer network to obtain the secondary fusion feature; The sampling modules of the 3rd to 5th layers of the network are used to double upsample the secondary fusion features from the previous layer of the network; the second summation modules of the 3rd to 5th layers of the network are used to add the initial SAR fusion features output by the first residual module of the same layer of the network with the secondary fusion features from the previous layer of the network after double upsampling to obtain the second sum feature output by the network of this layer; the second residual module is used to perform feature fusion on the second sum feature of the network of this layer to obtain the secondary fusion feature; among them, the secondary fusion feature output by the second residual module of the 5th layer of the network is the final SAR fusion classification result 6. The multimodal remote sensing image object classification method based on modal information constraint according to claim 1, characterized in that: The calculation method of the loss function L is: L=L fuse +λ(L opt +L sar ) Among them, L fuse is the cross entropy loss function corresponding to the MDF decoder, L opt is the cross entropy loss function corresponding to the MSIE-OPT decoder, L sar is the cross entropy loss function corresponding to the MSIE-SAR decoder, and λ is the set weight.

7. The multimodal remote sensing image object classification method based on modal information constraint as claimed in claim 6, characterized in that: The cross entropy loss function L corresponding to the MDF decoder fuse The calculation method is: Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

8. The multimodal remote sensing image object classification method based on modal information constraint as claimed in claim 6, characterized in that: The cross entropy loss function L corresponding to the MSIE-OPT decoder opt The calculation method is: Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

9. The multimodal remote sensing image object classification method based on modal information constraint according to claim 6, characterized in that: The cross entropy loss function L corresponding to the MSIE-SAR decoder sar The calculation method is: Where CE(·) represents the cross entropy loss function, and Y represents the optical image X used in training the multimodal classification model. opt With SAR image X sar The corresponding scene feature label.

Images