A robust land cover classification method and system based on multimodal relationship modeling and fusion network

Abstract

This invention proposes a robust land cover classification method and system based on multimodal relationship modeling and fusion networks. First, by analyzing the potential relationships between and within modalities, it proposes the hypothesis that high-frequency information corresponds to unique features within a modality, while low-frequency information corresponds to shared features between modalities. This modeling of intermodal relationships enables effective feature extraction, laying the foundation for subsequent feature fusion. Simultaneously, based on the criterion that unique features within a modality are uncorrelated while shared features between modalities are correlated, correlation constraints are applied to the extracted unique and shared features to further optimize the feature fusion process and guide the optimization of the feature extraction process. Based on the mutual promotion of feature extraction and feature fusion processes, effective extraction of multimodal features is achieved, enabling high-precision and robust multimodal fusion learning even in scenarios with missing input modalities. Finally, experiments demonstrate that the overall land cover classification accuracy of this invention is superior to existing methods.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and relates to a robust ground feature classification method based on multimodal relationship modeling and fusion network, which is applicable to robust ground feature classification application scenarios in modality missing scenarios. Background Technology

[0002] Remote sensing image land cover classification is widely used in Earth observation tasks such as environmental monitoring and land cover analysis, and it forms the basis for subsequent identification and segmentation tasks based on remote sensing images. Therefore, achieving high-precision and robust remote sensing image land cover classification is crucial. Research on remote sensing image land cover classification methods based on single-modal data has been ongoing and has achieved good results. However, the classification accuracy of these methods is limited by the effective information provided by the input modal data. For example, hyperspectral (HS) and multispectral (MS) data can reflect the material information of the detected target, but it is difficult to distinguish different objects of the same material; LiDAR (light detection and ranging) and digital surface model (DSM) contain elevation information of land cover, but it is difficult to distinguish land cover with similar spatial structures; synthetic aperture radar (SAR) can provide information on the Earth's surface structure, but image interpretation is difficult. In recent years, land cover classification methods based on multimodal remote sensing data fusion have played a significant role in the field of computer vision. By leveraging the complementary advantages of different modalities of remote sensing data, more efficient remote sensing image interpretation and high-precision land cover classification can be achieved. However, in real-world scenarios, the completeness and effectiveness of multimodal remote sensing data are often unreliable. Incomplete modal data input significantly reduces the effectiveness of such methods, making them difficult to apply in practice. Therefore, how to achieve high-precision and robust land cover classification methods based on multimodal remote sensing data fusion under the condition that input modal data may be missing warrants further investigation.

[0003] To achieve high-precision and robust multimodal fusion learning in scenarios with missing input modalities, existing research methods can be divided into three categories: generative learning-based methods, distillation learning-based methods, and unified model-based methods. Generative learning-based methods generate data with the same distribution as the missing modal data using a generative network, and then utilize traditional multimodal fusion learning algorithms for classification. For example, Kumar et al. proposed a general network architecture that can approximately generate missing modal data from available modal data during the classification decision stage (Kumar S, Banerjee B, Chaudhuri S. Improved landcover classification using online spectral data hallucination[J]. Neurocomputing,2021,439:316-326). However, the quality of the generated data directly affects the classification accuracy. Furthermore, unknown noise generated during data generation can also affect the model's stability. The key to distillation learning-based methods is learning the missing modal feature-level information from complete modal data, which requires the model to have complete modal data input during the training phase. For example, Wei et al. designed a new joint adaptive distillation method to overcome the heterogeneity of multimodal features (Wei S, Luo Y, Ma X, et al. MSH-Net: Modality-shared hallucination with joint adaptation distillation for remote sensing image classification using missing modalities[J]. IEEE Transactions on Geoscience and RemoteSensing, 2023, 61: 1-15). Such methods often assume that the modality type of the missing data is known, and a specific model needs to be trained for each modality missing case. Recently, the unified model-based method has achieved excellent results in the field of medical image segmentation. It learns joint feature representations based on the relationship between different modal data and randomly prunes modal data during the training and inference stages to simulate the modality missing case in actual applications. However, when dealing with remote sensing image land cover classification tasks, the unified model-based method cannot meet the high-precision classification requirements, mainly because: (1) A common practice of the unified model-based method is to use an independent encoder to extract the unique features of each modality data, and then fuse the specific features from different modal data for the next classification step. These methods model the relationships between modalities based on features extracted by different encoders, and often focus more on the differences between modalities.For remote sensing image land cover classification tasks based on multimodal data, overemphasizing the differences between different modalities is not conducive to effectively integrating the advantages of multimodality. Therefore, it is necessary to model the relationship between different modalities before feature extraction. (2) An effective way to model the relationship between multimodalities is to divide multimodal features into shared features between modalities and specific features within modalities. For example, Wang et al. designed a ShaSpec network framework to learn shared and specific features of multimodalities based on distribution alignment and domain classification auxiliary tasks (Wang H, Chen Y, Ma C, et al. Multi-modal learning with missing modality via shared-specific feature modelling[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2023:15878-15887). The existing general strategy for extracting shared features between modalities and unique features within modalities is to use an independent encoder for each modality's data to extract its unique features, and to use a shared encoder for all modalities to capture the common features across different modalities as shared features. This strategy does not deeply analyze the potential relationships between and within modalities, making it difficult to accurately extract shared and unique features from multimodal data, and its performance is limited in remote sensing image tasks with large differences in target size and spectral features. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a robust land cover classification method based on multimodal relationship modeling and fusion networks. First, by analyzing the potential relationships between and within modalities, it proposes the hypothesis that high-frequency information corresponds to modal-specific features, while low-frequency information corresponds to shared features between modalities. This modeling of intermodal relationships enables effective feature extraction, laying the foundation for subsequent feature fusion. Simultaneously, based on the criterion that intramodal-specific features are uncorrelated while intermodal shared features are correlated, correlation constraints are applied to the extracted specific and shared features to further optimize the feature fusion process and guide the optimization of the feature extraction process. Through the mutual promotion of feature extraction and feature fusion processes, effective extraction of multimodal features is achieved, enabling high-precision and robust multimodal fusion learning even in scenarios with missing input modalities.

[0005] The technical solution adopted in this invention is a robust land cover classification method based on a multimodal relationship modeling and fusion network. First, the multimodal remote sensing data is normalized. For each modality, a 7×7 data block centered at one pixel is cropped and used as input to the multimodal relationship modeling and fusion network. For the input modal data block, an initial encoder is constructed to extract modal base features. Then, a dual-branch structure consisting of a specific encoder and a shared encoder is constructed to extract intra-modal specific features and inter-modal shared features, respectively. A cross-modal fusion module is constructed to fuse intra-modal specific features and inter-modal shared features. Then, based on the dual-branch structure consisting of the initial encoder, specific encoder, and shared encoder, and the cross-modal fusion module, a multimodal relationship modeling and fusion network for robust remote sensing image land cover classification is constructed, and a loss function suitable for this network is designed. Finally, based on different modal missing conditions, the model was trained on the multimodal remote sensing datasets Houston2013, Trento, and MUUFL (composed of hyperspectral and lidar data) and Augsburg (composed of hyperspectral, synthetic aperture radar, and digital surface models), respectively. The resulting robust land cover classification model was then used to classify land cover in the remote sensing images. The method includes the following steps:

[0006] Step 1: Acquire multimodal remote sensing data of the same target or scene, normalize the remote sensing data of each modality and then crop it to obtain data blocks of fixed size;

[0007] Step 2: Construct a multimodal relationship modeling and fusion network for robust remote sensing image land cover classification. The multimodal relationship modeling and fusion network includes a dual-branch structure consisting of an initial encoder, a unique encoder, and a shared encoder, and a cross-modal fusion module, and takes the cropped data block as input.

[0008] The initial encoder is used to extract features from the data blocks of the input network to obtain modal basis features;

[0009] The dual-branch structure consisting of the unique encoder and the shared encoder comprises two parts: a unique encoder and a shared encoder. The unique encoder is used to further extract intra-modal unique features from the extracted modal basis features, and the shared encoder is used to further extract inter-modal shared features from the extracted modal basis features. After extracting the intra-modal unique features and inter-modal shared features for each modal data block in the input network, all intra-modal unique features and inter-modal shared features are summed to obtain the total intra-modal unique features and the total inter-modal shared features.

[0010] The cross-modal fusion module is used to fuse features unique within the total modality and features shared between the total modalities to obtain fused features;

[0011] Finally, the fused features are used to obtain the land cover classification results through a global average pooling layer and a fully connected layer;

[0012] Step 3: In the case of modality missing scenarios, a multimodal relationship modeling and fusion network is trained by combining the loss function, and the trained network is used to achieve robust remote sensing image land cover classification.

[0013] Furthermore, the normalization process described in step one refers to linear normalization, which is used to scale the data to the [0,1] interval, expressed as:

[0014]

[0015] in, It is the raw remote sensing data of the i-th mode input to the network. These are the minimum and maximum values ​​in the original remote sensing data of the i-th modality, respectively, M. (i) It is the normalized remote sensing data of the i-th modality.

[0016] Furthermore, the initial encoder is implemented by a residual convolution module, which includes a 1×1 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, a 1×1 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, and a residual branch. The residual branch includes a 1×1 convolutional layer and a BatchNorm normalization layer. The input to the branch is the data block input to the initial encoder, and the output of the branch is added to the input features of the last ReLU activation function layer in the residual convolution module. The ReLU activation function is:

[0017]

[0018] x represents the input feature of the ReLU activation function.

[0019] Furthermore, in the encoder initialization, the normalized and cropped data blocks are used as input, and after processing by the residual convolution module, modal basis features are obtained. The implementation process is represented as follows:

[0020]

[0021] in: For the i-th modal remote sensing data M (i) The data block after trimming. It is the initial encoder for the i-th mode. It is the modal basis feature of the i-th mode obtained after the operation.

[0022] Furthermore, the specific encoder is implemented by a residual convolution module. In the specific encoder, the modal basis features are taken as input and passed through the residual convolution module to obtain the in-modal specific features. The implementation process can be represented as follows:

[0023]

[0024] in: It is a unique encoder for the i-th mode. It is the modal basis feature of the i-th mode. It is the mode-specific feature of the i-th mode obtained after the operation.

[0025] Furthermore, the shared encoder is composed of several ConvNeXt convolutional modules and convolutional modules connected in an alternating manner; the ConvNeXt convolutional module includes a 7×7 convolutional layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, a fully connected layer, and a residual branch; the residual branch adds the modal basis features input to the shared encoder with the features output by the ConvNeXt convolutional module; the convolutional module includes a LayerNorm normalization layer and a 3×3 convolutional layer.

[0026] Furthermore, in the shared encoder, the modal basis features are taken as input and sequentially processed by k ConvNeXt convolutional modules and k-1 convolutional modules to obtain the shared features between modalities. The implementation process is represented as follows:

[0027]

[0028] in: It is a shared encoder for the i-th mode. It is the modal basis feature of the i-th mode. It is the intermodal shared feature of the i-th mode obtained after the operation;

[0029] After extracting the intra-modal specific features and inter-modal shared features from the data blocks of each modality in the input network, the intra-modal specific features and inter-modal shared features of all modalities are summed to obtain the total intra-modal specific features and total inter-modal shared features. The implementation process is represented as follows:

[0030]

[0031] Where: n is the number of modalities in the input multimodal relationship modeling and fusion network, F specific It is a feature specific to the overall mode, F shared It is a shared feature among all modes.

[0032] Furthermore, in the cross-modal fusion module, the intra-modal specific features and inter-modal shared features are used as inputs. The intra-modal specific features are first processed by a LayerNorm normalization layer, and the result is used as the query feature Q. The inter-modal shared features are processed by a LayerNorm normalization layer, and the results are used as the key feature K and value feature V. Then, the three features Q, K, and V are input to the attention module. The result is added to Q and then processed sequentially by a LayerNorm normalization layer, a 3×3 convolutional layer, a BatchNorm normalization layer, and a ReLU activation function layer to obtain the fused features. The implementation process is as follows:

[0033] F fused =C(F specific ,F shared )

[0034] Where C is the cross-modal fusion module, F fused This is the fused feature output by the cross-modal fusion module; the attention module includes a dot product layer, a scaling layer, a masking layer, a SoftMax activation function layer, and a dot product layer; after performing a dot product operation on Q and K, the result is sequentially processed through the scaling layer, the masking layer, and the SoftMax activation function layer, and the result is then dot-producted with V before being output; the expression for the SoftMax activation function is:

[0035]

[0036] Where: C is the total number of elements in the input feature z of the Softmax activation function, z a and z b These are the a-th and b-th elements in feature z, respectively.

[0037] Furthermore, the loss function used consists of three parts: the first is to guide the optimization of the feature extraction process through modal correlation constraints, which can be expressed as:

[0038]

[0039] Where: L cor is the modal association constraint loss, ∈ is a constant used to ensure that the modal association constraint loss always remains positive; P is the correlation coefficient loss function, based on the Pearson correlation coefficient. When the number of modal types in the input network is n=2, calculating the correlation coefficient loss function is equivalent to calculating the Pearson correlation coefficient between the features of two modes, expressed as:

[0040]

[0041] Where: ρ 1,2The feature f represents the relationship between mode 1 and mode 2. (1) ,F (2) The Pearson correlation coefficient, W and H are modal features F (1) ,F (2) The dimensions are width and height, F x,y Let F represent the feature value of the pixel at coordinates (x, y); when the number of modalities in the input network n≥2, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, expressed as:

[0042]

[0043] Where: ρ i,j The Pearson correlation coefficient represents the relationship between the features of the i-th mode and the features of the j-th mode.

[0044] Secondly, it calculates the accuracy of the land cover classification prediction results output by each part of the network based on the cross-entropy loss function, including the classification prediction result loss of the final network output, the classification prediction result loss of the modality-specific encoder output, and the classification prediction result loss of the shared encoder output; the classification prediction result loss function is expressed as:

[0045] L cls =L pred +L specific +L shared

[0046] Where: L cls It is the loss of the classification prediction result, L pred It is the loss of the classification prediction result output by the network at the end, L specific It is the loss of the classification prediction result output by the modality-specific encoder, L shared The loss function for the classification prediction results output by the modality-shared encoder is: The cross-entropy loss function is:

[0047]

[0048] Where: M is the number of categories, N is the number of samples, and s ec p is a symbolic function, taking the value 0 or 1. ec The predicted probability that observed sample e belongs to category c;

[0049] Thirdly, the error between the classification prediction results output by the mode-specific encoder and the classification prediction results output by the shared encoder is calculated based on the mean squared error loss function, and used as the cross-modal loss to maintain stability and balance between the two encoders; the mean squared error loss function is:

[0050]

[0051] Where: de and These are the actual observed value and the predicted value of observed sample e, respectively;

[0052] Therefore, the final network loss function is expressed as:

[0053] L = L cls +αL cor +L cro

[0054] Where: L cro It is the cross-modal loss, where α is a constant used to define the weights of the modal association constraint function.

[0055] The present invention also provides a robust land cover classification system based on multimodal relationship modeling and fusion network, characterized in that it includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a robust land cover classification method based on multimodal relationship modeling and fusion network as described above.

[0056] Compared with existing technologies, the advantages and beneficial effects of this invention are as follows: This invention proposes a robust land cover classification method based on multimodal relationship modeling and fusion networks. First, by analyzing the potential relationships between and within modalities, it proposes the hypothesis that high-frequency information corresponds to unique features within a modality, while low-frequency information corresponds to shared features between modalities. This modeling of intermodal relationships enables effective feature extraction, laying the foundation for subsequent feature fusion. Simultaneously, based on the criterion that unique features within a modality are uncorrelated, while shared features between modalities are correlated, correlation constraints are applied to the extracted unique and shared features to further optimize the feature fusion process and guide the optimization of the feature extraction process. Based on the mutual promotion of feature extraction and feature fusion processes, effective extraction of multimodal features is achieved, enabling high-precision and robust multimodal fusion learning in scenarios with missing input modalities. By setting a modal data missing rate to control the integrity of multimodal data input to the network at a certain ratio, this invention simulates modal missing scenarios that may occur in the real world. On the multimodal remote sensing datasets Houston2013, Trento, and MUUFL, which consist of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg, which consists of hyperspectral, synthetic aperture radar, and digital surface models, the overall land cover classification accuracy of this invention is better than that of existing methods. It can achieve high-precision and robust land cover classification under various input modal missing scenarios. Attached Figure Description

[0057] Figure 1 This is a structural diagram of the residual convolution module.

[0058] Figure 2 This is a diagram of the shared encoder structure.

[0059] Figure 3 This is a structural diagram of the cross-modal fusion module.

[0060] Figure 4 It is a multimodal relationship modeling and fusion network structure diagram. Detailed Implementation

[0061] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustrative purposes only and are not intended to limit the present invention.

[0062] This invention primarily addresses the robustness requirements of remote sensing image land cover classification applications. We propose a robust land cover classification method based on multimodal relationship modeling and fusion networks. First, by analyzing the latent relationships between and within modalities, we propose the hypothesis that high-frequency information corresponds to modal-specific features, while low-frequency information corresponds to shared features between modalities. This modeling of intermodal relationships enables effective feature extraction, laying the foundation for subsequent feature fusion. Simultaneously, based on the criterion that intramodal-specific features are uncorrelated while intermodal shared features are correlated, we impose correlation constraints on the extracted specific and shared features to further optimize the feature fusion process and guide the optimization of the feature extraction process. Through the mutual promotion of feature extraction and feature fusion processes, we achieve effective extraction of multimodal features, enabling high-precision, robust multimodal fusion learning even in scenarios with missing input modalities.

[0063] This embodiment provides a robust land cover classification method based on multimodal relationship modeling and fusion networks to achieve robust land cover classification in remote sensing images. Specifically, it includes the following steps:

[0064] Step 1: Acquire multimodal remote sensing data of the same target or scene, and normalize the remote sensing data for each modality. Normalization refers to linear normalization, used to scale the data to the [0,1] interval, which can be expressed as:

[0065]

[0066] in, It is the raw remote sensing data of the i-th mode input to the network. These are the minimum and maximum values ​​in the original remote sensing data of the i-th modality, respectively, M( i ) represents the normalized remote sensing data of the i-th modality. Then, the normalized remote sensing data is cropped into a 7×7 data block centered at one pixel.

[0067] Step 2: Construct the initial encoder, implemented by the residual convolution module, with the structure shown in the attached figure. Figure 1As shown. The residual convolutional module includes a 1×1 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, a 1×1 convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, and a residual branch. The residual branch includes a 1×1 convolutional layer and a BatchNorm normalization layer. The input of the branch is the data block used to initialize the encoder, and the output of the branch is added to the input features of the last ReLU activation function layer in the residual convolutional module. The ReLU activation function is:

[0068]

[0069] x represents the input feature of the ReLU activation function.

[0070] Step 2.1: In the encoder initialization, a normalized and cropped 7×7 data block is used as input, and after processing by the residual convolution module, modal basis features are obtained. The implementation process can be represented as follows:

[0071]

[0072] in: For the i-th modal remote sensing data M (i) After trimming, the data block has a size of 7×7. It is the initial encoder for the i-th mode. It is the modal basis feature of the i-th mode obtained after the operation.

[0073] Step 3: Analyze the potential relationships between and within modes. Due to differences in imaging principles and detection technologies, the edge and detail information reflected in the data for the same target represented by different modes may differ, but the overall image and contour should be similar. In digital image processing, low-frequency features of an image can reflect the general contour, while high-frequency features can represent edge and detail information. Therefore, for the same target using different detection technologies, low-frequency features should have commonalities, reflecting shared information between multiple modes, while high-frequency features should represent the unique features within a specific mode.

[0074] Step 4: Construct a unique encoder, implemented using a residual convolution module, with the structure shown in the attached figure. Figure 1 As shown. Utilizing the advantage of convolutional computation in effectively extracting features from the local receptive field, the unique encoder can further extract high-frequency information from the extracted modal basis features as in-modality unique features.

[0075] Step 4.1: In the specific encoder, the modal basis features are taken as input and passed through the residual convolution module to obtain the in-modal specific features. The implementation process can be represented as follows:

[0076]

[0077] in: It is a unique encoder for the i-th mode. It is the mode-specific feature of the i-th mode obtained after the operation.

[0078] Step 5: Build a shared encoder, the structure of which is shown in the attached figure. Figure 2 As shown, the shared encoder consists of four ConvNeXt convolutional modules and three convolutional modules connected in an alternating manner. The ConvNeXt convolutional modules, by expanding the receptive field and using larger convolutional kernels (from 3×3 to 7×7), can capture more spatial information, enhance feature extraction capabilities, and effectively grasp long-distance dependencies, thereby extracting low-frequency features. The shared encoder utilizes convolutional layers of different scales and depths to capture different features in the image, better adapting to complex image scenes and further extracting low-frequency information from the extracted modality basis features as inter-modality shared features. The ConvNeXt convolutional modules include a 7×7 convolutional layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, a fully connected layer, and a residual branch. The residual branch adds the modality basis features input to the shared encoder to the features output by the ConvNeXt convolutional modules. The convolutional modules include a LayerNorm normalization layer and a 3×3 convolutional layer.

[0079] Step 5.1: In the shared encoder, the modal basis features are taken as input and sequentially processed by 4 ConvNeXt convolutional modules and 3 convolutional modules to obtain the shared features between modalities. The implementation process can be represented as follows:

[0080]

[0081] in: It is a shared encoder for the i-th mode. It is the intermodal shared feature of the i-th mode obtained after the operation.

[0082] Step 5.2: After extracting the intra-modal specific features and inter-modal shared features for each data block of the input network, sum all the intra-modal specific features and inter-modal shared features to obtain the total intra-modal specific features and total inter-modal shared features. The implementation process can be represented as follows:

[0083]

[0084] Where: n is the number of modalities in the input multimodal relationship modeling and fusion network, F specific It is a feature specific to the overall mode, F sharedIt is a shared feature among all modes.

[0085] Step 6: Construct the cross-modal fusion module, the structure of which is shown in the attached figure. Figure 3 As shown, the cross-modal fusion module, based on a cross-attention mechanism, can capture the correlations between input multimodal features. Simultaneously, it can utilize unique information within each modality to guide the feature representation of shared information between modalities. Thus, by leveraging the differences and correlations between multimodal features, efficient multimodal feature fusion is achieved.

[0086] Step 6.1: In the cross-modal fusion module, the features specific to the overall modality and the shared features between the overall modalities are used as input. The features specific to the overall modality are first processed by a LayerNorm normalization layer, and the result is used as the query feature (Query, Q). The shared features between the overall modalities are processed by a LayerNorm normalization layer, and the results are used as the key feature (Key, K) and value feature (Value, V). Then, the three features Q, K, and V are input into the attention module. The result is added to Q and then processed sequentially by a LayerNorm normalization layer, a 3×3 convolutional layer, a BatchNorm normalization layer, and a ReLU activation function layer to obtain the fused feature. The implementation process can be represented as follows:

[0087] F fused =C(F specific ,F shared )

[0088] Where C is the cross-modal fusion module, F fused This is the fused feature output by the cross-modal fusion module. The attention module includes a dot product layer, a scaling layer, a masking layer, a SoftMax activation function layer, and another dot product layer. After performing a dot product operation on Q and K, the result is sequentially processed through the scaling layer, the masking layer, and the SoftMax activation function layer. Finally, the result is dot-producted with V and output. The expression for the SoftMax activation function is:

[0089]

[0090] Where: C is the total number of elements in the input feature z of the Softmax activation function, z a and z b These are the a-th and b-th elements in feature z, respectively.

[0091] Step 7: Construct a multimodal relationship modeling and fusion network for robust remote sensing image land cover classification, the structure of which is shown in the attached figure. Figure 4As shown. Using the encoder initialization described in steps 2, 4, and 5, the unique encoder and the shared encoder are added, along with the cross-modal fusion module described in step 6. The fused features are then passed through a global average pooling layer and a fully connected layer to obtain the land cover classification result. The implementation process can be represented as follows:

[0092] Pred = FC(Pool(F) fused ))

[0093] Where FC and Pool are the fully connected layer and global average pooling layer operations, respectively, and Pred is the obtained land cover classification result.

[0094] Step 7.1 involves normalizing and pruning the multimodal data into 7×7 blocks, i.e., attaching... Figure 4 middle Input the initial encoder of each mode to obtain the basic features of each mode, i.e., the appended features. Figure 4 middle

[0095] Step 7.2: For each input mode, the modal basis features obtained through its initial encoder are fed into the specific encoder and shared encoder described in steps 4 and 5 to obtain the intra-modal specific features and inter-modal shared features of each mode, i.e., the appended features. Figure 4 middle and

[0096] Step 7.3: Sum all intra-modal specific features and inter-modal shared features to obtain the total intra-modal specific features and the total inter-modal shared features, i.e., appendix. Figure 4 F in specific ,F shared ;

[0097] Step 7.4: The obtained intra-modal specific features and inter-modal shared features are fed into the cross-modal fusion module described in step 6. Utilizing the differences and correlations between multimodal features, efficient multimodal feature fusion is achieved to obtain the fused features, i.e., the attached features. Figure 4 F in fused ;

[0098] Step 7.5: The fused features are passed through a global average pooling layer and a fully connected layer to obtain the land cover classification result in this embodiment, i.e., the attached... Figure 4 Pred in the text.

[0099] Step 8: Design a loss function suitable for this network. The loss function consists of three parts: First, it guides the optimization of the feature extraction process through modal correlation constraints: based on the criterion that intra-modal features are uncorrelated while inter-modal shared features are correlated, correlation constraints are applied to the extracted unique features and shared features to further optimize the feature fusion process and guide the optimization of the feature extraction process, achieving mutual enhancement between feature extraction and feature fusion. The modal correlation constraint function can be expressed as:

[0100]

[0101] Where: L cor is the modal association constraint loss, where ∈ is a constant of 1.01 to ensure that the modal association constraint loss always remains positive. P is the correlation coefficient loss function, implemented based on the Pearson correlation coefficient. When the number of modalities in the input network is n=2, calculating the correlation coefficient loss function is equivalent to calculating the Pearson correlation coefficient between the features of two modes, which can be expressed as:

[0102]

[0103] Where: ρ 1,2 The feature F represents the relationship between mode 1 and mode 2. (1) ,F (2) The Pearson correlation coefficient, W and H are modal features F (1) ,F (2) The dimensions are width and height, F x,y This represents the feature value of feature F at the pixel point with coordinates (x, y). When the number of modalities in the input network n ≥ 2, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, which can be expressed as:

[0104]

[0105] Where: ρ i,j The Pearson correlation coefficient represents the relationship between the features of the i-th mode and the features of the j-th mode.

[0106] Secondly, it calculates the accuracy of the land cover classification prediction results output by each part of the network based on the cross-entropy loss function, including the classification prediction result loss of the final network output, the classification prediction result loss of the modality-specific encoder output, and the classification prediction result loss of the shared encoder output. The classification prediction result loss function can be expressed as:

[0107] L cls =L pred +L specific +L shared

[0108] Where: Lcls It is the loss of the classification prediction result, L pred It is the loss of the classification prediction result output by the network at the end, L specific It is the loss of the classification prediction result output by the modality-specific encoder, L shsred The loss function is the classification prediction result output by the modality-shared encoder. The cross-entropy loss function is:

[0109]

[0110] Where: M is the number of categories, N is the number of samples, and s ec p is a symbolic function, taking the value 0 or 1. ec Let e ​​be the predicted probability that the observed sample e belongs to category c.

[0111] Thirdly, the error between the classification prediction results output by the mode-specific encoder and the classification prediction results output by the shared encoder is calculated based on the mean squared error loss function, and used as the cross-modal loss to maintain stability and balance between the two encoders. The mean squared error loss function is:

[0112]

[0113] Where: d e and These are the actual observed value and the predicted value of the observed sample e, respectively.

[0114] Therefore, the final network loss function can be expressed as:

[0115] L = L cls +αL cor +L cro

[0116] Where: L cro This is the cross-modal loss, where α is a constant used to define the weights of the modal association constraint function, and is set to 2 in this embodiment.

[0117] To simulate possible scenarios of modal missing data in the real world, this embodiment sets the modal data missing rate to 70%, and trains the model on the multimodal remote sensing datasets Houston2013, Trento, and MUUFL, which consist of hyperspectral and lidar data, and Augsburg, which consists of hyperspectral, synthetic aperture radar, and digital surface models. The resulting land cover classification model is then used to classify land cover on the corresponding test set.

[0118] Based on the above steps, we obtained land cover classification results on the multimodal remote sensing datasets Houston2013, Trento, and MUUFL, which consist of hyperspectral and lidar data, and the Augsburg, a multimodal remote sensing dataset consisting of hyperspectral, synthetic aperture radar, and digital surface models. To compare with other methods, we used three traditional multimodal remote sensing image land cover classification methods: S... 2 ENet(Fang S,Li K,Li ZS 2ENet: Spatial-spectral cross-modal enhancement network for classification of hyperspectral and LiDAR data[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 1-5.), MFT(Roy S K, Deria A, Hong D, et al. Multimodal fusion transformer for remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-20.) and Cross-HL(Roy S K, Sukul A, Jamali A, et al. Cross hyperspectral and LiDAR attention transformer: An extended self-attention for land use and land cover classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024.), as well as three robust multi-modal fusion methods based on unified models: RFNet(Ding Y, Yu X, Yang Y. RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 3975-3984.), mmFormer(Zhang Y, He N, Yang J, et al.mmformer: Multimodal medical transformer for incomplete multimodal learning of braintumor segmentation [C] / / International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022:107-117.) and ShaSpec (Wang H, Chen Y, Ma C, et al. Multi-modal learning with missing modality via shared-specific feature modelling [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:15878-15887.) are compared with our method on the above datasets.

[0119] To quantitatively evaluate the 3D reconstruction results, we introduce Overall Accuracy (OA), Average Accuracy (AA), and the Kappa coefficient to assess the accuracy of land cover classification in remote sensing images. Higher numerical values ​​indicate higher classification accuracy. Quantitative comparison results on the Houston 2013 dataset are as follows:

[0120] Table 1. Quantitative analysis of different methods on the Houston 2013 dataset.

[0121]

[0122]

[0123] The quantitative comparison results on the Trento dataset are as follows:

[0124] Table 2 Quantitative analysis of different methods on the Trento dataset

[0125]

[0126] The quantitative comparison results on the MUUFL dataset are as follows:

[0127] Table 3 Quantitative analysis of different methods on the MUUFL dataset

[0128]

[0129]

[0130] The quantitative comparison results on the Augsburg dataset are as follows:

[0131] Table 4 Quantitative analysis of different methods on the Augsburg dataset

[0132]

[0133] Quantitative results show that, under simulated scenarios with missing multimodal data, the land cover classification results obtained by the method proposed in this invention are generally superior to existing methods on multimodal remote sensing datasets consisting of hyperspectral and lidar data (Houston2013, Trento, and MUUFL) and multimodal remote sensing datasets consisting of hyperspectral, synthetic aperture radar, and digital surface models (Augsburg). This method can achieve high-precision and robust land cover classification under scenarios with missing input modalities.

[0134] On the other hand, embodiments of the present invention also provide a robust land cover classification system based on multimodal relationship modeling and fusion network, characterized in that: it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a robust land cover classification method based on multimodal relationship modeling and fusion network as described above.

[0135] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0136] It should be understood that the above description of the embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art can make substitutions or modifications under the guidance of this invention without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A robust land cover classification method based on multimodal relationship modeling and fusion networks, characterized in that, Includes the following steps: Step 1: Acquire multimodal remote sensing data of the same target or scene, normalize the remote sensing data of each modality and then crop it to obtain data blocks of fixed size; Step 2: Construct a multimodal relationship modeling and fusion network for robust remote sensing image land cover classification. The multimodal relationship modeling and fusion network includes a dual-branch structure consisting of an initial encoder, a unique encoder, and a shared encoder, and a cross-modal fusion module, and takes the cropped data block as input. The initial encoder is used to extract features from the data blocks of the input network to obtain modal basis features; The dual-branch structure consisting of the unique encoder and the shared encoder comprises two parts: a unique encoder and a shared encoder. The unique encoder is used to further extract intra-modal unique features from the extracted modal basis features, and the shared encoder is used to further extract inter-modal shared features from the extracted modal basis features. After extracting the intra-modal unique features and inter-modal shared features for each modal data block in the input network, all intra-modal unique features and inter-modal shared features are summed to obtain the total intra-modal unique features and the total inter-modal shared features. The shared encoder is composed of several ConvNeXt convolutional modules and convolutional modules connected in an alternating manner; the ConvNeXt convolutional module includes The system consists of a convolutional layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, another fully connected layer, and a residual branch. The residual branch adds the modal basis features of the input shared encoder to the features output from the ConvNeXt convolutional module. The convolutional module includes a LayerNorm normalization layer and... Convolutional layer; The cross-modal fusion module is used to fuse features unique within the total modality and features shared between the total modalities to obtain fused features; In the cross-modal fusion module, both intra-modal specific features and inter-modal shared features are taken as input. The intra-modal specific features are first processed by a LayerNorm normalization layer, and the result is used as the query feature Q. The inter-modal shared features are also processed by a LayerNorm normalization layer, and the results are used as the key feature K and value feature V. Then, the three features Q, K, and V are input to the attention module, and the result is added to Q and then passed through a LayerNorm normalization layer. The convolutional layer, BatchNorm normalization layer, and ReLU activation function layer process the data to obtain fused features; the implementation process is represented as follows: in, It is a cross-modal fusion module. This is the fused feature output by the cross-modal fusion module; the attention module includes a dot product layer, a scaling layer, a masking layer, a SoftMax activation function layer, and a dot product layer; after performing a dot product operation on Q and K, the result is sequentially processed through the scaling layer, the masking layer, and the SoftMax activation function layer, and the result is then dot-producted with V before being output; the expression for the SoftMax activation function is: in: Input features for the Softmax activation function The total number of elements, and Features The first in The element and the first One element; Finally, the fused features are used to obtain the land cover classification results through a global average pooling layer and a fully connected layer; Step 3: In the case of modality missing scenarios, a multimodal relationship modeling and fusion network is trained by combining the loss function, and the trained network is used to achieve robust remote sensing image land cover classification.

2. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 1, characterized in that: The normalization process mentioned in step one refers to linear normalization, which is used to scale the data to... Within the interval, it is represented as: in, It is the first input to the network Raw remote sensing data for each modality, They are the first The minimum and maximum values ​​in the raw remote sensing data of each modality. It is the normalized first Remote sensing data for each modality.

3. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 1, characterized in that: The initial encoder is implemented by a residual convolution module, which includes... Convolutional layer, BatchNorm normalization layer, ReLU activation function layer Convolutional layer, BatchNorm normalization layer, ReLU activation function layer A convolutional layer, a BatchNorm normalization layer, a ReLU activation function layer, and a residual branch; the residual branch includes The convolutional layer, a BatchNorm normalization layer, takes the data block used to initialize the encoder as input to its branch. The output of the branch is added to the input features of the last ReLU activation function layer in the residual convolutional module. The ReLU activation function is: This represents the input features of the ReLU activation function.

4. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 3, characterized in that: In the encoder initialization, normalized and cropped data blocks are used as input, and after processing by the residual convolution module, modal basis features are obtained. The implementation process is represented as follows: in: It is the first Modal remote sensing data The data block after trimming. It is the first The initialization encoder for each mode, The result obtained after the calculation is the first one. Modal basis features of each mode.

5. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 3, characterized in that: The specific encoder is implemented by a residual convolution module. In the specific encoder, the modal basis features are taken as input and passed through the residual convolution module to obtain the in-modal specific features. The implementation process can be represented as follows: in: It is the first A unique encoder for each mode. It is the first Modal basis features of each mode, The result obtained after the calculation is the first one. Modal-specific features of each mode.

6. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 1, characterized in that: In the shared encoder, the modal basis features are taken as input and sequentially processed by k ConvNeXt convolutional modules and k-1 convolutional modules to obtain the shared features between modalities. The implementation process is represented as follows: in: It is the first A shared encoder for each modality. It is the first Modal basis features of each mode, The result obtained after the calculation is the first one. Modal shared features among modalities; After extracting the intra-modal specific features and inter-modal shared features from the data blocks of each modality in the input network, the intra-modal specific features and inter-modal shared features of all modalities are summed to obtain the total intra-modal specific features and total inter-modal shared features. The implementation process is represented as follows: in: It refers to the number of modalities in the input multimodal relationship modeling and fusion network. It is a feature unique to the overall mode. It is a shared feature among all modes.

7. The robust land cover classification method based on multimodal relationship modeling and fusion network as described in claim 1, characterized in that: The loss function used consists of three parts: the first is to guide the optimization of the feature extraction process through modal correlation constraints, which can be expressed as: in: It is the modal correlation constraint loss. It is a constant used to ensure that the modal correlation constraint loss always remains positive; It is a correlation coefficient loss function, implemented based on the Pearson correlation coefficient, which is used when the number of modes input to the network increases. When calculating the correlation coefficient loss function, it is equivalent to calculating the Pearson correlation coefficient of the features between the two modes, expressed as: in: Representing modes and modality feature The Pearson correlation coefficient between them and Modal features The dimensions include width and height. Representation of features In coordinates The feature values ​​of the pixels; the number of modal types input into the network. In this case, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, expressed as: in: Indicates the first The characteristics of the first mode and the first Pearson correlation coefficient between features of each modality; Secondly, it calculates the accuracy of the land cover classification prediction results output by each part of the network based on the cross-entropy loss function, including the classification prediction result loss of the final network output, the classification prediction result loss of the modality-specific encoder output, and the classification prediction result loss of the shared encoder output; the classification prediction result loss function is expressed as: in: It is the loss of classification prediction results. It is the loss of the classification prediction result finally output by the network. It is the loss of the classification prediction result output by the modality-specific encoder. The loss function for the classification prediction results output by the modality-shared encoder is: The cross-entropy loss function is: in: For the number of categories, For the number of samples, This is a sign function, taking the value 0 or 1; For observation samples Category The predicted probability; Thirdly, the error between the classification prediction results output by the mode-specific encoder and the classification prediction results output by the shared encoder is calculated based on the mean squared error loss function, and used as the cross-modal loss to maintain stability and balance between the two encoders; the mean squared error loss function is: in: and These are the observed samples Actual observed values ​​and predicted values; Therefore, the final network loss function is expressed as: in: It is cross-modal loss. It is a constant used to define the weights of the modal correlation constraint function.

8. A robust land cover classification system based on multimodal relationship modeling and fusion networks, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a robust land cover classification method based on multimodal relationship modeling and fusion network as described in any one of claims 1-7.

Images