Robustness ground feature classification method and system based on multi-modal relation modeling and fusion network

Abstract

The invention provides a robustness ground feature classification method and system based on multi-modal relation modeling and a fusion network. Firstly, by analyzing potential relations between modals and in the modals, assumption that high-frequency information corresponds to specific features in the modals and low-frequency information corresponds to shared features between the modals is provided, modeling is carried out on the relations between multiple modals, effective feature extraction is achieved, and a foundation is laid for subsequent feature fusion. Meanwhile, based on the criterion that specific features in modals are not related and shared features between modals are related, correlation constraint is carried out on the extracted specific features and shared features, so that the feature fusion process is further optimized, and the feature extraction process is guided and optimized. On the basis of mutual promotion of feature extraction and feature fusion processes, effective extraction of multi-modal features is realized, so that high-precision and robust multi-modal fusion learning in an input modal missing scene is completed. And finally, experiments prove that the overall ground feature classification precision of the method is superior to the result of the existing method.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning technology, and relates to a robust object classification method based on multimodal relationship modeling and fusion network, which is suitable for robust object classification application scenarios in modality missing scenarios. Background Art

[0002] Remote sensing image object classification is widely used in earth observation tasks such as environmental monitoring and object analysis, and is the basis for subsequent recognition and segmentation tasks based on remote sensing images. Therefore, how to achieve high-precision and robust remote sensing image object classification is very critical. The research on remote sensing image object classification methods based on single modality data has been carried out for a long time and has achieved good results. However, the classification accuracy of such methods is limited by the effective information that the input modality data can provide. For example, hyperspectral (HS) and multispectral (MS) data can reflect the material information of the detection target, but it is difficult to distinguish different objects of the same material; light detection and ranging (LiDAR) and digital surface model (DSM) contain the altitude information of the objects, but it is difficult to distinguish objects with similar spatial structures; synthetic aperture radar (SAR) can provide information on the surface structure of the earth, but the image interpretation is difficult. In recent years, the object classification method based on multimodal remote sensing data fusion has played a significant role in the field of computer vision. By utilizing the complementary advantages of different modal remote sensing data, more efficient remote sensing image interpretation and high-precision remote sensing image object classification can be achieved. However, in real scenarios, the integrity and effectiveness of multimodal remote sensing data cannot be guaranteed. Incomplete modal data input greatly reduces the effectiveness of such methods and makes them difficult to put into practical application. Therefore, it is worth further exploring how to achieve high-precision and robust object classification methods based on multimodal remote sensing data fusion when the input modal data may be missing.

[0003] In order to achieve high-precision and robust multimodal fusion learning in the scenario of missing input modality, existing research methods can be divided into three categories: methods based on generative learning, methods based on distillation learning, and methods based on unified models. The generative learning-based method generates data with the same distribution as the missing modal data based on the generative network, and uses the traditional multimodal fusion learning algorithm to achieve classification. For example, Kumar et al. proposed a general network architecture that can approximate the missing modal data from the available modal data in the classification decision stage (Kumar S, Banerjee B, Chaudhuri S. Improved landcover classification using online spectral data hallucination [J]. Neurocomputing, 2021, 439: 316-326). However, the quality of the generated data will directly affect the accuracy of the classification. In addition, the unknown noise generated in the process of generating data will also affect the stability of the model. The key to the distillation learning-based method is to learn the missing modal feature-level information from the complete modal data, which requires the model to have complete modal data input during the training stage. For example, Wei et al. designed a new joint adaptive distillation method to overcome the heterogeneity of multimodal features (Wei S, Luo Y, Ma X, et al. MSH-Net: Modality-shared hallucination with joint adaptation distillation for remote sensing image classification using missing modalities [J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-15). Such methods often assume that the modality type of the missing data is known, and a specific model needs to be trained for each missing modality. Recently, the unified model-based method has achieved good results in the field of medical image segmentation. It learns joint feature representation based on the relationship between different modal data and randomly deletes modal data in the training and inference stages to simulate the modality missing situation in practical applications. However, when dealing with remote sensing image object classification tasks, the unified model-based method cannot meet the high-precision classification requirements, mainly because: (1) A common practice of the unified model-based method is to use an independent encoder for each modality data to extract the unique features of the modality, and then fuse the specific features from different modal data for the next classification. This type of method models the relationship between modalities based on the features extracted by different encoders, and often focuses more on the differences between modalities.For remote sensing image object classification tasks based on multimodal data, overemphasizing the differences between different modalities is not conducive to effectively integrating the advantages of multimodality. Therefore, the relationship between different modalities needs to be modeled before feature extraction. (2) An effective way to model the relationship between multimodal features is to divide multimodal features into shared features between modalities and unique features within modalities. For example, Wang et al. designed a ShaSpec network framework to learn multimodal shared and unique features based on the auxiliary tasks of distribution alignment and domain classification (Wang H, Chen Y, Ma C, et al. Multi-modal learning with missing modality via shared-specific feature modelling[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2023:15878-15887). The common strategy for extracting shared features between modalities and unique features within a modality is to use an independent encoder for each modality to extract the unique features of the modality; and to use a shared encoder for all modal data to capture the unified features between different modalities as shared features. This strategy does not deeply analyze the potential relationships between modalities and within modalities, and it is difficult to accurately extract shared and unique features of multimodal data. It has limited performance in remote sensing image tasks with large differences in target size and spectral characteristics. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a robust land feature classification method based on multimodal relationship modeling and fusion network. First, by analyzing the potential relationships between and within modalities, the hypothesis that high-frequency information corresponds to unique features within the modality and low-frequency information corresponds to shared features between modalities is proposed, and the relationship between multiple modalities is modeled to achieve effective feature extraction, laying the foundation for subsequent feature fusion. At the same time, based on the principle that unique features within the modality are unrelated, while shared features between modalities are related, the extracted unique features and shared features are associated and constrained to further optimize the feature fusion process and guide the optimization of the feature extraction process. Based on the mutual promotion of feature extraction and feature fusion processes, effective extraction of multimodal features is achieved to complete high-precision and robust multimodal fusion learning in scenarios where the input modality is missing.

[0005] The technical solution adopted by the present invention is: a robust object classification method based on multimodal relationship modeling and fusion network. First, the multimodal remote sensing data is normalized, and for each modal data, a data block with a size of 7×7 centered on one pixel is cropped as the input of the multimodal relationship modeling and fusion network. For the modal data block input to the network, an initialization encoder is constructed to extract modal base features. Then, a dual-branch structure consisting of a unique encoder and a shared encoder is constructed to extract unique features within the modality and shared features between modalities, respectively. A cross-modal fusion module is constructed to realize the fusion of unique features within the modality and shared features between modalities. Then, based on the dual-branch structure consisting of the above-mentioned initialization encoder, unique encoder and shared encoder and the cross-modal fusion module, a multimodal relationship modeling and fusion network for robust remote sensing image object classification is constructed, and a loss function suitable for the network is designed. Finally, based on the different modality missing conditions, training is performed on the multimodal remote sensing datasets Houston2013, Trento and MUUFL composed of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg composed of hyperspectral, synthetic aperture radar and digital surface model, and remote sensing image ground object classification is performed based on the obtained robust ground object classification model. The method includes the following steps:

[0006] Step 1: obtain multi-modal remote sensing data of the same detection target or scene, normalize and crop the remote sensing data of each modality to obtain a data block of a fixed size;

[0007] Step 2: construct a multimodal relationship modeling and fusion network for robust remote sensing image object classification, wherein the multimodal relationship modeling and fusion network includes a dual-branch structure consisting of an initialization encoder, a unique encoder, and a shared encoder, and a cross-modal fusion module, and takes the cropped data block as input;

[0008] The initialization encoder is used to extract features from the data blocks of the input network to obtain modal base features;

[0009] The dual-branch structure composed of the unique encoder and the shared encoder includes two parts: a unique encoder and a shared encoder; the unique encoder is used to further extract the intra-modal unique features from the extracted modal base features, and the shared encoder is used to further extract the inter-modal shared features from the extracted modal base features; after extracting the intra-modal unique features and inter-modal shared features of the data block of each modality of the input network, all the intra-modal unique features and inter-modal shared features are added and summed respectively to obtain the total intra-modal unique features and total inter-modal shared features;

[0010] The cross-modal fusion module is used to achieve the fusion of the unique features within the total modality and the shared features between the total modalities to obtain fused features;

[0011] Finally, the fusion features are passed through the global average pooling layer and the fully connected layer to obtain the object classification result;

[0012] Step three, combine the loss function to train the multimodal relationship modeling and fusion network in the modality missing scenario, and use the trained network to achieve robust remote sensing image object classification.

[0013] Furthermore, the normalization process described in step 1 refers to linear normalization, which is used to scale the data to the interval [0,1], expressed as:

[0014]

[0015] in, is the original remote sensing data of the ith modality input into the network, are the minimum and maximum values ​​in the original remote sensing data of the i-th mode, M (i) is the normalized remote sensing data of the ith mode.

[0016] Furthermore, the initialization encoder is implemented by a residual convolution module, which includes a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 3×3 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer and a residual branch; the residual branch includes a 1×1 convolution layer, a BatchNorm normalization layer, the input of the branch is a data block input to the initialization encoder, and the output of the branch is added to the input feature of the last ReLU activation function layer in the residual convolution module, wherein the ReLU activation function is:

[0017]

[0018] x represents the input feature of the ReLU activation function.

[0019] Furthermore, in the initialization encoder, the normalized and trimmed data block is used as input, and the modal basis features are obtained after the residual convolution module operation. The implementation process is expressed as:

[0020]

[0021] in: is the remote sensing data M of the i-th modality (i) After trimming the data block, is the initialization encoder of the ith modality, is the modal basis feature of the i-th mode obtained after the operation.

[0022] Furthermore, the unique encoder is implemented by a residual convolution module. In the unique encoder, the modal base features are taken as input, and the modal intra-specific features are obtained through the residual convolution module. The implementation process can be expressed as:

[0023]

[0024] in: is the unique encoder of the ith mode, is the modal basis feature of the ith mode, is the intra-modal characteristic of the i-th mode obtained after the operation.

[0025] Furthermore, the shared encoder is composed of several ConvNeXt convolution modules and convolution modules that are connected in sequence in an interlaced manner; the ConvNeXt convolution module includes a 7×7 convolution layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, a fully connected layer and a residual branch; the residual branch adds the modal basis features of the input shared encoder to the features output by the ConvNeXt convolution module; the convolution module includes a LayerNorm normalization layer and a 3×3 convolution layer.

[0026] Furthermore, in the shared encoder, the modal base features are taken as input, and after being operated by k ConvNeXt convolutional modules and k-1 convolutional modules in sequence, the shared features between the modalities are obtained. The implementation process is expressed as:

[0027]

[0028] in: is the shared encoder of the ith modality, is the modal basis feature of the ith mode, is the inter-modal shared feature of the i-th mode obtained after the operation;

[0029] After extracting the unique features within the modality and the shared features between modalities from the data blocks of each modality of the input network, all the unique features within the modality and the shared features between modalities are added and summed up to obtain the total unique features within the modality and the total shared features between modalities. The implementation process is expressed as:

[0030]

[0031] Where: n is the number of modal types input into the multimodal relationship modeling and fusion network, F specific is a unique feature within the total mode, F shared is the total shared feature among the modalities.

[0032] Furthermore, in the cross-modal fusion module, the unique features within the total modality and the shared features between the total modalities are used as input, wherein the unique features within the total modality are first processed by the LayerNorm normalization layer, and the results are used as query features Q, and the shared features between the total modalities are processed by the LayerNorm normalization layer, and the results are used as key features K and value features V; then, the three features Q, K and V are input into the attention module, and the results are added to Q and processed in turn by the LayerNorm normalization layer, 3×3 convolution layer, BatchNorm normalization layer and ReLU activation function layer to obtain fusion features; the implementation process is expressed as:

[0033] F fused =C(F specific ,F shared )

[0034] Among them, C is the cross-modal fusion module, F fused is the fusion feature output by the cross-modal fusion module; the attention module includes a dot product operation layer, a scale transformation layer, a mask layer, a SoftMax activation function layer and a dot product operation layer; after performing a dot product operation on Q and K, the result is sequentially operated through the scale transformation layer, the mask layer and the SoftMax activation function layer, and the result is output after performing a dot product operation with V; the expression of the Softmax activation function is:

[0035]

[0036] Where: C is the total number of elements of the input feature z of the Softmax activation function, z a and z b are the a-th element and b-th element in feature z respectively.

[0037] Furthermore, the adopted loss function includes three parts: the first is to guide the optimization of the feature extraction process through the modal association constraint, and the modal association constraint function can be expressed as:

[0038]

[0039] Where: L cor is the modal association constraint loss, ∈ is a constant, which is used to ensure that the modal association constraint loss always maintains a positive value; P is the correlation coefficient loss function, which is implemented based on the Pearson correlation coefficient. When the number of modal types of the input network n = 2, calculating the correlation coefficient loss function is equivalent to calculating the Pearson correlation coefficient of the features between the two modalities, which is expressed as:

[0040]

[0041] Where: 1,2Represents the feature f between mode 1 and mode 2 (1) ,F (2) Pearson correlation coefficient, W and H are modal features F (1) ,F (2) The width and height of the dimensions, F x,y Represents the feature value of the pixel point of feature F at coordinate (x, y); when the number of modal types of the input network n ≥ 2, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, expressed as:

[0042]

[0043] Where: i,j Represents the Pearson correlation coefficient between the features of the i-th modality and the features of the j-th modality.

[0044] The second is to calculate the accuracy of the classification prediction results of the objects output by each part of the network based on the cross entropy loss function, including the classification prediction result loss of the final output of the network, the classification prediction result loss of the modality-specific encoder output and the classification prediction result loss of the shared encoder output; the classification prediction result loss function is expressed as:

[0045] L cls =L pred +L specific +L shared

[0046] Where: L cls is the classification prediction result loss, L pred is the classification prediction loss of the final output of the network, L specific is the classification prediction loss of the modality-specific encoder output, L shared The classification prediction result loss of the modality sharing encoder output; the cross entropy loss function is:

[0047]

[0048] Where: M is the number of categories, N is the number of samples, s ec is a sign function, taking values ​​of 0 or 1; p ec is the predicted probability that the observed sample e belongs to category c;

[0049] The third is to calculate the error between the classification prediction results output by the modality-specific encoder and the classification prediction results output by the shared encoder based on the mean square error loss function as a cross-modal loss to maintain stability and balance between the two encoders; the mean square error loss function is:

[0050]

[0051] Where: de and are the actual observed value and predicted value of the observed sample e respectively;

[0052] Therefore, the final network loss function is expressed as:

[0053] L=L cls +αL cor +L cro

[0054] Where: L cro is the cross-modal loss, and α is a constant that defines the weight of the modal association constraint function.

[0055] The present invention also provides a robust land feature classification system based on multimodal relationship modeling and fusion network, characterized in that it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a robust land feature classification method based on multimodal relationship modeling and fusion network as described in the above scheme.

[0056] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: The present invention proposes a robust land feature classification method based on multimodal relationship modeling and fusion network. First, by analyzing the potential relationships between and within modalities, the hypothesis that high-frequency information corresponds to unique features within the modality and low-frequency information corresponds to shared features between modalities is proposed, and the relationship between multiple modalities is modeled to achieve effective feature extraction, laying the foundation for subsequent feature fusion. At the same time, based on the principle that unique features within the modality are unrelated, while shared features between modalities are related, the extracted unique features and shared features are associated and constrained to further optimize the feature fusion process and guide the optimization of the feature extraction process. Based on the mutual promotion of feature extraction and feature fusion processes, effective extraction of multimodal features is achieved to complete high-precision and robust multimodal fusion learning in scenarios where the input modality is missing. By setting the modal data missing rate and controlling the multimodal data integrity of the input network at a certain ratio, the modal missing scenarios that may occur in the real world are simulated. On the multimodal remote sensing datasets Houston2013, Trento and MUUFL composed of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg composed of hyperspectral, synthetic aperture radar and digital surface model, the overall land object classification accuracy of the present invention is better than the results of existing methods, and high-precision and robust land object classification can be achieved in a variety of input modal missing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is the structure diagram of the residual convolution module.

[0058] Figure 2 This is a diagram of the shared encoder structure.

[0059] Figure 3 This is the structure diagram of the cross-modal fusion module.

[0060] Figure 4 It is a multimodal relationship modeling and fusion network structure diagram. DETAILED DESCRIPTION

[0061] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] The present invention is mainly aimed at the application needs of remote sensing image object classification with robustness requirements. We propose a robust object classification method based on multimodal relationship modeling and fusion network. First, by analyzing the potential relationship between modalities and within modalities, the assumption that high-frequency information corresponds to unique features within the modality and low-frequency information corresponds to shared features between modalities is proposed, and the relationship between multimodalities is modeled to achieve effective feature extraction and lay the foundation for subsequent feature fusion. At the same time, based on the principle that unique features within the modality are unrelated, while shared features between modalities are related, the extracted unique features and shared features are associated and constrained to further optimize the feature fusion process and guide the optimization of the feature extraction process. Based on the mutual promotion of feature extraction and feature fusion processes, the effective extraction of multimodal features is achieved to complete high-precision and robust multimodal fusion learning in scenarios where the input modality is missing.

[0063] This embodiment provides a robust object classification method based on multimodal relationship modeling and fusion network to achieve robust remote sensing image object classification, which specifically includes the following steps:

[0064] Step 1: Obtain multi-modal remote sensing data of the same detection target or scene, and normalize the remote sensing data of each modality. Normalization refers to linear normalization, which is used to scale the data to the [0,1] interval and can be expressed as:

[0065]

[0066] in, is the original remote sensing data of the ith modality input into the network, are the minimum and maximum values ​​of the original remote sensing data of the i-th mode, M( i ) is the normalized remote sensing data of the ith mode. Then, the normalized remote sensing data is cropped into a data block with a size of 7×7 and a pixel as the center.

[0067] Step 2: Construct the initialization encoder, which is implemented by the residual convolution module. The structure is as shown in the attached figure. Figure 1As shown. The residual convolution module includes a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 3×3 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer and a residual branch. The residual branch includes a 1×1 convolution layer, a BatchNorm normalization layer, the input of the branch is the data block of the input initialization encoder, and the output of the branch is added to the input feature of the last ReLU activation function layer in the residual convolution module. Among them, the ReLU activation function is:

[0068]

[0069] x represents the input feature of the ReLU activation function.

[0070] Step 2.1, in the initialization encoder, the normalized and cropped 7×7 data block is used as input, and the modal basis features are obtained through residual convolution module operation. The implementation process can be expressed as:

[0071]

[0072] in: is the remote sensing data M of the i-th modality (i) After trimming, the size of the data block is 7×7. is the initialization encoder of the ith modality, is the modal basis feature of the i-th mode obtained after the operation.

[0073] Step 3: Analyze the potential relationship between and within modalities. Due to differences in imaging principles and detection technologies, the edge and detail information reflected by the data of the same detection target represented by different modalities may be different, but the overall image and contour should be similar. In digital image processing, the low-frequency features of the image can reflect the general contour, while the high-frequency features of the image can represent the edge and detail information. Therefore, for the same detection target using different detection technologies, the low-frequency features should have commonalities, reflecting the shared information between multiple modalities, and the high-frequency features should be able to represent the unique features within a certain modality.

[0074] Step 4: Construct a unique encoder, which is implemented by a residual convolution module. The structure is shown in the attached figure. Figure 1 As shown in Figure 2, by taking advantage of the fact that convolutional computation can effectively extract features in the local receptive field, the unique encoder can further extract high-frequency information from the extracted modal base features as the unique features within the modality.

[0075] Step 4.1, in the unique encoder, the modal base features are taken as input and passed through the residual convolution module to obtain the unique features within the modality. The implementation process can be expressed as:

[0076]

[0077] in: is the unique encoder of the ith mode, is the intra-modal characteristic of the i-th mode obtained after the operation.

[0078] Step 5: Build a shared encoder, as shown in the following figure. Figure 2 As shown in the figure. The shared encoder consists of 4 ConvNeXt convolution modules and 3 convolution modules connected in sequence. Among them, the ConvNeXt convolution module can capture more spatial information, enhance the feature extraction ability, and effectively grasp the long-distance dependency by expanding the receptive field and using a larger convolution kernel (from 3×3 to 7×7), thereby extracting low-frequency features. The shared encoder uses convolution layers of different scales and depths to capture different features in the image, which can better adapt to complex image scenes and can further extract low-frequency information from the extracted modal base features as inter-modal shared features. The ConvNeXt convolution module includes a 7×7 convolution layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, a fully connected layer and a residual branch. The residual branch adds the modal base features input to the shared encoder with the features output by the ConvNeXt convolution module. The convolution module includes a LayerNorm normalization layer and a 3×3 convolution layer.

[0079] Step 5.1: In the shared encoder, the modal base features are taken as input, and after being operated by 4 ConvNeXt convolution modules and 3 convolution modules in sequence, the shared features between modalities are obtained. The implementation process can be expressed as:

[0080]

[0081] in: is the shared encoder of the ith modality, is the inter-modal shared feature of the i-th mode obtained after the operation.

[0082] Step 5.2: After extracting the unique features of each modality and the shared features between modalities from the data blocks of each modality of the input network, all the unique features of each modality and the shared features between modalities are added and summed up to obtain the total unique features of each modality and the total shared features between modalities. The implementation process can be expressed as:

[0083]

[0084] Where: n is the number of modal types input into the multimodal relationship modeling and fusion network, F specific is a unique feature within the total mode, F sharedis the total shared feature among the modalities.

[0085] Step 6: Construct a cross-modal fusion module, whose structure is shown in the attached figure. Figure 3 As shown in the figure. Based on the cross-attention mechanism, the cross-modal fusion module can capture the correlation between the input multi-modal features. At the same time, the unique information within the modality can be used to guide the feature expression of shared information between modalities. In this way, the differences and correlations between multi-modal features are utilized to achieve efficient multi-modal feature fusion.

[0086] Step 6.1, in the cross-modal fusion module, the unique features within the total modality and the shared features between the total modalities are used as input. Among them, the unique features within the total modality are first processed by the LayerNorm normalization layer, and the results are used as query features (Query, Q). The shared features between the total modalities are processed by the LayerNorm normalization layer, and the results are used as key features (Key, K) and value features (Value, V). Then, the three features Q, K and V are input into the attention module, and the result is added to Q and processed in turn by the LayerNorm normalization layer, 3×3 convolution layer, BatchNorm normalization layer and ReLU activation function layer to obtain the fused features. The implementation process can be expressed as:

[0087] F fused =C(F specific ,F shared )

[0088] Among them, C is the cross-modal fusion module, F fused is the fusion feature output by the cross-modal fusion module. The attention module includes a dot product operation layer, a scale transformation layer, a mask layer, a SoftMax activation function layer and a dot product operation layer. After performing a dot product operation on Q and K, the result is sequentially operated through the scale transformation layer, the mask layer and the SoftMax activation function layer, and the result is dot-producted with V and then output. The expression of the Softmax activation function is:

[0089]

[0090] Where: C is the total number of elements of the input feature z of the Softmax activation function, z a and z b are the a-th element and b-th element in feature z respectively.

[0091] Step 7: Construct a multimodal relationship modeling and fusion network for robust remote sensing image object classification. Figure 4As shown. Using the initialization encoder, unique encoder and shared encoder described in step 2, step 4 and step 5, adding the cross-modal fusion module described in step 6, and passing the fusion features through the global average pooling layer and the fully connected layer to obtain the object classification result, the implementation process can be expressed as:

[0092] Pred=FC(Pool(F fused ))

[0093] Among them, FC and Pool are the fully connected layer and global average pooling layer operations respectively, and Pred is the obtained object classification result.

[0094] Step 7.1: normalize and trim the multimodal 7×7 data blocks, i.e. Figure 4 middle Input the initialization encoder of the mode respectively to obtain the basic features of each mode, namely, Figure 4 middle

[0095] Step 7.2, for each input modality, the modal base features obtained by its initialization encoder are sent to the unique encoder and shared encoder described in steps 4 and 5 to obtain the modal unique features and inter-modal shared features of each modality, namely, Figure 4 middle and

[0096] Step 7.3, add up all the unique features within the modality and the shared features between the modalities to obtain the total unique features within the modality and the total shared features between the modalities, that is, Figure 4 F specific ,F shared ;

[0097] Step 7.4, the obtained unique features within the total modality and the shared features between the total modalities are sent to the cross-modal fusion module described in step 6, and the differences and correlations between the multi-modalities are used to achieve efficient multi-modal feature fusion to obtain the fused features, namely, Figure 4 F fused ;

[0098] Step 7.5, the fusion features are passed through the global average pooling layer and the fully connected layer to obtain the ground object classification result in this embodiment, that is, Figure 4 Pred in.

[0099] Step 8: Design a loss function suitable for the network. The loss function used consists of three parts: The first is to guide the optimization of the feature extraction process through modal association constraints: based on the principle that the unique features within the modality are unrelated, while the shared features between the modalities are related, the extracted unique features and shared features are associated with constraints to further optimize the feature fusion process, guide the optimization of the feature extraction process, and achieve mutual enhancement of feature extraction and feature fusion. The modal association constraint function can be expressed as:

[0100]

[0101] Where: L cor is the modal association constraint loss, ∈ is a constant with a value of 1.01, which is used to ensure that the modal association constraint loss always maintains a positive value. P is the correlation coefficient loss function, which is implemented based on the Pearson correlation coefficient. When the number of modal types of the input network n = 2, calculating the correlation coefficient loss function is equivalent to calculating the Pearson correlation coefficient of the features between the two modalities, which can be expressed as:

[0102]

[0103] Where: 1,2 Represents the feature F between mode 1 and mode 2 (1) ,F (2) Pearson correlation coefficient, W and H are modal features F (1) ,F (2) The width and height of the dimensions, F x,y Represents the feature value of the pixel point of feature F at coordinate (x, y). When the number of modal types of the input network n≥2, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, which can be expressed as:

[0104]

[0105] Where: i,j Represents the Pearson correlation coefficient between the features of the i-th modality and the features of the j-th modality.

[0106] The second is to calculate the accuracy of the classification prediction results of the objects output by each part of the network based on the cross entropy loss function, including the classification prediction result loss of the final output of the network, the classification prediction result loss of the modality-specific encoder output, and the classification prediction result loss of the shared encoder output. The classification prediction result loss function can be expressed as:

[0107] L cls =L pred +L specific +L shared

[0108] Where: Lcls is the classification prediction result loss, L pred is the classification prediction loss of the final output of the network, L specific is the classification prediction loss of the modality-specific encoder output, L shsred The classification prediction result loss of the modality sharing encoder output. The cross entropy loss function is:

[0109]

[0110] Where: M is the number of categories, N is the number of samples, s ec is a sign function, taking values ​​of 0 or 1; p ec is the predicted probability that the observed sample e belongs to category c.

[0111] The third is to calculate the error between the classification prediction results output by the modality-specific encoder and the classification prediction results output by the shared encoder based on the mean square error loss function as a cross-modal loss to maintain stability and balance between the two encoders. The mean square error loss function is:

[0112]

[0113] Where: d e and are the actual observed value and predicted value of the observed sample e respectively.

[0114] Therefore, the final network loss function can be expressed as:

[0115] L=L cls +αL cor +L cro

[0116] Where: L cro is the cross-modal loss, and α is a constant used to define the weight of the modal association constraint function, which is set to 2 in this embodiment.

[0117] In order to simulate the possible scenario of modal missing in the real world, the modal data missing rate is set to 70% in this embodiment, and training is performed on the multimodal remote sensing datasets Houston2013, Trento and MUUFL composed of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg composed of hyperspectral, synthetic aperture radar and digital surface model, respectively. The object classification model obtained by training is used to perform object classification on the corresponding test set.

[0118] Based on the above steps, we obtain the object classification results on the multimodal remote sensing datasets Houston2013, Trento and MUUFL composed of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg composed of hyperspectral, synthetic aperture radar and digital surface model. In order to compare with other methods, we use three traditional multimodal remote sensing image object classification methods: S 2 ENet(Fang S,Li K,Li ZS 2ENet: Spatial-spectral cross-modal enhancement network for classification of hyperspectral and LiDAR data[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 1-5.), MFT(Roy S K, Deria A, Hong D, et al. Multimodal fusion transformer for remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-20.) and Cross-HL(Roy S K, Sukul A, Jamali A, et al. Cross hyperspectral and LiDAR attention transformer: An extended self-attention for land use and land cover classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024.), as well as three robust multi-modal fusion methods based on a unified model: RFNet(Ding Y, Yu X, Yang Y. RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 3975-3984.), mmFormer(Zhang Y, He N, Yang J, et al.mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation [C] / / International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 107-117.) and ShaSpec (Wang H, Chen Y, Ma C, et al. Multi-modal learning with missing modality via shared-specific feature modelling [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 15878-15887.), and compared with our method on the above datasets. .

[0119] In order to quantitatively evaluate the 3D reconstruction results, we introduce the overall accuracy (OA), average accuracy (AA) and Kappa coefficient to evaluate the accuracy of remote sensing image classification. The larger the index value, the higher the classification accuracy. The quantitative comparison results on the Houston2013 dataset are as follows:

[0120] Table 1 Quantitative analysis of different methods on the Houston2013 dataset

[0121]

[0122]

[0123] The quantitative comparison results on the Trento dataset are as follows:

[0124] Table 2 Quantitative analysis of different methods on the Trento dataset

[0125]

[0126] The quantitative comparison results on the MUUFL dataset are as follows:

[0127] Table 3 Quantitative analysis of different methods on the MUUFL dataset

[0128]

[0129]

[0130] The quantitative comparison results on the Augsburg dataset are as follows:

[0131] Table 4 Quantitative analysis of different methods on the Augsburg dataset

[0132]

[0133] The quantitative indicator results show that in the simulated scenario of missing data in multiple modalities, the object classification results obtained by the method proposed in the present invention are generally better than the existing methods on the multimodal remote sensing datasets Houston2013, Trento and MUUFL composed of hyperspectral and lidar data, and the multimodal remote sensing dataset Augsburg composed of hyperspectral, synthetic aperture radar and digital surface model, and can achieve high-precision and robust object classification in scenarios with missing input modalities.

[0134] On the other hand, an embodiment of the present invention also provides a robust land feature classification system based on multimodal relationship modeling and fusion network, characterized in that it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a robust land feature classification method based on multimodal relationship modeling and fusion network as described in the above scheme.

[0135] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0136] It should be understood that the above description of the embodiments is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A robust feature classification method based on multimodal relationship modeling and fusion network, characterized in that: The steps include: Step 1: obtain multi-modal remote sensing data of the same detection target or scene, normalize and crop the remote sensing data of each modality to obtain a data block of a fixed size; Step 2: construct a multimodal relationship modeling and fusion network for robust remote sensing image object classification, wherein the multimodal relationship modeling and fusion network includes a dual-branch structure consisting of an initialization encoder, a unique encoder, and a shared encoder, and a cross-modal fusion module, and takes the cropped data block as input; The initialization encoder is used to extract features from the data blocks of the input network to obtain modal base features; The dual-branch structure composed of the unique encoder and the shared encoder includes two parts: a unique encoder and a shared encoder; the unique encoder is used to further extract the intra-modal unique features from the extracted modal base features, and the shared encoder is used to further extract the inter-modal shared features from the extracted modal base features; after extracting the intra-modal unique features and inter-modal shared features of the data block of each modality of the input network, all the intra-modal unique features and inter-modal shared features are added and summed respectively to obtain the total intra-modal unique features and total inter-modal shared features; The cross-modal fusion module is used to achieve the fusion of the unique features within the total modality and the shared features between the total modalities to obtain fused features; Finally, the fusion features are passed through the global average pooling layer and the fully connected layer to obtain the object classification result; Step three, combine the loss function to train the multimodal relationship modeling and fusion network in the modality missing scenario, and use the trained network to achieve robust remote sensing image object classification.

2. A robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 1, characterized in that: The normalization process described in step 1 refers to linear normalization, which is used to scale the data to the interval [0,1], expressed as: in, is the original remote sensing data of the ith modality input into the network, are the minimum and maximum values ​​in the original remote sensing data of the i-th mode, M (i) is the normalized remote sensing data of the ith mode.

3. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 1, characterized in that: The initialization encoder is implemented by a residual convolution module, which includes a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 3×3 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer, a 1×1 convolution layer, a BatchNorm normalization layer, a ReLU activation function layer and a residual branch; the residual branch includes a 1×1 convolution layer, a BatchNorm normalization layer, the input of the branch is the data block input to the initialization encoder, and the output of the branch is added to the input feature of the last ReLU activation function layer in the residual convolution module, wherein the ReLU activation function is: x represents the input feature of the ReLU activation function.

4. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 3, characterized in that: In the initialization encoder, the normalized and trimmed data block is taken as input, and the modal basis features are obtained after the residual convolution module operation. The implementation process is expressed as: in: is the remote sensing data M of the i-th modality (i) After trimming the data block, is the initialization encoder of the ith modality, is the modal basis feature of the i-th mode obtained after the operation.

5. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 3, characterized in that: The unique encoder is implemented by a residual convolution module. In the unique encoder, the modal base features are taken as input and passed through the residual convolution module to obtain the unique features within the modality. The implementation process can be expressed as: in: is the unique encoder of the ith mode, is the modal basis feature of the ith mode, is the intra-modal characteristic of the i-th mode obtained after the operation.

6. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 1, characterized in that: The shared encoder is composed of several ConvNeXt convolution modules and convolution modules that are connected in sequence in an interlaced manner; the ConvNeXt convolution module includes a 7×7 convolution layer, a LayerNorm normalization layer, a fully connected layer, a GELU activation function layer, a fully connected layer and a residual branch; the residual branch adds the modal basis features of the input shared encoder to the features output by the ConvNeXt convolution module; the convolution module includes a LayerNorm normalization layer and a 3×3 convolution layer.

7. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 6, characterized in that: In the shared encoder, the modal base features are taken as input, and after being operated by k ConvNeXt convolution modules and k-1 convolution modules in sequence, the shared features between the modalities are obtained. The implementation process is expressed as: in: is the shared encoder of the ith modality, is the modal basis feature of the ith mode, is the inter-modal shared feature of the i-th mode obtained after the operation; After extracting the unique features within the modality and the shared features between modalities from the data blocks of each modality of the input network, all the unique features within the modality and the shared features between modalities are added and summed up to obtain the total unique features within the modality and the total shared features between modalities. The implementation process is expressed as: Where: n is the number of modal types input into the multimodal relationship modeling and fusion network, F specific is a unique feature within the total mode, F shared is the total shared feature among the modalities.

8. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 1, characterized in that: In the cross-modal fusion module, the unique features within the total modality and the shared features between the total modalities are taken as input, where the unique features within the total modality are first processed by the LayerNorm normalization layer, and the results are used as query features Q, and the shared features between the total modalities are processed by the LayerNorm normalization layer, and the results are used as key features K and value features V; then, the three features Q, K and V are input into the attention module, and the result is added to Q and processed by the LayerNorm normalization layer, 3×3 convolution layer, BatchNorm normalization layer and ReLU activation function layer in turn to obtain the fused features; the implementation process is expressed as: F fused =C(F specific ,F shared ) Among them, C is the cross-modal fusion module, F fused is the fusion feature output by the cross-modal fusion module; the attention module includes a dot product operation layer, a scale transformation layer, a mask layer, a SoftMax activation function layer and a dot product operation layer; after performing a dot product operation on Q and K, the result is sequentially operated through the scale transformation layer, the mask layer and the SoftMax activation function layer, and the result is output after performing a dot product operation with V; the expression of the Softmax activation function is: Where: C is the total number of elements of the input feature z of the Softmax activation function, z a and z b are the a-th element and b-th element in feature z respectively.

9. The robust feature classification method based on multimodal relationship modeling and fusion network as claimed in claim 1, characterized in that: The loss function used consists of three parts: the first is to guide the optimization of the feature extraction process through modal association constraints. The modal association constraint function can be expressed as: Where: L cor is the modal association constraint loss, ∈ is a constant, which is used to ensure that the modal association constraint loss always maintains a positive value; P is the correlation coefficient loss function, which is implemented based on the Pearson correlation coefficient. When the number of modal types of the input network n = 2, calculating the correlation coefficient loss function is equivalent to calculating the Pearson correlation coefficient of the features between the two modalities, which is expressed as: Where: 1,2 Represents the feature F between mode 1 and mode 2 (1) ,F (2) Pearson correlation coefficient, W and H are modal features F (1) ,F (2) The width and height of the dimensions, F x,y Represents the feature value of the pixel point of feature F at coordinate (x, y); when the number of modal types of the input network n ≥ 2, calculating the correlation coefficient loss function is equivalent to calculating the sum of the Pearson correlation coefficients of any two input features, expressed as: Where: i,j Represents the Pearson correlation coefficient between the features of the i-th modality and the features of the j-th modality. The second is to calculate the accuracy of the classification prediction results of the objects output by each part of the network based on the cross entropy loss function, including the classification prediction result loss of the final output of the network, the classification prediction result loss of the modality-specific encoder output and the classification prediction result loss of the shared encoder output; the classification prediction result loss function is expressed as: L cls =L pred +L specific +L shared Where: L cls is the classification prediction result loss, L pred is the classification prediction loss of the final output of the network, L specific is the classification prediction loss of the modality-specific encoder output, L shared The classification prediction result loss of the modality sharing encoder output; the cross entropy loss function is: Where: N is the number of categories, N is the number of samples, s ec is a sign function, taking values ​​of 0 or 1; p ec is the predicted probability that the observed sample e belongs to category c; The third is to calculate the error between the classification prediction results output by the modality-specific encoder and the classification prediction results output by the shared encoder based on the mean square error loss function as a cross-modal loss to maintain stability and balance between the two encoders; the mean square error loss function is: Where: d e and are the actual observed value and predicted value of the observed sample e respectively; Therefore, the final network loss function is expressed as: L=L cls +αL cor +L cro Where: L cro is the cross-modal loss, and α is a constant that defines the weight of the modal association constraint function.

10. A robust feature classification system based on multimodal relationship modeling and fusion network, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a robust land feature classification method based on multimodal relationship modeling and fusion network as described in any one of claims 1 to 9.

Images