Land coverage classification method and system for image fusion based on multi-modal capsule network

Abstract

The invention relates to the technical field of remote sensing image processing, in particular to a land cover classification method and system for image fusion based on a multi-mode capsule network, and the method comprises the steps: inputting the optical image data and synthetic aperture radar image data of a target region into a pre-trained multi-mode capsule network model for land cover classification prediction; using the multi-modal capsule network model to obtain a target area land coverage classification prediction result; wherein the multi-modal capsule network model performs complementation on image high-level features among different modalities through a multi-modal capsule and predicts and outputs a land cover classification label through fusion of the high-level features and shallow-layer features, and the multi-modal capsule performs complementation on the image high-level features among the different modalities by using a multi-modal capsule learning mechanism. According to the method, the multi-mode remote sensing image can be effectively utilized, high-precision and automatic land coverage classification in a complex environment is realized, and the method has a relatively good application prospect in the fields of land utilization condition and change monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a land cover classification method and system based on multimodal capsule network for image fusion. Background Art

[0002] Land cover classification is an important topic in the field of remote sensing image interpretation. It aims to extract and classify the coverage of the surface pixel level. This technology has wide applications in urban planning, change detection, disaster monitoring, and environmental monitoring and protection. Remote sensing images have significant advantages in obtaining surface cover information: on the one hand, they can cover the entire world; on the other hand, different types of remote sensing images can reveal different surface information, such as height information, texture structure, spectral brightness, and reflectivity. With the continuous advancement of earth observation sensor technology, the acquisition of OPT images and SAR images of the same area is becoming more and more common. Due to its high spatial resolution and rich texture information, OPT images are widely used in many applications. However, OPT images have limitations under certain conditions and are particularly susceptible to weather conditions. In contrast, SAR sensors can work normally under all-weather conditions and have special advantages in identifying specific features of the surface.

[0003] Multimodal fusion strategies have a significant impact on model performance under different data characteristics. In the land cover classification task, the traditional OPT and SAR image fusion methods mainly include data-level fusion, feature-level fusion and decision-level fusion. Based on these basic methods, new technologies and strategies continue to emerge. For example, in recent years, attention mechanisms have been widely used in multimodal fusion, such as effectively capturing the complementary information between different modal features by introducing channel and spatial attention mechanisms; in addition, the loss function is a key link in the fusion method, which accurately constructs the correspondence between input data and fusion results, such as maintaining the consistency of single-modal and multi-modal features by introducing penalty terms. However, these fusion methods may ignore key high-level semantic information, such as the geometric relationship between features and their complex spatial structure, which is crucial for land cover classification tasks. To this end, there is an urgent need for a remote sensing image processing method that can utilize high-level semantic features and multi-scale feature information between different modalities to improve the land cover classification effect in complex environments. Summary of the invention

[0004] To this end, the present invention provides a land cover classification method and system based on multimodal capsule network for image fusion, which solves the problems of insufficient fusion of high-level semantic information between different modalities and insufficient utilization of multi-scale feature information.

[0005] According to the design scheme provided by the present invention, on the one hand, a land cover classification method based on multimodal capsule network for image fusion is provided, comprising:

[0006] Acquire optical image data and synthetic aperture radar image data of the target area;

[0007] The optical image data and synthetic aperture radar image data of the target area are input into the pre-trained multimodal capsule network model for land cover classification prediction, and the land cover classification prediction result of the target area is obtained by using the multimodal capsule network model;

[0008] Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

[0009] As a land cover classification method based on multimodal capsule network for image fusion of the present invention, further, the multimodal capsule network model includes: a first backbone network for extracting shallow features and high-level features of optical images, a second backbone network for extracting shallow features and high-level features of synthetic aperture radar images, a multimodal capsule for combining high-level features of both optical images and synthetic aperture radar images and performing feature complementation based on a multimodal capsule learning mechanism to obtain high-level joint semantic features, a multi-scale fusion unit for performing feature cascade fusion of shallow features and high-level joint semantic features of both optical images and synthetic aperture radar images, and a segmentation head for classifying and predicting land cover based on cascade fusion features.

[0010] As a land cover classification method based on multimodal capsule network for image fusion in the present invention, further, based on the multimodal capsule learning mechanism, high-level features of images between different modalities are complemented, including:

[0011] For the high-level features of optical images and synthetic aperture radar images, the cross-attention mechanism is used to model and fuse the semantic information and abstract information of different image modalities and generate low-level capsules;

[0012] A dynamic routing mechanism is used to summarize low-level capsules and obtain multimodal high-level joint semantic features.

[0013] As a land cover classification method based on multimodal capsule network for image fusion in the present invention, further, a cross-attention mechanism is used to model the semantic information and abstract information of different image modalities and fuse them to generate low-level capsules, including:

[0014] For the high-level features of optical images, the high-level features of synthetic aperture radar images are combined into an optical query vector, and an optical key vector and a value vector are generated based on the high-level features of the optical image. Based on the optical query vector, the key vector and the value vector, the optical feature vector having a potential relationship with the synthetic aperture radar image is searched and mined in the optical feature space.

[0015] According to the high-level features of synthetic aperture radar images, the high-level features of optical images are combined into synthetic aperture radar query vectors, and synthetic aperture radar key vectors and value vectors are combined based on the high-level features of synthetic aperture radar. Based on the synthetic aperture radar query vector, key vector and value vector, synthetic aperture radar feature vectors with potential relationship with optical images are searched and mined in synthetic aperture radar space.

[0016] The optical feature vector and synthetic aperture radar feature vector obtained by search and mining are connected to obtain multi-modal low-level capsules.

[0017] As a land cover classification method based on multimodal capsule network for image fusion in the present invention, further, a dynamic routing mechanism is used to summarize low-level capsules and obtain multimodal high-level joint semantic features, including:

[0018] The multi-modal low-level capsules are weighted summed using a coupling coefficient, where the coupling coefficient is a coefficient of contribution of the low-level capsule to the high-level capsule;

[0019] The squash function is used to obtain the activation value of the weighted sum of the low-level capsules, and the coupling coefficient is dynamically updated according to the scalar product between the bottom-level capsules and the high-level capsules, so as to reshape and compress the multimodal low-level capsules by dynamically updating the coupling coefficient.

[0020] As a land cover classification method based on multimodal capsule network for image fusion in the present invention, further, the process of weighted summation of multimodal low-level capsules using coupling coefficients is expressed as: Among them, e i is the i-th bottom-level capsule p i The coupling coefficient.

[0021] As a land cover classification method based on multimodal capsule network for image fusion in the present invention, further, the high-level feature and shallow-level feature fusion process includes:

[0022] At the element level, shallow features of corresponding scales of different modal images are combined to form joint shallow features;

[0023] The features after the high-level features of different modal images are complemented are used as the features to be spliced, the features to be spliced ​​are spliced ​​with the joint shallow features, and the residual structure is used for feature fusion. The residual structure is composed of a convolutional layer, a batch normalization layer and a ReLU activation function, and the number of residual structures is set according to the number of scales of the shallow feature extraction of the image;

[0024] When the number of residual structures is greater than 1, the feature after feature fusion is used as the feature to be spliced, and the feature to be spliced ​​is spliced ​​with the joint shallow feature of the corresponding scale to perform feature fusion processing using the corresponding residual structure.

[0025] On the other hand, the present invention also provides a land cover classification system based on multimodal capsule network for image fusion, comprising: a data acquisition module and a classification prediction module, wherein:

[0026] A data acquisition module is used to acquire optical image data and synthetic aperture radar image data of the target area;

[0027] A classification prediction module is used to input the optical image data and synthetic aperture radar image data of the target area into the pre-trained multimodal capsule network model for land cover classification prediction, and obtain the land cover classification prediction result of the target area using the multimodal capsule network model;

[0028] Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

[0029] Beneficial effects of the present invention:

[0030] The present invention can effectively establish high-level semantic information and abstract spatial relationships by representing high-level semantic information and abstract relationships between remote sensing images of different modalities as vectors, generate multimodal joint features with high discrimination and robustness, and promote the high discrimination and robustness extraction of joint multimodal features; integrate various feature scales through feature cascade, and simultaneously process deep multimodal features, shallow OPT features and shallow SAR features at each layer to ensure accurate description of local details and global semantics, make full use of the characteristics of different modalities and scales, and effectively overcome the limitations of single-modal land cover classification. And further verified by experimental data, the classification accuracy of this case scheme on the WHU-OPT-SAR data set is improved by 2.86% to 12.9%, indicating that this case scheme can effectively utilize multimodal remote sensing images to achieve high-precision and automatic land cover classification in complex environments, and has good application prospects in the fields of land use status and change monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the land cover classification process based on multimodal capsule network for image fusion in the embodiment;

[0032] Figure 2 This is a schematic diagram of the multimodal capsule network model training process in the embodiment;

[0033] Figure 3 Schematic diagram of a cross-attention multimodal capsule generation example in an embodiment;

[0034] Figure 4 It is a schematic diagram of a dynamic routing example of multi-modal feature induction in the embodiment;

[0035] Figure 5 Schematic diagram of a cascade feature example that fuses deep multi-modal features and shallow single-modal features in an embodiment;

[0036] Figure 6 This is a schematic diagram of the land cover classification results of the WHU-OPTSAR dataset 256×256 pixel image in the embodiment;

[0037] Figure 7 The figure is a visualization diagram of the effectiveness of the multimodal capsule and cascade feature fusion module on the WHU-OPT-SAR dataset in the embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0039] Land cover classification is an important means to monitor land use status and changes. This study explores the effectiveness of integrated optical projection (OPT) and synthetic aperture radar (SAR) images in high-precision land cover classification. However, due to the radiometric and geometric differences between OPT and SAR images, there are two main challenges: insufficient fusion of high-level semantic information between different modalities and insufficient utilization of multi-scale feature information. To this end, an embodiment of the present invention provides a land cover classification method based on multimodal capsule network for image fusion, such as Figure 1 As shown, including:

[0040] S101, acquiring optical image data and synthetic aperture radar image data of the target area;

[0041] S102, inputting the optical image data and synthetic aperture radar image data of the target area into the pre-trained multimodal capsule network model for land cover classification prediction, and using the multimodal capsule network model to obtain the land cover classification prediction result of the target area;

[0042] Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

[0043] Specifically, the multimodal capsule network model can be designed to include: a first backbone network for extracting shallow features and high-level features of optical images, a second backbone network for extracting shallow features and high-level features of synthetic aperture radar images, a multimodal capsule for combining the high-level features of both optical images and synthetic aperture radar images and performing feature complementation based on a multimodal capsule learning mechanism to obtain high-level joint semantic features, a multi-scale fusion unit for cascading the shallow features and high-level joint semantic features of both optical images and synthetic aperture radar images, and a segmentation head for classifying and predicting land cover based on cascaded fusion features.

[0044] Multi-scale feature fusion is a strategy that aims to enhance the representation and generalization performance of the model by effectively fusing features of different scales. This strategy is particularly important when processing high-resolution images, because high-resolution images contain rich details. Simply splicing high-level and low-level features together may not accurately extract location information and more semantically meaningful features from the feature map.

[0045] In this embodiment, see Figure 2The architecture and training process of the multimodal capsule model (M2Caps) shown in Figure 1 takes OPT and SAR images as input and outputs land cover classification results. The architecture is mainly divided into four modules: backbone networks G1 and G2, multimodal capsule modules, cascade feature fusion modules F, and segmentation head H. First, OPT and SAR images are input into their respective backbone networks to avoid mutual interference of multimodal features during the extraction process. The extracted OPT and SAR features are denoted as X o and X s . Then, these extracted OPT and SAR features are combined to generate multimodal capsules, and high-level joint semantic features are guided through cross-attention and dynamic routing mechanisms. After that, these high-level features are fused with shallow OPT and SAR features in the backbone network to further improve the utilization of inter-layer information. Finally, the segmentation head is responsible for predicting the land cover classification and calculating the loss between the predicted result and the true value.

[0046] The backbone network G and the segmentation head H can be constructed in any way. In the present embodiment, the key to achieving high-precision land cover classification is to effectively fuse multimodal features to ensure that the fused features have stronger semantic discrimination and robustness.

[0047] Different from the scalar calculation method in convolutional neural networks, the information transmission mechanism of vectorized capsule neurons and dynamic routing gives capsule networks a higher level of abstract representation capabilities. In order to apply the advantages of capsule networks to multimodal remote sensing image feature fusion, in the embodiment of this case, the high-level features of images between different modalities are complemented based on the multimodal capsule learning mechanism, which can be designed to include:

[0048] For the high-level features of optical images and synthetic aperture radar images, the cross-attention mechanism is used to model and fuse the semantic information and abstract information of different image modalities and generate low-level capsules;

[0049] A dynamic routing mechanism is used to summarize low-level capsules and obtain multimodal high-level joint semantic features.

[0050] Among them, the cross-attention mechanism is used to model the semantic information and abstract information of different image modalities and fuse them to generate low-level capsules, which may include:

[0051] For the high-level features of optical images, the high-level features of synthetic aperture radar images are combined into an optical query vector, and an optical key vector and a value vector are generated based on the high-level features of the optical image. Based on the optical query vector, the key vector and the value vector, the optical feature vector having a potential relationship with the synthetic aperture radar image is searched and mined in the optical feature space.

[0052] According to the high-level features of synthetic aperture radar images, the high-level features of optical images are combined into synthetic aperture radar query vectors, and synthetic aperture radar key vectors and value vectors are combined based on the high-level features of synthetic aperture radar. Based on the synthetic aperture radar query vector, key vector and value vector, synthetic aperture radar feature vectors with potential relationship with optical images are searched and mined in synthetic aperture radar space.

[0053] The optical feature vector and synthetic aperture radar feature vector obtained by search and mining are connected to obtain multi-modal low-level capsules.

[0054] Among them, the dynamic routing mechanism is used to summarize the low-level capsules and obtain multimodal high-level joint semantic features, which may include:

[0055] The multi-modal low-level capsules are weighted summed using a coupling coefficient, where the coupling coefficient is a coefficient of contribution of the low-level capsule to the high-level capsule;

[0056] The squash function is used to obtain the activation value of the weighted sum of the low-level capsules, and the coupling coefficient is dynamically updated according to the scalar product between the bottom-level capsules and the high-level capsules, so as to reshape and compress the multimodal low-level capsules by dynamically updating the coupling coefficient.

[0057] Multimodal capsule learning can be summarized as including capsule generation and feature induction. The purpose of multimodal capsule learning is to model the high-level semantic information and abstract relationships between different remote sensing image modalities, so as to better summarize multimodal joint features with high discriminability and robustness. In existing capsule networks, convolution operations are widely used to generate capsule neurons and have been applied in single-modal data processing. However, the above-mentioned capsule generation method is difficult to effectively handle multimodal remote sensing image fusion, and cannot achieve efficient interaction and fusion of multimodal complementary information. In the embodiment of this case, Figure 3 As shown, the cross-attention mechanism is used to generate multimodal capsules.

[0058] The cross-attention mechanism is implemented as follows:

[0059]

[0060] In the multimodal fusion process, in this embodiment, SAR features are used as queries to search for potential relationships in the OPT feature space. At the same time, for the SAR branch, the OPT query vector is used to calculate the attention map. The advantage of cross attention is that it can effectively mine the correlation and complementarity between different modalities, thereby achieving the initial fusion of multimodal features. In order to further enhance the richness and representation ability of the output features, a multi-head attention mechanism can be used. Finally, all output vectors are connected and mapped to the final output vector

[0061] AM =MHA(Z)=Concat(A1,A2,...,A l )W (2)

[0062] In the formula, l is the number of attention heads and W is the learnable matrix of linear transformation. Finally, by combining OPT output features at the element level And SAR output characteristics The results obtained by reshaping and squashing the activation function form a multimodal capsule Where n is the number of capsules, is the dimension of the capsule.

[0063] The multimodal capsules generated by the cross-attention mechanism contain information from different remote sensing image modalities, but this information may be redundant in practical applications. In order to improve the accuracy of multimodal land cover classification, the key is to learn more compact and discriminative deep features and effectively exclude information irrelevant to the classification, which is the design goal of the feature inducer. Traditional dynamic routing algorithms are mainly used to model the complex spatial interaction relationship between low-level capsules and high-level capsules. In the multimodal fusion task of remote sensing images, the semantic information of different modalities can be regarded as local features, while the multimodal deep features can be regarded as global features. Therefore, in the embodiment of this case, by improving the dynamic routing algorithm and modeling the abstract relationship and semantic information between multimodal modalities, multimodal deep features that are more discriminative and easy to separate can be summarized.

[0064] like Figure 4 As shown in Figure 1, the dynamic routing process of multimodal feature induction is mainly divided into three steps:

[0065] (1) Calculate the weighted sum of multimodal capsules:

[0066]

[0067] where b i It represents the logarithm of the coupling coefficient, with an initial value of 0. Through the softmax activation function, it can ensure that the sum of the weights of all capsules is equal to 1. The coupling coefficient e plays a key role in the dynamic routing algorithm, which determines the contribution of the lower-level capsules to the upper-level capsules.

[0068] (2) Calculate the activation value:

[0069]

[0070] The squash function is a nonlinear operation designed for vectors that can scale the magnitude to [0,1] without changing the direction of the vector.

[0071] (3) Update the dynamic routing coupling coefficient:

[0072] b i =b i +p i u (5)

[0073] The coupling coefficient is updated according to the scalar product between the lower layer capsule and the upper layer capsule. Intuitively, the greater the correlation between the lower layer capsule and the higher layer capsule, the higher the coupling coefficient between the two, and vice versa.

[0074] In this embodiment, the high-level feature and shallow-level feature fusion process can be designed to include:

[0075] At the element level, shallow features of corresponding scales of different modal images are combined to form joint shallow features;

[0076] The features after the high-level features of different modal images are complemented are used as the features to be spliced, the features to be spliced ​​are spliced ​​with the joint shallow features, and the residual structure is used for feature fusion. The residual structure is composed of a convolutional layer, a batch normalization layer and a ReLU activation function, and the number of residual structures is set according to the number of scales of the shallow feature extraction of the image;

[0077] When the number of residual structures is greater than 1, the feature after feature fusion is used as the feature to be spliced, and the feature to be spliced ​​is spliced ​​with the joint shallow feature of the corresponding scale to perform feature fusion processing using the corresponding residual structure.

[0078] The feature maps of objects at different scales focus on the different features of objects and their distribution patterns. By making full use of cascade features, the fusion effect of different remote sensing image modalities can be further improved. To this end, the embodiment of this case uses cascade feature fusion to further improve the accuracy of multimodal land cover classification. Figure 5 As shown in Figure 1, for the multi-scale output of the backbone network, a feature fusion mechanism consisting of three fusion scales can be used. Each fusion layer adopts a standard residual structure, which consists of a convolutional layer, a batch normalization layer, and a ReLU activation function. In the first layer, the obtained multimodal capsules are first Reshape into a deep feature map, denoted as At the same time, shallow OPT features and SAR features are added at the element level. Then, the feature vector obtained above is combined with After splicing, the input is sent to the first residual module for feature fusion. In the subsequent fusion layer, similar operations are performed. In the cascade feature fusion, each layer processes deep multimodal features, shallow OPT features, and shallow SAR features simultaneously. The high-level features of the previous layers can effectively guide the high-level feature learning in the subsequent stages, thereby improving the effectiveness of multi-scale feature fusion and ensuring the full utilization of multimodal features. In addition, the three-layer cascade structure can effectively enhance the interaction and fusion between features of different scales and levels. These two aspects together improve the accuracy of multimodal land cover classification.

[0079] Furthermore, based on the above method, an embodiment of the present invention also provides a land cover classification system based on multimodal capsule network for image fusion, comprising: a data acquisition module and a classification prediction module, wherein:

[0080] A data acquisition module is used to acquire optical image data and synthetic aperture radar image data of the target area;

[0081] A classification prediction module is used to input the optical image data and synthetic aperture radar image data of the target area into the pre-trained multimodal capsule network model for land cover classification prediction, and obtain the land cover classification prediction result of the target area using the multimodal capsule network model;

[0082] Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

[0083] In order to verify the effectiveness of this solution, the following is a further explanation of this solution combined with experimental data:

[0084] The WHU-OPT-SAR dataset contains OPT images from GF-2 and SAR images from GF-3, both of which cover the same geographical area of ​​Hubei Province, China, which mainly includes various land cover types such as "farmland", "city", "village", "water body", "forest", "road" and "other". The preprocessed OPT images and SAR images have a spatial resolution of 5 meters, ensuring the precise correspondence between pixels. Specifically, the WHU-OPT-SAR dataset consists of 100 OPT images with a resolution of 5,556×3,704 pixels and SAR images of the corresponding areas. For the convenience of subsequent analysis, these images were segmented into 7,000 non-overlapping tiles, each with a size of 256×256 pixels, which were allocated to the training set and the test set in a ratio of 9:1.

[0085] The experiment was conducted on a computer equipped with 16GB RAM, i7-9750H processor and A100 graphics card. The experimental program was developed and implemented in Python language and deep learning library TensorFlow. To ensure the consistency and comparability of the experiments, each experiment used the same parameters and training strategy, including: batch size set to 8, learning rate set to 8e-4, number of training rounds (epochs) set to 300, and AdamW optimizer was used. Since the experiment involves random processes such as weight initialization and data partitioning, a fixed random seed is set in each experiment to ensure the repeatability of the experimental results and reduce the volatility of the experimental results.

[0086] Through experiments, various algorithms are comprehensively evaluated. In order to accurately evaluate the effectiveness of different methods, multiple evaluation indicators are used, including overall accuracy (OA), mIoU, IoU of each category, category accuracy, and Kappa coefficient. Overall accuracy (OA) is a direct measure of precision, which represents the percentage of correctly predicted pixels. However, due to the imbalance of sample sizes in different categories, OA itself may not be sufficient to fully reflect the performance of the model. For this reason, mIoU and Kappa coefficients are also introduced to provide a more comprehensive evaluation, taking into account the deviation of model predictions and measuring their consistency with actual classifications. The specific calculation methods of these indicators are listed below.

[0087]

[0088]

[0089]

[0090]

[0091] Where k represents the total number of different land cover types. In the experiment, k is set to 7. ii represents the number of pixels from the i-th class that are accurately predicted, p ij represents the number of pixels originally from the i-th class that are incorrectly assigned to the j-th class. Similarly, p ji It represents the number of pixels from class j that are incorrectly predicted to be of class i.

[0092] In order to evaluate the M 2The performance of the Caps model in the land cover classification task is compared with six current best methods. These methods include AMM-FuseNet, CaFE, MCANet, MIEFNet, MFGF-UNet and SOCF. AMM-FuseNet adopts a hybrid strategy of channel attention mechanism and combines densely connected spatial pyramid pooling. CaFE assigns weights to each category area according to the proportion of samples in the entire image. MCANet designs a multimodal cross-attention network to enhance information fusion between different modalities. MIEFNet builds an information exchange mechanism between features to improve classification. MFGF-UNet proposes a gated multi-filter inception module and introduces a gated channel conversion mechanism, using the attention mechanism to further improve the performance of the model. SOCF adopts a channel attention module to highlight feature-rich channels while suppressing the interference of irrelevant information.

[0093] The spatial resolution of the WHU-OPT-SAR dataset is 5 meters. The results are shown in Table 1, and the optimal evaluation index results are marked in bold. It should be pointed out that the SAR images in this dataset are single-channel grayscale images with limited information, which leads to low values ​​of the evaluation index. According to the results in Table 1, M 2 The Caps model has made significant progress in various evaluation indicators. 2 Caps’ mIoU is improved by 2.86%. 2 Caps’ IoU scores in the “city” category are 3.1% to 8.98% higher than those of other methods, indicating that the model performs well in building detection. 2 The IoU of Caps is 3.28% to 21.07% higher than that of other methods, proving that it has a clear advantage in accurately depicting road edges. The higher OA and Kappa coefficients further indicate that M 2 Caps can fuse multi-modal shared information while retaining the independent characteristics of each modality.

[0094] Table 1 Quantitative comparison results of the models on the WHU-OPT-SAR dataset.

[0095]

[0096] Figure 6Examples of segmentation results of different methods are shown, (a) OPT image; (b) SAR image; (c) true label; (dj) M2Caps, AMM-Fusenet, CaFE, MCANet, MFGF-UNet, MIEFNET and SOCF. In image 1, a large area of ​​"forest" is incorrectly classified as "farmland". This error is mainly attributed to the fact that these categories are visually difficult to distinguish and there is a certain degree of confusion between OPT and SAR images on these categories. However, M 2 The advantage of Caps is that it can not only correctly identify "farmland" and "forest", but also accurately depict their boundaries. 2 Caps accurately identified the narrow “water body” and extracted clear boundaries, while other methods incorrectly classified it as a “road” due to its narrow shape. 2 Caps successfully depicts the complete “water body” shape adjacent to “city”, while other methods misclassify it as “city” due to texture interference. In addition, in image 4, M 2 Caps accurately outlines narrow roads without breaks, which other methods fail to achieve. This high accuracy is not only reflected in narrow roads, but also in complex and ambiguous areas. Overall, M 2 The boundaries generated by Caps are significantly better than those of other methods, providing a clearer and more accurate description, confirming the reliability of our scheme in edge refinement and feature differentiation.

[0097] Further ablation experiments were designed to test the 2 The effectiveness of the multimodal capsule module and cascade feature fusion module in Caps, as well as the necessity of combining SAR and OPT data. The experimental results are shown in Table 2.

[0098] Table 2 Ablation study of different modules and data combinations on the WHU-OPT-SAR dataset. The SAR-OPT* experiment is based on the baseline.

[0099]

[0100] As shown in Table 2, on the test set, the M 2 Caps performs best in terms of overall accuracy (OA), Kappa coefficient, and mIoU score. Specifically, compared with the methods using only OPT or SAR images, the M 2The mIoU scores of the Caps model increased by 4.98% and 11.42% respectively. This result not only demonstrates the complementarity between OPT images and SAR images, but also further verifies the effectiveness of using both images simultaneously in land cover classification tasks. When the multimodal capsule module is introduced, the mIoU value is increased by 2.45% compared to the baseline; on this basis, after adding cascade feature fusion, the mIoU value is further increased by 1.92%. This is because M 2 Caps makes full use of the capsule module and cascade feature fusion module to effectively capture features of different shapes and scales. The experimental results fully prove that M 2 Superiority of CAPS in land cover classification tasks.

[0101] The capsule network uses a dynamic routing algorithm to aggregate relevant features to the corresponding capsule units. The feature points are converted and propagated through multimodal capsules, and the features are refined and aggregated while retaining the local spatial relationship. During the feature fusion process, high-level semantic information is combined with low-level features to enrich the semantic content of low-level features. Based on this principle, the cascade feature fusion module optimizes and aggregates features from different scales. In order to intuitively prove the effectiveness of the multimodal capsule and cascade feature fusion module, a visual analysis was performed in the experiment: (a) OPT image feature point distribution; (b) SAR image feature point distribution; (c) feature point distribution generated by M2Caps without multimodal capsule and cascade feature fusion module; (d) feature point distribution generated by M2Caps with multimodal capsule and cascade feature fusion module. Figure 6 As shown in (c) of the figure, although the SAR image and OPT image features are 2 The Caps baseline network shows some aggregation, but there are still significant outliers, and the effect is not satisfactory. Figure 6 (d) shows that with the help of multimodal capsule and cascade feature fusion module, M 2 Caps significantly improves the aggregation effect of feature points, and clustered feature points are more concentrated. Based on the above analysis, the visualization experiment confirms the effectiveness of the scheme integrating multimodal capsules and cascade feature fusion modules in improving the accuracy of land cover classification.

[0102] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0103] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0104] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0105] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0106] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A land cover classification method based on multimodal capsule network for image fusion, characterized in that: Include: Acquire optical image data and synthetic aperture radar image data of the target area; The optical image data and synthetic aperture radar image data of the target area are input into the pre-trained multimodal capsule network model for land cover classification prediction, and the land cover classification prediction result of the target area is obtained by using the multimodal capsule network model; Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

2. The land cover classification method based on multimodal capsule network for image fusion according to claim 1, characterized in that: The multimodal capsule network model includes: a first backbone network for extracting shallow features and high-level features of optical images, a second backbone network for extracting shallow features and high-level features of synthetic aperture radar images, a multimodal capsule for combining high-level features of both optical images and synthetic aperture radar images and performing feature complementation based on a multimodal capsule learning mechanism to obtain high-level joint semantic features, a multi-scale fusion unit for performing feature cascade fusion of shallow features and high-level joint semantic features of both optical images and synthetic aperture radar images, and a segmentation head for classifying and predicting land cover based on cascade fusion features.

3. The land cover classification method based on multimodal capsule network for image fusion according to claim 1 or 2, characterized in that: Based on the multimodal capsule learning mechanism, high-level image features between different modalities are complemented, including: For the high-level features of optical images and synthetic aperture radar images, the cross-attention mechanism is used to model and fuse the semantic information and abstract information of different image modalities and generate low-level capsules; A dynamic routing mechanism is used to summarize low-level capsules and obtain multimodal high-level joint semantic features.

4. The land cover classification method based on multimodal capsule network for image fusion according to claim 3 is characterized in that: The cross-attention mechanism is used to model the semantic information and abstract information of different image modalities and fuse them to generate low-level capsules, including: For the high-level features of optical images, the high-level features of synthetic aperture radar images are combined into an optical query vector, and an optical key vector and a value vector are generated based on the high-level features of the optical image. Based on the optical query vector, the key vector and the value vector, the optical feature vector having a potential relationship with the synthetic aperture radar image is searched and mined in the optical feature space. According to the high-level features of synthetic aperture radar images, the high-level features of optical images are combined into synthetic aperture radar query vectors, and synthetic aperture radar key vectors and value vectors are combined based on the high-level features of synthetic aperture radar. Based on the synthetic aperture radar query vector, key vector and value vector, synthetic aperture radar feature vectors with potential relationship with optical images are searched and mined in synthetic aperture radar space. The optical feature vector and synthetic aperture radar feature vector obtained by search and mining are connected to obtain multi-modal low-level capsules.

5. The land cover classification method based on multimodal capsule network for image fusion according to claim 3 is characterized in that: The dynamic routing mechanism is used to summarize low-level capsules and obtain multimodal high-level joint semantic features, including: The multi-modal low-level capsules are weighted summed using a coupling coefficient, where the coupling coefficient is a coefficient of contribution of the low-level capsule to the high-level capsule; The squash function is used to obtain the activation value of the weighted sum of the low-level capsules, and the coupling coefficient is dynamically updated according to the scalar product between the bottom-level capsules and the high-level capsules, so as to reshape and compress the multimodal low-level capsules by dynamically updating the coupling coefficient.

6. The land cover classification method based on multimodal capsule network for image fusion according to claim 5, characterized in that: The process of weighted summation of multimodal low-level capsules using coupling coefficients is expressed as: Among them, e i is the i-th bottom-level capsule p i The coupling coefficient.

7. The land cover classification method based on multimodal capsule network for image fusion according to claim 1, characterized in that: The high-level feature and shallow-level feature fusion process includes: At the element level, shallow features of corresponding scales of different modal images are combined to form joint shallow features; The features after the high-level features of different modal images are complemented are used as the features to be spliced, the features to be spliced ​​are spliced ​​with the joint shallow features, and the residual structure is used for feature fusion. The residual structure is composed of a convolutional layer, a batch normalization layer and a ReLU activation function, and the number of residual structures is set according to the number of scales of the shallow feature extraction of the image; When the number of residual structures is greater than 1, the feature after feature fusion is used as the feature to be spliced, and the feature to be spliced ​​is spliced ​​with the joint shallow feature of the corresponding scale to perform feature fusion processing using the corresponding residual structure.

8. A land cover classification system based on multimodal capsule network for image fusion, characterized in that: Contains: data acquisition module and classification prediction module, among which, A data acquisition module is used to acquire optical image data and synthetic aperture radar image data of the target area; A classification prediction module is used to input the optical image data and synthetic aperture radar image data of the target area into the pre-trained multimodal capsule network model for land cover classification prediction, and obtain the land cover classification prediction result of the target area using the multimodal capsule network model; Among them, the multimodal capsule network model complements the high-level features of images between different modalities through multimodal capsules and predicts and outputs land cover classification labels by fusing high-level features and shallow features. The multimodal capsules complement the high-level features of images between different modalities using a multimodal capsule learning mechanism. The high-level features and the shallow features are different scale features extracted using the corresponding modality backbone network.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images