A multi-modal land cover classification method and system based on category-aware asymmetric dynamic fusion network

Abstract

The application provides a kind of multi-modal land cover classification method and system based on category perception asymmetric dynamic fusion network, method includes: obtaining the remote sensing optical-SAR image pair of the land cover of study area;The remote sensing optical-SAR image pair of the land cover of the region to be studied is input into the trained multi-modal remote sensing semantic segmentation model, and the classification result of the land cover of the study area is output;Wherein, in the encoder part of multi-modal remote sensing semantic segmentation model, the feature extraction layer adopts two independent non-shared weight branches to process each modality image respectively, and the multi-scale features of each modality are extracted in stages, after each stage, the multi-scale feature difference of each modality extracted in the current stage is enhanced by a multi-level difference pyramid fusion module, then the adaptive feature fusion and alignment based on target category semantic information are carried out by dynamic sparse category attention module, to obtain the feature map of the multi-scale geometric structure enhancement and semantic feature fusion of each modality.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a multimodal land cover classification method and system based on a category-aware asymmetric dynamic fusion network. Background Technology

[0002] The differences in radiometric and geometric features between optical and SAR images provide complementary information, and integrating their data to improve the accuracy of urban land cover classification has proven effective. However, traditional machine learning-based models typically rely on domain prior knowledge and manually designed feature descriptors, resulting in limited feature representation capabilities and poor domain adaptability, which may lead to information loss and insufficient segmentation. Deep learning, especially networks based on the CNN collaborative attention mechanism, has shown strong application potential in multimodal land use classification tasks. However, existing methods have two key limitations: (1) modality independence is suppressed, and (2) category-based information is ignored.

[0003] The first problem is the suppression of modal independence. Current ConvNet-based multimodal fusion methods mostly employ symmetric fusion frameworks. This approach assigns the same attention weights to different branches, allowing the two modalities to share key features, thus achieving more efficient feature fusion. However, optical images and SAR images have different imaging mechanisms, and such methods ignore the heterogeneity gaps between features of different modalities, hindering the model from learning semantic differences between features. Although recent studies have attempted to alleviate this problem using asymmetric fusion frameworks and have focused on the varying degrees of contribution of feature information from different modalities to model performance improvement during segmentation, these methods still have certain limitations. For example, Li et al. proposed a progressive fusion learning framework (MMFNet), but this method cannot dynamically adjust the fusion strategy, and the progressive fusion framework fixes the information transmission path, lacking the flexibility for cross-modal interaction. Huang et al. proposed a two-stream asymmetric fusion network (DAFNet), but the asymmetric feature fusion module (AFF) proposed in this method relies on static convolutions and skip connections, lacking dynamic interaction.

[0004] One problem with attention-based fusion frameworks is the neglect of category-based information. Category-based information is often embedded in different spectra, i.e., different channels of the input features. Research shows that category information is a key factor in semantic segmentation tasks. The category imbalance problem can be alleviated by learning the relationship between the channels and categories of feature maps and enhancing the response of important features in the category channels. Therefore, while pursuing the model's ability to vertically model the target, it is crucial to acquire category-based information in the intermediate stages of the network and consider the relationships between features of different categories. However, attention-based methods often ignore the importance of "category semantic" fusion from a spatial and channel perspective. For example, Ye et al. proposed an Attention Multi-Scale Network (AMSNet), which uses spatial attention combined with an adaptive weight learning strategy to capture important modal information and extracts multi-scale features through different convolutional kernels to improve the accuracy of semantic segmentation. Zhang et al. proposed an Asymmetric Semantic Alignment Network (ASANet), which learns the weights of two modal channels and spatial features through feature interaction, thereby promoting the fusion of features of the two modalities. None of these methods extract class-level information and its corresponding attention representations; instead, they are based on global feature selection. Summary of the Invention

[0005] To overcome the limitations of existing methods that combine optical and SAR images to improve the accuracy of urban land cover classification, such as the suppression of modal independence and the neglect of category-based information, this invention provides a multimodal land cover classification method and system based on a category-aware asymmetric dynamic fusion network. The method provides a multimodal remote sensing semantic segmentation model that extracts features from optical and synthetic aperture radar images simultaneously in parallel through two branches. Then, it combines a multi-level differential pyramid fusion module and a dynamic sparse category attention module to effectively mine the specific and complementary features of optical and SAR images and fuse the semantic information of categories, significantly improving the accuracy of land cover classification in category-imbalanced scenarios.

[0006] According to one aspect of the present invention, a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network is provided, comprising:

[0007] Acquire remote sensing optical-SAR image pairs of land cover in the study area;

[0008] The remote sensing optical-SAR image pairs of land cover in the study area are input into the trained multimodal remote sensing semantic segmentation model, and the classification results of land cover in the study area are output.

[0009] The training process of the multimodal remote sensing semantic segmentation model includes:

[0010] Obtain the training dataset for the multimodal remote sensing semantic segmentation model;

[0011] A multimodal remote sensing semantic segmentation model based on CADFNet is constructed: In the encoder part, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the multi-level differential pyramid fusion module performs cross-modal differential feature enhancement on the multi-scale feature maps of each modality extracted in the current stage. Then, the dynamic sparse class attention module performs adaptive feature fusion and alignment based on the semantic information of the target class to obtain feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales. Through the feature integration layer, the feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales output by the encoder part are stitched together and the number of channels is restored to the original input dimension. Then, they are sent to the decoder part, and the decoder part decodes to obtain the land cover classification result. The multimodal remote sensing semantic segmentation model is trained using the training dataset, and the trained multimodal remote sensing semantic segmentation model is output.

[0012] As a further technical solution, the multi-level differential pyramid fusion module takes the multi-scale feature maps of RGB and SAR modes as the first input, and processes them through a pyramid pooling strategy combined with bidirectional pixel-wise subtraction to obtain multi-scale differential feature maps of each mode. The differential convolution module processes the multi-scale differential feature maps of each mode separately and outputs the weight map of each mode. The gating generator processes the spliced ​​features of the first input based on the gating mechanism to generate the channel adaptive fusion weights of each mode. The feature fusion part outputs the differentially enhanced multi-scale feature maps of different modes.

[0013] As a further technical solution, the process of obtaining multi-scale difference feature maps by combining pyramid pooling strategy with bidirectional pixel-wise subtraction operations includes:

[0014] The first input is processed using a pyramid pooling strategy, including: multi-scale max pooling of the feature map of the RGB mode and multi-scale average pooling of the feature map of the SAR mode, and multiplying them by adaptive scale weights to balance the contributions of feature maps at different scales.

[0015] Following the rule of the same spatial location and scale, pixel-by-pixel subtraction is performed on each modality. The feature map corresponding to the modality in the first input is subtracted from the feature map of another modality processed by the pyramid pooling strategy to obtain the multi-scale difference feature map of each modality.

[0016] As a further technical solution, in the feature fusion part, the weight map of each modality is multiplied with the channel adaptive fusion weight to obtain the weight map of each modality channel weighted. The weight map of each modality channel weighted is multiplied with the learnable scaling coefficient, and then connected with the residual of the multi-scale feature map of each modality of the original input of the multi-level differential pyramid fusion module through element-wise addition operation to obtain the differentially enhanced multi-scale feature map of each modality.

[0017] As a further technical solution, the dynamic sparse category attention module adopts a channel-space attention mechanism composed of a category adaptive weighting module and a cross-modal collaborative module.

[0018] As a further technical solution, the category adaptive weighting module processes the differentially enhanced multi-scale feature maps of each modality through a dynamic offset generator to obtain the dynamic offset of each modality; the initial category-specific weights of the category adaptive weighting module are added element-wise to each dynamic offset after being weighted by the learnable scaling coefficient to obtain the category-specific weights adjusted by the category adaptive weighting module; the category-specific weights adjusted by the category adaptive weighting module are multiplied by the dynamic sparse mask of each target class, and then normalized to obtain the sparse weight map of each modality and each target class.

[0019] As a further technical solution, the cross-modal collaboration module averages the sparse weight maps of each modality and each target category output by the category adaptive weighting module, and then performs global pooling to obtain a category importance vector. Based on the category importance vector, the multi-scale feature maps of each modality enhanced by differential enhancement are fused using an attention mechanism to obtain a semantically enhanced cross-modal attention map. The semantically enhanced cross-modal attention map is mapped to a spatial-category attention weight map through a category projection layer. After expanding the channel dimension, it is multiplied element-wise with the multi-scale feature maps of each modality enhanced by differential enhancement to obtain a feature map of geometric structure enhancement and semantic feature fusion at each modality multi-scale.

[0020] According to another aspect of this specification, a multimodal land cover classification system based on a category-aware asymmetric dynamic fusion network is provided, comprising:

[0021] The input module is used to acquire remote sensing optical-SAR image pairs of land cover in the study area;

[0022] The land cover classification module is used to input remote sensing optical-SAR image pairs of land cover in the study area into the trained multimodal remote sensing semantic segmentation model, and output the classification results of land cover in the study area.

[0023] The training process of the multimodal remote sensing semantic segmentation model includes:

[0024] Obtain the training dataset for the multimodal remote sensing semantic segmentation model;

[0025] A multimodal remote sensing semantic segmentation model based on CADFNet is constructed: In the encoder part, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the multi-level differential pyramid fusion module performs cross-modal differential feature enhancement on the multi-scale feature maps of each modality extracted in the current stage. Then, the dynamic sparse class attention module performs adaptive feature fusion and alignment based on the semantic information of the target class to obtain feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales. Through the feature integration layer, the feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales output by the encoder part are stitched together and the number of channels is restored to the original input dimension. Then, they are sent to the decoder part, and the decoder part decodes to obtain the land cover classification result. The multimodal remote sensing semantic segmentation model is trained using the training dataset, and the trained multimodal remote sensing semantic segmentation model is output.

[0026] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network.

[0027] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network.

[0028] Compared with existing technologies, the beneficial effects of this invention are as follows: By employing a dual-branch non-shared weight encoder to extract features from both RGB and SAR images simultaneously, and then combining a multi-level differential pyramid fusion module and a dynamic sparse class attention module, the invention effectively mines the unique and complementary features of optical and SAR images, and fuses class semantic information. This fully utilizes and enhances modal difference features, achieving dynamic adaptive feature fusion oriented towards classes, significantly improving the accuracy of land use classification in class-imbalanced scenarios. The multimodal remote sensing semantic segmentation model, through a class-adaptive weighting mechanism and cross-modal semantic collaboration, achieves class-level differentiated weight allocation and intermodal feature interaction and alignment, effectively enhancing the model's feature expression ability for small sample classes and alleviating the class imbalance problem. Furthermore, by introducing a dynamic sparse mechanism and semantically aware attention fusion, the multimodal remote sensing semantic segmentation model makes the extracted land cover classification results have smoother boundaries and are closer to the real labels, exhibiting a more complete extraction effect and higher semantic consistency in small sample classes. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network provided in an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the encoder part of the multimodal remote sensing semantic segmentation model in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a multi-level differential pyramid fusion module in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the category adaptive weighting module in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the final layer cross-modal collaboration module and the feature integration layer in an embodiment of the present invention;

[0035] Figure 6 The pixel ratios of each category in the two datasets in this embodiment of the invention;

[0036] Figure 7 This is a schematic diagram of the segmentation results of the CADFNet model and various comparison methods on the PIE-RGB-SAR dataset in an embodiment of the present invention;

[0037] Figure 8 This is a schematic diagram of the segmentation results of the CADFNet model and various comparison methods on the WHU-OPT-SAR dataset in an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of the structure of a multimodal land cover classification system based on a category-aware asymmetric dynamic fusion network, provided in an embodiment of the present invention.

[0039] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0040] It should be noted that:

[0041] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0044] like Figure 1 As shown, a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network includes:

[0045] Step 1: Acquire remote sensing optical-SAR image pairs of land cover in the study area;

[0046] Step 2: Input the remote sensing optical-SAR image pairs of land cover in the study area into the trained multimodal remote sensing semantic segmentation model, and output the classification results of land cover in the study area.

[0047] Step 2, the training process of the multimodal remote sensing semantic segmentation model includes:

[0048] Step 2-1: Obtain the training dataset for the multimodal remote sensing semantic segmentation model;

[0049] Step 2-2: Construct a multimodal remote sensing semantic segmentation model based on CADFNet: such as Figure 2 As shown, in the encoder section, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the feature maps of each modality extracted in the current stage are enhanced by differential enhancement through a multi-level differential pyramid fusion module. Then, adaptive feature fusion and alignment based on target category semantic information are performed through a dynamic sparse class attention module. Finally, the encoder section outputs feature maps with multi-scale geometric structure enhancement and semantic feature fusion. The decoder section decodes the encoding results of the encoder section and outputs the land cover classification results.

[0050] Steps 2-3: Train the model using the training dataset and output the trained multimodal remote sensing semantic segmentation model.

[0051] Specifically, in step 2-1, the process of constructing the training dataset for the multimodal remote sensing semantic segmentation model includes:

[0052] A high-resolution multimodal remote sensing image dataset was collected, containing RGB-SAR remote sensing image pairs and their corresponding label data. To adapt to GPU limitations, the multimodal remote sensing image dataset underwent the following preprocessing:

[0053] Image cropping: Since the images generated by the CADFNet network are all of the same size, and the inputs to its encoders are also all images of the same size, the images in the dataset are cropped into K*K image blocks. This process uses a sliding window non-overlapping cutting method to generate a dataset that is adapted to GPU limitations.

[0054] Dataset augmentation: To obtain more high-quality data, we performed data augmentation operations such as random flipping, random scaling, and random cropping on image patches to increase data diversity, expand the dataset, and obtain the training dataset for the multimodal remote sensing semantic segmentation model.

[0055] In step 2-2, the CADFNet-based multimodal remote sensing semantic segmentation model includes an encoder and a decoder with an asymmetric structure. The encoder comprises a feature extraction layer, a multi-level differential pyramid fusion module (MDPF), and a dynamic sparse category attention module (DSCA). The DSCA module is further composed of a cascaded Category Adaptive Weighting Module (CAW) and a Cross-modal collaboration module (CMC) to achieve category-aware feature fusion and semantic alignment. The decoder uses a feature pyramid structure to upsample and integrate the multi-scale fused features output by the encoder. The input to the CADFNet-based multimodal remote sensing semantic segmentation model is an RGB-SAR image pair and its corresponding ground truth labels, and the output is the land cover classification result.

[0056] In the encoder section, the feature extraction layer is a parallel dual-branch process, with one branch processing RGB images and the other processing SAR images, using ConvNeXtV2-tiny as the backbone network. Each branch processes the input image through four feature extraction stages, each consisting of a 7×7 depthwise separable convolutional layer, LayerNorm normalization, 1×1 upscaling convolution, GELU activation, global response normalization (GRN module), 1×1 downscaling convolution, and DropPath regularization to process the feature map. The GRN module calculates the L2 norm of the feature map as the global response and, combined with a learnable scaling factor and bias parameters, dynamically calibrates the features, thereby enhancing inter-channel contrast and selectivity and effectively strengthening the expressive power of modality-specific features.

[0057] Optionally, as shown in Table 1, the four feature extraction stages are as follows: Stage 0 (4× downsampling, generating a 96-channel, 128×128 feature map); Stage 1 (8× downsampling, 192 channels, 64×64); Stage 2 (16× downsampling, 384 channels, 32×32); and Stage 3 (32× downsampling, 768 channels, 16×16).

[0058] Table 1. Specific network parameters of the feature extraction layer

[0059]

[0060] In the encoder section, the Multi-level Difference Pyramid Fusion Module (MDPF) takes the feature maps of RGB and SAR modes as the first input and processes them using an asymmetric pyramid pooling strategy combined with bidirectional pixel-wise subtraction to obtain multi-scale difference feature maps for each mode. The differential convolution module processes the multi-scale difference feature maps of each mode separately and outputs a weight map specific to each mode. The gated generator processes the concatenated features of the first input based on the gating mechanism to generate channel-adaptive fusion weights. Finally, in the feature fusion part, the fusion weights output by the differential convolution module and the gated generator are multiplied to obtain weighted features, which are then fused to the first input through residual connections to obtain the differentially enhanced multi-scale feature maps for each mode.

[0061] The process of obtaining multi-scale difference feature maps by combining asymmetric pyramid pooling with bidirectional pixel-wise subtraction includes:

[0062] Firstly, an asymmetric pyramid pooling strategy is adopted to process the first input, including: multi-scale max pooling of the feature map of the RGB mode and multi-scale average pooling of the feature map of the SAR mode, and multiplying them by an adaptive scale weight factor to balance the contribution of features at different scales.

[0063] Specifically, considering the differences in scale sensitivity among different land cover categories, an asymmetric pyramid pooling strategy is adopted to capture contextual information from local details to the global region: First, multi-scale max pooling is performed on the RGB modal feature map, and multi-scale average pooling is performed on the SAR modal feature map. The contribution of multi-scale features is balanced by an adaptive scale weight factor to avoid large-scale features dominating the output, while avoiding excessive smoothing of details of small targets, ensuring that features of different scales have balanced weights.

[0064] Secondly, a pyramid pooling strategy is used to process the first input, including: multi-scale max pooling of the feature map of RGB mode and multi-scale average pooling of the feature map of SAR mode, and multiplying them by adaptive scale weights to balance the contributions of feature maps of different scales.

[0065] Following the rule of the same spatial location and scale, pixel-by-pixel subtraction is performed on each modality. The feature map corresponding to the modality in the first input is subtracted from the feature map of another modality processed by the pyramid pooling strategy to obtain the multi-scale difference feature map of each modality.

[0066] The MDPF module employs an asymmetric differential computation strategy to explicitly model cross-modal feature differences. The process includes: following the differential amplification principle, performing bidirectional pixel-wise subtraction between the enhanced RGB or SAR modal feature map (after pyramid pooling) and the original feature map of another modality to obtain the differential feature map. This includes subtracting the SAR modal features (after multi-scale average pooling) from the original RGB modal features to obtain the differential feature map of the RGB modality. To enhance the unique texture and structural information of RGB images, the RGB mode features after multi-scale max pooling are subtracted from the original SAR mode features to obtain the differential feature map of the SAR mode. This method enhances the ability of SAR images to characterize geometric structures and surface scattering properties, while reducing noise interference. Through this differential computation of feature maps, the most discriminative complementary information between the two modes can be effectively extracted and enhanced, achieving synergistic optimization of mode-specific and complementary features.

[0067] To further achieve adaptive fusion of differential features, the MDPF module is designed with a three-layer technical architecture: it captures and enhances the differential features between modes under different receptive fields through a multi-scale pyramid difference strategy (pyramid pooling strategy combined with bidirectional pixel-wise subtraction operation); and it introduces a learnable scaling coefficient as a control parameter by generating spatially adaptive fusion weights to precisely adjust the influence of differential features on the final output, ensuring a balanced fusion of modality-specific information and complementary information.

[0068] Specifically, such as Figure 3 The diagram shows the data flow of the MDPF module processing multi-scale feature maps of different modalities:

[0069] The multi-scale difference feature generation part, for the RGB modal feature map F RGB The original RGB mode feature map is subtracted pixel-by-pixel from the SAR feature map after multi-scale average pooling to obtain the multi-scale difference feature map of the RGB mode; for the SAR mode feature map F SAR The original feature map of the SAR mode is subtracted pixel by pixel from the RGB feature map after multi-scale max pooling to obtain the multi-scale difference feature map of the SAR mode.

[0070] Subsequently, adaptive scale weights are introduced to balance the contributions of features at different scales. The difference features at each scale are then weighted...

[0071] Each with its corresponding weight After multiplication, the results are accumulated to obtain the fused multi-scale difference feature map. The adaptive scale weights... The scale is calculated using global hybrid pooling (the sum of global average pooling and global max pooling) and the inverse scale factor. Differential convolutional blocks process the multi-scale differential feature maps of RGB and SAR modes respectively. Each differential convolutional block consists of a convolutional layer, a group normalization layer, and an activation function, and outputs weight maps of RGB and SAR modes to guide feature fusion.

[0072] The gating generator takes the SAR modality feature map and RGB modality feature map after channel concatenation (C) as input, and processes them sequentially through adaptive average pooling, convolutional layer (Conv), GELU activation function, and convolutional layer (Conv). After sigmoid activation, a channel attention vector is obtained. This vector is split into two parts along the channel dimension, and finally, the adaptive channel fusion weights W for each modality are generated. RGB— (RGB modality), W SAR— (SAR mode).

[0073] In the feature fusion section, the spatial weight maps of RGB and SAR modes output by the differential convolutional block are first fused with the channel adaptive fusion weights W output by the gated generator. RGB W SAR Multiplying these together yields a feature weight map jointly weighted by channel and spatial factors; then, a learnable scaling factor is introduced. (corresponding to different modes are respectively) and Using this as a control parameter, the weighted graph is multiplied by a learnable scaling factor to balance the relative importance of different modalities. The adjusted weight map is concatenated with the original input residual of the MDPF module through element-wise addition to obtain a differentially enhanced multi-scale feature map. (SAR mode) (RBG mode) enables the alignment and enhancement of geometric structural features.

[0074] Furthermore, in the encoder part, the Dynamic Sparse Category Attention Module (DSCA) adopts a channel-spatial attention mechanism consisting of a category-adaptive weighted module (CAW) and a cross-modal collaborative module (CMC), which realizes deep semantic fusion of RGB and SAR features under category guidance.

[0075] First, the Category Adaptive Weighted Module (CAW) assigns an independent feature learning space to each target category and generates differentiated weights for each category in an adaptive manner using a dynamic sparsity mechanism. This breaks the weight sharing limitation of traditional symmetric fusion and effectively enhances the feature representation capability of small sample categories.

[0076] The category-adaptive weighting module processes the multi-scale stitched features of bimodal differential enhancement through a dynamic offset generator to calculate the dynamic offset of each modality. The initial category-specific weights of the category-adaptive weighting module are added element-wise to each dynamic offset after being weighted by the learnable scaling coefficient to obtain the category-specific weights adjusted by the category-adaptive weighting module. The category-specific weights adjusted by the category-adaptive weighting module are multiplied by the dynamic sparse mask of each target class to achieve sparsity of feature selection. Then, weight normalization is performed to finally output the sparse weight map of each target class of each modality.

[0077] Specifically, in the class-adaptive weighted module, the dynamic offset generator consists of two convolutional layers. First, it concatenates the differentially enhanced multi-scale feature maps of RGB and SAR modes, reduces the dimensionality of the concatenated feature map to C / reduction, activates it with GELU, and outputs a dynamic offset of num_classes×2 channels through a 1×1 convolutional layer. Subsequently, the dynamic offset range is constrained by the tanh function and multiplied by the learnable scaling factor λ to adjust the initial class weights of the CAW module.

[0078] Specifically, the formula for calculating the dynamic offset is:

[0079]

[0080] in, This represents a mode-specific dynamic offset; This represents a dynamic offset generator, consisting of two convolutional layers; These represent the differentially enhanced multi-scale feature maps of RGB and SAR modes obtained after processing by the MDPF module.

[0081] Specifically, in the category adaptive weighting module, the steps for obtaining the dynamic sparse mask for each target category are as follows: For each target category, calculate the number of elements K to be retained based on the current adaptive sparsity rate of the target category. Subsequently, in the modal weight map after the target category is adjusted, retain the top K spatial positions with the highest weight values, and set the remaining positions to zero to obtain the dynamic sparse mask for the target category.

[0082] Specifically, such as Figure 4 As shown, the processing flow of the final stage Category Adaptive Weighting (CAW) module is as follows:

[0083] Multi-scale feature maps with differential enhancement of different modalities are processed by a dynamic offset generator. , The dynamic offsets specific to each modality are calculated to correct spatial mismatches in the initial weights. Next, a Top-K global sparse mask is generated based on the current sparsity of the target class (the dynamic sparse mask of the target class). Figure 4 The RGB and SAR sparse weight maps for each target category are obtained by multiplying the sparse_mask_current by a dynamic offset. Further normalization is performed on these sparse RGB and SAR weight maps to ensure that the sum of the RGB and SAR weights at the same spatial location is 1, thus creating a balanced competition between modalities. The final output sparse RGB and SAR weight maps for each target category are used to guide cross-modal feature fusion, achieving refined feature selection through a target category-specific sparse attention mechanism.

[0084] Then initialize the class-specific weights of the CAW module. ( Figure 4 China and Israel W S1 W R1 , ..., W S7 W R7 As shown, S and R in the subscripts correspond to SAR mode and RGB mode, and 1-7 correspond to different target categories) and the mode-specific offset weighted by the learnable scaling factor. By adding elements one by one, we obtain the class-specific weights adjusted by the CAW module. ( Figure 4 middle , Mathematically, this is represented as:

[0085]

[0086] For each target category, the weight value of each spatial location of the target category on the differentially enhanced multi-scale feature map is calculated based on the current sparsity of the target category. The top K spatial locations with the highest weight values ​​are retained, and the remaining locations are set to zero to suppress redundant feature responses and enhance the contribution of key features. Simultaneously, the sparsity is updated based on performance feedback (F-score and IoU). For each target category, the change in performance metrics for each category is calculated and combined with a sensitivity coefficient to achieve dynamic adjustment of the sparsity of each category.

[0087] Furthermore, the adjusted weights of each target category in the CAW module are multiplied by the dynamic sparse mask to obtain sparse weight maps of each target category in the RGB and SAR modes. Normalization is then performed to ensure that the sum of the RGB and SAR weights at the same spatial location is 1, creating a balanced competition between modes. Finally, the sparse weight maps of each target category in the RGB and SAR modes are output (W in the figure).RGB and W SAR It is used to guide cross-modal feature fusion and achieves refined feature selection through a category-specific sparse attention mechanism.

[0088] During training, the sparsity rate is adaptively updated based on the segmentation performance of each class on the validation set. In this application, F-score and IoU (Intersection over Union) are used. For each target class, the change in accuracy (ΔW) is calculated. cross_RGB , △W cross_SAR By combining the sensitivity coefficients (Fscore_sensitivity, IoU_sensitivity), the sparsity rate and the sparsification weight map can be dynamically adjusted.

[0089] Secondly, the cross-modal collaboration module (CMC module) introduces category importance information from the CAW module. Through a semantically aware cross-attention mechanism, it achieves refined alignment and fusion of cross-modal features.

[0090] Specifically, such as Figure 5 The diagram shown is a schematic of the final layer cross-modal collaboration module and the feature integration layer.

[0091] The cross-modal collaboration module globally pools the sparse weight maps of each modality and target class output by the class adaptive weighting module to obtain a class importance vector. Based on the class importance vector, it fuses the multi-scale feature maps of each modality enhanced by differential enhancement using an attention mechanism to obtain a semantically enhanced cross-modal feature representation (attention map). The enhanced cross-modal feature representation is mapped to spatial-class attention weights through a class projection layer. After expanding the channel dimension, it is multiplied element-wise with the multi-scale feature maps of each modality enhanced by differential enhancement to obtain a feature map (W) that fuses the geometric structure enhancement and semantic features of each modality at multiple scales. cross_RGB W cross_SAR ).

[0092] Specifically, the CMC module performs differential enhancement on the multi-scale feature maps of the RGB modes. Projection as query matrix The output SAR mode differentially enhanced multi-scale feature map Projection as a key matrix The dual-modal splicing features [ , Projected into a value matrix through a convolutional layer , This indicates channel splicing. Then calculate... (Q) and The similarity matrix of (K) yields a cross-modal attention map, modeling modal complementarity. Simultaneously, the global average pooling result of the sparse weight map output by the CAW module is used. Introducing category importance information into the attention mechanism results in an enhanced cross-modal attention map.

[0093] The enhanced cross-modal attention map calculation process can be mathematically expressed as follows:

[0094]

[0095] In the formula, Let be the Energy matrix, representing the similarity between Q and K; Use the Sigmoid activation function; The query matrix is ​​a self-similarity matrix, representing the self-similarity within RGB modes; A represents the attention weights, obtained by normalizing the Energy matrix using the softmax function. This represents the semantically enhanced cross-modal feature representation.

[0096] Finally, the enhanced cross-modal features are achieved through a category projection layer. The mapping is done as a spatial-class attention weight map, which is used for subsequent segmentation.

[0097]

[0098] Where Conv is a convolutional layer; The sigmoid function outputs the class attention weights for each spatial location; W class For spatial-category attention weights.

[0099] Furthermore, the semantically weighted bimodal features (W) output by the final layer cross-modal collaborative module cross_RGB W cross_SAR The features are concatenated along the channel dimension and fused through a fusion convolutional layer. Finally, a channel restoration layer restores the number of channels of the fused features to the original input dimension, resulting in a feature map F that combines geometric enhancement and semantic feature fusion. DSCA As a feature map for multi-scale geometric enhancement and semantic feature fusion, it is used for subsequent decoding.

[0100] See details Figure 5 This stage uses the spatial-class attention weight map generated by the CMC module to broadcast and expand the channel dimension, making it compatible with the input feature map ( , The channel dimensions of the input feature map are matched, and then the attention weights with broadcast expansion are multiplied element-wise with the input feature map. The weighted bimodal features are concatenated along the channel dimension, and feature integration is achieved through a fusion convolutional layer. Finally, a channel restoration layer restores the number of channels of the fused features to the original input dimension, resulting in a feature map with cross-modal geometric enhancement and semantic feature fusion.

[0101]

[0102] Among them, F DSCA This represents the feature map output by the DSCA module, which represents the fusion of cross-modal geometric enhancement and semantic features. ⊙ represents a channel adjustment layer containing GroupNorm, GELU function, and 1×1 convolution; ⊙ represents element-wise multiplication.

[0103] In step 2-2, the decoder is built based on the UperNet architecture, effectively achieving spatial-semantic alignment and efficient information aggregation of heterogeneous data. The decoder takes the feature maps with geometric enhancement and semantic feature fusion output from the DSCA module in the encoder as input, corresponding to four stages of feature maps: Stage 0 (1 / 4 resolution), Stage 1 (1 / 8 resolution), Stage 2 (1 / 16 resolution), and Stage 3 (1 / 32 resolution). These feature maps have already undergone geometric enhancement and semantic feature fusion through the MDPF and DSCA modules.

[0104] The decoder consists of three parts:

[0105] A. Pyramid Pooling Module (PPM): This module introduces multi-scale pooling operations on the highest semantic-level features (Stage 3, 1 / 32 resolution) output by the feature encoder. It employs four different adaptive average pooling kernels (1×1, 2×2, 3×3, 6×6) to capture contextual information from local to global scales. Each pooling branch is upsampled to the original feature map space size through adaptive pooling, 1×1 convolution, and bilinear interpolation. Finally, the outputs of all branches are concatenated with the Stage 3 features output by the encoder along the channel dimension and fused through convolutional layers, significantly enhancing the model's ability to understand the global distribution patterns of various land features.

[0106] B. Feature Pyramid (FPN): This method fuses multi-scale features output from the encoder's DSCA module layer by layer through a top-down path and lateral connections. Specifically, starting with the highest-level Stage 3 features enhanced by PPM, the features are upsampled by 2x and added element-wise to the features output from the encoder's Stage 2. Then, a 3×3 convolution is performed to obtain the FPN's Stage 2 features, and so on, constructing a pyramid feature representation rich in semantic information. This design enhances semantic information while preserving spatial details.

[0107] C. Feature Fusion Layer: The four level features (Stage 0 to Stage 3) output by the FPN path are upsampled to the same spatial size (1 / 4 resolution of the input image), and then stitched together in the channel dimension and deep fusion of multi-scale features is achieved through a convolutional layer.

[0108] Finally, the fused feature map is restored to the input image size by 4x bilinear interpolation upsampling, generating a land cover classification result map with the same size as the input image.

[0109] This invention pertains to image processing technology, specifically a method for intelligently extracting land cover categories from multimodal remote sensing imagery. The invention proposes a multimodal remote sensing semantic segmentation model based on a category-aware asymmetric dynamic fusion network, enabling the network to output a generated image with the same size as the input image, achieving end-to-end operation.

[0110] In one specific embodiment, a computer was used to train the network, implemented using Python 3.8 on the Ubuntu operating system. The specific experimental environment configuration is shown in Table 2.

[0111] Table 2 Experimental Environment Configuration Table

[0112]

[0113] The construction and training process of the CADFNet-based multimodal remote sensing semantic segmentation model is as follows:

[0114] S1. Dataset Construction. Preprocess the two publicly available datasets. Train the CADFNet network using RGB-SAR image pairs and corresponding label data from the training set. The network input is a remote sensing image pair with a ground truth label, and the output is a land cover classification result of 256*256 (384*384) pixels.

[0115] This embodiment uses land cover classification as an example. The dataset used is a publicly available multimodal dataset containing RGB-SAR image pairs and corresponding label data; specifically, the images in the PIE-RGB-SAR dataset are cropped into 256×256 image patches, containing 3405 training images and 1460 test images. The images in the WHU-OPT-SAR dataset are cropped into 384×384 image patches, with 8820 images used for training and 3780 images used for validation. The pixel ratios of each category in the two datasets are shown below. Figure 6 As shown, the label types (target categories) include: Farmland, City, Village, Water, Forest, Road, and Others.

[0116] Due to GPU limitations, images in the WHU-OPT-SAR dataset are difficult to directly input into the network. To facilitate model training and testing while considering overall model performance, a sliding window method is used to crop the images in the dataset using a non-overlapping cutting technique.

[0117] The CADFNet-based multimodal remote sensing semantic segmentation model comprises an asymmetric feature encoder and a feature decoder. The input consists of RGB-SAR image pairs and their corresponding ground truth labels, while the decoder outputs land cover classification results. The encoder includes a multi-level differential pyramid fusion module (MDPF) and a dynamic sparse category attention module (DSCA). The latter is composed of a cascaded Category Adaptive Weighting Module (CAW) and a Cross-modal collaboration module (CMC).

[0118] Furthermore, the steps for constructing a multimodal remote sensing semantic segmentation model include:

[0119] (1) Design a network structure for a non-shared weight dual-branch encoder and decoder to achieve end-to-end operation;

[0120] (2) Design a multi-level differential pyramid fusion module (MDPF) that uses asymmetric pyramid pooling combined with multi-scale differential computation to explicitly model and capture cross-modal geometric and semantic differences, making full use of the independent features of each modality.

[0121] Specifically, in the multimodal remote sensing semantic segmentation model, the multi-scale differential features of each branch are calculated in parallel and interactively through a multi-level differential pyramid fusion module. This multi-level mining of differential features between modalities eliminates low-level noise between modalities and achieves dynamic fusion, thereby providing a more discriminative feature foundation for the subsequent DSCA module.

[0122] (3) In the multimodal remote sensing semantic segmentation model, the dynamic sparse category attention module can dynamically calibrate and more effectively capture complementary information between multiple modalities through the cascaded channel-space attention mechanism, realizing the progressive fusion from local feature calibration to global relationship modeling, and significantly improving the accuracy of land cover classification.

[0123] (3-1) Design a category-adaptive dynamic weight allocation module (CAW). Specifically, this module explicitly models the correlation between the input feature channels and each category, and enhances the few-sample feature representation capability by learning class-level information and generating dynamic attention weights for each category.

[0124] Specifically, in the class adaptive weighting module of the multimodal remote sensing semantic segmentation model, class-level dynamic weights are generated by adopting independent sparsity rates for different classes and updating the sparsity rates as the network is trained, so as to enhance the feature representation of small sample classes and effectively alleviate the class imbalance problem in semantic segmentation tasks.

[0125] (3-2) Fine alignment of key regions between modalities is achieved through the cross-modal collaboration module (CMC). Specifically, cross-modal complementary regions are strengthened by dynamically calculating spatial dependencies between modalities through cross-attention.

[0126] Specifically, in the multimodal remote sensing semantic segmentation model, the cross-modal collaborative module establishes intermodal feature associations through a cross-attention mechanism, generates category-aware attention weights, and enhances the interaction of modal features for specific categories. At the same time, it effectively captures long-distance dependencies and avoids limitations imposed by local receptive fields.

[0127] Step 4: Training the multimodal remote sensing semantic segmentation model. The CADFNet model is trained using RGB-SAR image pairs and corresponding land cover annotation data from the training set, and the trained model parameters are saved.

[0128] Regarding the loss function, due to the severe class imbalance problem in the dataset (e.g.) Figure 5 As shown, the multimodal remote sensing semantic segmentation model uses a joint loss function composed of a weighted average of the cross-entropy loss function, the Lovasz loss function, and the Dice loss function to calculate the model loss.

[0129] Using RGB-SAR image pairs and corresponding land cover labels from the test set, the model parameters are updated by calculating the loss function, and the process is iterated multiple times until a land cover result with smooth and complete boundaries is extracted.

[0130] Land cover classification results of each model on the PIE-RGB-SAR dataset are as follows: Figure 7 As shown. From left to right, the results are: SAR image, RGB image, ground truth label, ASMFNet result, MCANet result, SOLC result, and CADFNet result. In terms of subjective visual perception, these visualization results strongly demonstrate the superior performance of the method of this invention. Through comparison... Figure 7 As can be seen from groups a and d, compared with the other three representative methods (ASMFNet, MCANet, SOLC), this invention better combines the advantages of SAR images and exhibits better integrity and accuracy in water body categories. Figure 7 Group b shows that SOLC and the method of this invention perform best overall. However, the method of this invention is more complete in extracting small sample categories such as roads. Figure 7 In group C, although the method of this invention misclassifies some roads as urban areas, it extracts vegetation in farmland and between roads more accurately than other methods.

[0131] In terms of objective evaluation, the models were evaluated and analyzed by calculating four accuracy evaluation indicators: Intersection over Union (IoU) for each category, Overall Accuracy (OA), Kappa coefficient, and Mean Intersection over Union (MIoU). The results are shown in Table 3. Table 3: Quantitative evaluation results of each model on the PIE-RGB-SAR dataset.

[0132]

[0133] Experimental data shows that, compared with five other multimodal algorithms, this invention outperforms other methods in all overall evaluation metrics (mIoU=75.86%, Kappa=83.84%, OA=88.51%). This confirms that the proposed multi-level differential pyramid fusion module and category-guided dynamic sparse attention module effectively capture complementary information between modalities and successfully enhance few-sample features. Compared with MCANet, DDHRNet, SOLC, FTransUNet, and ASMFNet, the method of this invention achieves significant improvements in the mIoU metric, increasing by 4.26%, 9.12%, 3.13%, 5.68%, and 4.46%, respectively. Among them, DDHRNet has lower overall performance, with an OA of 85.14% and an mIoU of only 66.74%.

[0134] Table 3 also shows the accuracy of different methods based on the IoU metric across various categories. The method of this invention achieved the highest IoU value in all categories except "other". Compared to the second-highest accuracy network SOLC, the method of this invention outperforms in multiple categories, improving accuracy by 0.91%, 11.72%, 0.87%, 1.53%, and 3.84% in the urban, road, water, forest, and farmland categories, respectively. Notably, ASMFNet and the method of this invention perform particularly well in the farmland category.

[0135] The land cover classification results of each model on the WHU-OPT-SAR dataset are as follows: Figure 8 As shown in the diagram, these visualizations demonstrate that the method of the present invention achieves optimal overall results. Figure 8 Group A shows that the method of the present invention has the ability to resist noise in SAR images, thus effectively distinguishing forests from farmland. Figure 8 Groups a and b show that the water bodies extracted by the method of the present invention more closely resemble the actual ground conditions. Notably, the method of the present invention exhibits good performance in identifying road features and demonstrates better connectivity and integrity. Figure 8 Groups b, c, and d). Figure 8 The box above group C highlights the effective use of SAR information by the method of this invention, which solves the misclassification problem in RGB images and extracts relatively completely the roads not marked in the labels and the water body between the roads in the upper left corner. The visualization results emphasize the effectiveness of the category-aware dynamic fusion mechanism in the method of this invention, that is, it can adaptively select the dominant modality for feature learning according to different categories, and use the information of another modality for supplementation and enhancement, thereby achieving accurate feature fusion.

[0136] In terms of objective evaluation, the same evaluation indicators were used to evaluate and analyze the model, as shown in Table 4.

[0137] Table 4. Quantitative evaluation results of each model on the WHU-OPT-SAR dataset.

[0138]

[0139] As shown in Table 4, the method of this invention achieves the best overall performance, with mIoU scores exceeding those of MCANet, DDHRNet, SOLC, FTransUNet, and ASMFNet by 6.33%, 15.32%, 14.87%, 13.34%, and 2.88%, respectively. These results highlight the superiority of the method of this invention across various metrics on the WHU-OPT-SAR dataset. Compared to other state-of-the-art methods, the method of this invention exhibits superior segmentation accuracy and class sensitivity across all categories. Particularly in small-sample categories such as "road" and "other," the method of this invention achieves IoU scores of 41.31% and 24.64%, respectively, outperforming the second-place method by 9.96% and 6.59%. This demonstrates that the present invention effectively improves learning performance for categories with fewer samples.

[0140] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a multimodal land cover classification system based on a category-aware asymmetric dynamic fusion network. This system is used to execute a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network from the above method embodiments.

[0141] See Figure 9 The system includes:

[0142] The input module is used to acquire remote sensing optical-SAR image pairs of land cover in the study area;

[0143] The land cover classification module is used to input remote sensing optical-SAR image pairs of land cover in the study area into the trained multimodal remote sensing semantic segmentation model, and output the classification results of land cover in the study area.

[0144] The training process of the multimodal remote sensing semantic segmentation model includes:

[0145] Obtain the training dataset for the multimodal remote sensing semantic segmentation model;

[0146] A multimodal remote sensing semantic segmentation model based on CADFNet is constructed: In the encoder part, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the multi-level differential pyramid fusion module performs cross-modal differential feature enhancement on the multi-scale feature maps of each modality extracted in the current stage. Then, the dynamic sparse class attention module performs adaptive feature fusion and alignment based on the semantic information of the target class to obtain feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales. Through the feature integration layer, the feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales output by the encoder part are stitched together and the number of channels is restored to the original input dimension. Then, they are sent to the decoder part, and the decoder part decodes to obtain the land cover classification result. The multimodal remote sensing semantic segmentation model is trained using the training dataset, and the trained multimodal remote sensing semantic segmentation model is output.

[0147] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0148] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 10 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0149] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0150] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network, characterized in that, include: Acquire remote sensing optical-SAR image pairs of land cover in the study area; The remote sensing optical-SAR image pairs of land cover in the study area are input into the trained multimodal remote sensing semantic segmentation model, and the classification results of land cover in the study area are output. The training process of the multimodal remote sensing semantic segmentation model includes: Obtain the training dataset for the multimodal remote sensing semantic segmentation model; A multimodal remote sensing semantic segmentation model based on CADFNet is constructed: In the encoder part, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the multi-level differential pyramid fusion module performs cross-modal differential feature enhancement on the multi-scale feature maps of each modality extracted in the current stage. Then, the dynamic sparse class attention module performs adaptive feature fusion and alignment based on the semantic information of the target class to obtain feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales. Through the feature integration layer, the feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales output by the encoder part are stitched together and the number of channels is restored to the original input dimension. Then, they are sent to the decoder part, and the decoder part decodes to obtain the land cover classification result. The multimodal remote sensing semantic segmentation model is trained using the training dataset, and the trained multimodal remote sensing semantic segmentation model is output.

2. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 1, characterized in that, The multi-level differential pyramid fusion module takes the multi-scale feature maps of RGB and SAR modes as the first input, and processes them by combining pyramid pooling strategy with bidirectional pixel-wise subtraction operation to obtain the multi-scale differential feature maps of each mode; the differential convolution module processes the multi-scale differential feature maps of each mode respectively, and outputs the weight map of each mode; the gate generator processes the spliced ​​features of the first input based on the gate mechanism to generate the channel adaptive fusion weights of each mode. The feature fusion part outputs multi-scale feature maps of different modalities with differential enhancement.

3. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 2, characterized in that... The process of obtaining multi-scale difference feature maps by combining pyramid pooling with bidirectional pixel-wise subtraction includes: The first input is processed using a pyramid pooling strategy, including: multi-scale max pooling of the feature map of the RGB mode and multi-scale average pooling of the feature map of the SAR mode, and multiplying them by adaptive scale weights to balance the contributions of feature maps at different scales. Following the rule of the same spatial location and scale, pixel-by-pixel subtraction is performed on each modality. The feature map corresponding to the modality in the first input is subtracted from the feature map of another modality processed by the pyramid pooling strategy to obtain the multi-scale difference feature map of each modality.

4. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 2, characterized in that, The feature fusion section multiplies the weight map of each modality with the channel adaptive fusion weight to obtain the weight map of each modality channel weighted. The weight map of each modality channel weighted is then multiplied with the learnable scaling coefficient, and then connected with the residual of the multi-scale feature map of each modality originally input by the multi-level differential pyramid fusion module through element-wise addition operation to obtain the differentially enhanced multi-scale feature map of each modality.

5. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 1, characterized in that, The dynamic sparse category attention module adopts a channel-space attention mechanism composed of a category adaptive weighting module and a cross-modal collaborative module.

6. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 5, characterized in that, The category-adaptive weighting module processes the differentially enhanced multi-scale feature maps of each modality through a dynamic offset generator to obtain the dynamic offset of each modality; the initial category-specific weights of the category-adaptive weighting module are added element by element to each dynamic offset after being weighted by the learnable scaling coefficients to obtain the category-specific weights adjusted by the category-adaptive weighting module. The category-specific weights adjusted by the category adaptive weighting module are multiplied by the dynamic sparse mask of each target category, and then normalized to obtain the sparse weight map of each modality and each target category.

7. The multimodal land cover classification method based on a category-aware asymmetric dynamic fusion network as described in claim 5, characterized in that, The cross-modal collaboration module averages the sparse weight maps of each modality and each target category output by the category adaptive weighting module, then performs global pooling to obtain a category importance vector, and fuses the differentially enhanced multi-scale feature maps of each modality based on the category importance vector using an attention mechanism to obtain a semantically enhanced cross-modal attention map. The semantically enhanced cross-modal attention map is mapped to a spatial-class attention weight map through a category projection layer. After expanding the channel dimension, it is multiplied element-wise with the differentially enhanced multi-scale feature maps of each modality to obtain feature maps of geometric structure enhancement and semantic feature fusion at each modality multi-scale.

8. A multimodal land cover classification system based on a category-aware asymmetric dynamic fusion network, characterized in that, include: The input module is used to acquire remote sensing optical-SAR image pairs of land cover in the study area; The land cover classification module is used to input remote sensing optical-SAR image pairs of land cover in the study area into the trained multimodal remote sensing semantic segmentation model, and output the classification results of land cover in the study area. The training process of the multimodal remote sensing semantic segmentation model includes: Obtain the training dataset for the multimodal remote sensing semantic segmentation model; A multimodal remote sensing semantic segmentation model based on CADFNet is constructed: In the encoder part, the feature extraction layer uses two independent non-shared weight branches to process the optical image and SAR image in the remote sensing optical-SAR image pair respectively, extracting multi-scale feature maps of each modality in stages. After each stage, the multi-level differential pyramid fusion module performs cross-modal differential feature enhancement on the multi-scale feature maps of each modality extracted in the current stage. Then, the dynamic sparse class attention module performs adaptive feature fusion and alignment based on the semantic information of the target class to obtain feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales. Through the feature integration layer, the feature maps of geometric structure enhancement and semantic feature fusion of each modality at multiple scales output by the encoder part are stitched together and the number of channels is restored to the original input dimension. Then, they are sent to the decoder part, and the decoder part decodes to obtain the land cover classification result. The model is trained using the training dataset, and the trained multimodal remote sensing semantic segmentation model is output.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 7.

Images