An edge-guided cross-modal interaction fusion method for land cover classification

Abstract

The present application relates to a kind of edge-guided cross-modal interaction fusion land cover classification method, especially applicable to the collaborative semantic segmentation of optical image and synthetic aperture radar (SAR) image.The method includes constructing pseudo-twin multi-flow encoder to extract multiscale features of optical and SAR images, and embedding cross-modal adaptive feature interaction module between encoding stages, realizing dynamic alignment and re-labeling of features between modalities through two-level interaction mechanism of channel level and spatial level, and relieving fusion deviation caused by heterogeneity and spatial misplacement.Furthermore, through the light feature fusion module, neuron-level attention and multi-scale depth separable convolution are fused to suppress noise and aggregate context information.Meanwhile, an edge auxiliary module is introduced to extract multi-scale boundary information from low-level features and jointly decode with semantic features to improve boundary details and small target recognition capability.The present application significantly improves the precision and robustness of land cover segmentation in complex scenes while ensuring computational efficiency.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision, specifically to an edge-guided cross-modal interactive fusion land cover classification method. Background Technology

[0002] Multimodal remote sensing semantic segmentation is a key technology for land cover mapping and urban planning. Optical images have rich spectral textures, while SAR images have all-weather imaging capabilities; their fusion can complement each other's advantages. However, due to differences in imaging mechanisms and observation geometry, cross-modal features exhibit significant heterogeneity and spatial misalignment, making direct fusion prone to introducing noise and limiting the improvement of segmentation accuracy.

[0003] Traditional methods rely on manually generated features and classifiers, which have limited expressive power and struggle to handle complex scenarios. While deep learning methods have become mainstream, existing dual-branch architectures still have shortcomings: simple fusion strategies struggle to uncover deep semantic relationships; cross-attention-based methods, while able to suppress noise, suffer from excessive computational complexity and are prone to attenuating local details. CNNs are insufficient at modeling long-range dependencies, and while Transformers can capture global context, they pay insufficient attention to edges and small objects, resulting in significant computational overhead at high resolutions.

[0004] The current core challenges are: modal heterogeneity and spatial misalignment leading to feature inconsistency; difficulty in balancing fusion accuracy and computational efficiency; and insufficient utilization of edge information resulting in blurred boundaries and difficulty in identifying small targets. Therefore, there is an urgent need for a multimodal semantic segmentation method that balances accuracy and efficiency and can effectively address the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide an edge-guided, cross-modal interactive fusion land cover classification method to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution.

[0007] An edge-guided cross-modal interactive fusion land cover classification method includes the following steps: S1, acquiring optical images and synthetic aperture radar (SAR) images of the same area to construct an optical-SAR multimodal remote sensing image dataset; performing unified spatial reference and pixel alignment on the optical images and SAR images, cropping the remote sensing images and labels using the same non-overlapping step size, and performing data augmentation through random rotation, random flipping, and random scaling.

[0008] S2. Construct an edge-guided multimodal semantic segmentation network; the network includes an optical branch encoder and a SAR branch encoder, a cross-modal adaptive feature interaction module (MAFI), a lightweight feature fusion module (LFF), an edge-assisted module (EA), and a decoder; wherein MAFI performs dynamic reweighting and residual correction on optical features and SAR features at multiple scales to suppress modal feature distribution shifts and local spatial misalignments caused by differences in imaging mechanisms; EA extracts boundary priors from shallow detail features and performs edge-guided refinement on the fused semantic features.

[0009] S3. Use the dataset constructed in step S1 to train and validate the network described in step S2. Through backpropagation joint optimization, the semantic segmentation output and the edge guidance output converge collaboratively to obtain the trained semantic segmentation model.

[0010] S4. Acquire the optical and SAR image pairs to be segmented and perform preprocessing and synchronous block segmentation consistent with the training phase. Input the segmented image pairs into the trained semantic segmentation model, output pixel-level land cover classification results, and seamlessly stitch the segmented results according to the cropping rules to obtain a complete classification map.

[0011] As a further aspect of the present invention: step S1 includes the following sub-steps: S11, screening optical images and SAR images that meet the quality standards, completing unified resolution processing and strict spatial registration, so that the two modes achieve corresponding points.

[0012] S12. Perform pixel-level annotation on the registered image pairs to obtain land feature category labels, and use the Canny operator to generate boundary labels.

[0013] S13. Perform simultaneous cropping on optical and SAR images to obtain sub-image block pairs with consistent size, and use filling or discarding strategies for boundary regions with insufficient size to ensure consistent sample scale.

[0014] S14. Organize the image pair files and corresponding change label files to construct a multimodal remote sensing image semantic segmentation standard dataset containing training and test sets.

[0015] As a further aspect of the present invention: the input to the edge-guided multimodal semantic segmentation network in step S2 is a cropped optical image and a SAR image of the same size. The optical image is input into the semantic segmentation network and the edge branch, and the SAR image is input into the semantic segmentation network. The semantic segmentation network includes two sets of independent encoders for feature extraction. Each encoder consists of four bidirectional self-attention blocks, and each bidirectional self-attention block includes an embedding layer and a two-layer routing attention module. The two-layer routing attention module divides the input features into two-dimensional windows of m×m and performs automatic filling and output cropping restoration when the size is not divisible. It performs QKV linear projection on the windowed features and performs two-dimensional downsampling on the key values ​​to control complexity. The window-level segmentation unit routing filters the relevant window indexes and weights of the segmentation unit according to the average features of the window, and calculates multi-head attention on the key value set of the filtered window at the pixel level to achieve cross-window sparse global interaction. At the same time, a depthwise separable convolutional local enhancement term is introduced to perform residual compensation on the attention output. Finally, the output features of the same size as the input are used for subsequent feature input and decoding output.

[0016] As a further aspect of the present invention: the edge-guided multimodal semantic segmentation network further includes an edge branch, which uses ResNet-18 as the backbone encoder and contains n residual convolutional blocks; the edge branch takes an optical image as input, and performs initial feature extraction sequentially through convolutional layers, batch normalization layers, and nonlinear activation layers, and performs stepwise downsampling encoding through the n residual convolutional blocks to obtain multi-scale edge representations; wherein each residual convolutional block includes at least two layers of convolution operations and identity residual connections, the residual connections are used to perform residual addition between the input features and the convolution output to alleviate gradient vanishing and enhance detail preservation capabilities; when the feature dimension or resolution of adjacent scales changes, the downsampling branch is used to perform channel matching and scale alignment on the input features; the edge branch outputs edge information after the features pass through the edge assistance module, the edge information including edge feature maps and edge probability maps, and injects the edge information into the corresponding scale semantic features of the semantic segmentation decoder through upsampling and feature fusion operations to enhance the boundary continuity, clarity, and localization accuracy of the segmentation results.

[0017] As a further aspect of the present invention: the cross-modal adaptive feature interaction module (MAFI) includes: a channel interaction unit: performing global average pooling, global max pooling, and global standard deviation pooling on optical features and SAR features respectively to obtain a global description of multiple statistics, and generating corresponding channel attention weights through nonlinear mapping of the global descriptions; the channel attention weights dynamically reweight the features of another modality in a cross-modal interaction manner and update the channel interaction features using residuals, thereby strengthening the complementary features of the two modalities and suppressing modal noise interference without introducing global attention.

[0018] Spatial Interaction Unit: Based on the channel interaction features, further extract fusion spatial saliency information, generate spatial attention weights that include spatial location and channel semantic coupling, and introduce trainable scaling factors to adaptively adjust the cross-modal spatial correction intensity, so as to achieve pixel-level alignment correction of cross-modal local misalignment and boundary uncertainty areas, thereby significantly reducing the impact of imaging mechanism differences and geometric offset on subsequent fusion and decoding segmentation.

[0019] As a further aspect of the present invention: the lightweight feature fusion module (LFF) includes: original feature enhancement path: introducing a neuronal attention mechanism before and after fusion to suppress unstable noise responses in optical and SAR features, so as to improve the reliability of cross-modal fusion input.

[0020] Multi-scale feature extraction path: After concatenating the features of two modalities, four sets of depthwise separable convolutions with different scales of 5, 7, 9, and 11 are used to extract cross-modal contextual information in parallel, so as to cover the scale differences of remote sensing targets and keep the computational cost controllable.

[0021] Fusion and reconstruction path: Multi-scale features are aggregated and reconstructed through pointwise convolution and standard convolution, and the global representation is further weighted by the weight maps generated by the two modalities. This achieves the fusion of global complementary information and modality discrimination information under low complexity conditions, improving the ability to preserve details and the robustness against noise in high-resolution scenes.

[0022] As a further aspect of the present invention: the edge assist module (EA) upsamples and stitches the multi-scale shallow features of the edge branches to form a multi-scale boundary representation, and adaptively integrates local edge details with global semantic information through the edge and semantic fusion unit, so that the ground feature boundary and complex background can be more clearly separated; the edge feature map and edge probability map output by the edge assist module are fused with the corresponding scale features of the semantic decoder to provide stable boundary constraints for small targets, slender targets and similar border areas, thereby further improving the continuity of the segmentation contour and the accuracy of boundary positioning.

[0023] As a further aspect of the present invention: step S3 includes the following sub-steps: S31, setting the parameter values ​​required for network training according to the multimodal remote sensing image semantic segmentation dataset, mainly including the initial learning rate, optimizer and loss function, maximum number of training iterations, etc.; the loss function adopts a comprehensive loss function that combines edge and semantic loss, wherein the edge loss adopts dice loss to avoid generating excessively thick edges, and the semantic loss adopts multi-class loss and dice loss, which improves the inter-class discriminability while alleviating class non-smoothness.

[0024] S32. Input the training dataset into the edge-guided cross-modal fusion network for training, and validate it on the validation dataset. During training, update the network parameters through backpropagation so that the semantic segmentation output and the edge output converge simultaneously, thereby achieving the best learning effect. After the network training is completed, save the network parameters that perform best on the validation set for model testing.

[0025] As a further aspect of the present invention: step S4 includes the following sub-steps: S41, acquiring optical and SAR image pairs to be processed, wherein the spatial resolution and number of channels of the image pairs are consistent with the training data; and synchronously cropping the image pairs according to the input size set in step S2 to obtain several pairs of sub-image blocks; and sorting them, and filling in the sub-image blocks that are not large enough; finally forming a multimodal remote sensing image test dataset.

[0026] S42. Input the paired images from the multimodal remote sensing image test dataset into the trained edge-guided cross-modal fusion network to obtain the land cover classification result, i.e., the pixel-level segmentation result of the multimodal remote sensing image test dataset.

[0027] S43. The detection results of the multimodal remote sensing image test dataset are stitched together according to the cropping method in step S41, and the parts of the images with insufficient size in step S41 that are filled in are deleted, and finally the complete land cover classification results are obtained.

[0028] Compared with the prior art, the beneficial effects of the present invention are: 1. Cross-modal interaction is more accurate and efficient: By embedding cross-modal adaptive feature interaction modules (MAFI) between each stage of the encoder, and adopting a two-stage serial interaction mechanism of channel and space, dynamic alignment and recalibration of optical and SAR features at multiple scales are realized, which fundamentally alleviates the problem of modal heterogeneity and local spatial misalignment caused by differences in imaging mechanisms, and improves the complementarity and consistency of feature expression.

[0029] 2. Strong noise resistance and low computational overhead in the fusion process: The Lightweight Feature Fusion Module (LFF) combines neuron-level attention mechanism with multi-scale deep separable convolution. While effectively suppressing optical salt-and-pepper noise and SAR speckle noise, it achieves full aggregation across modal contexts. This module significantly reduces the number of model parameters and inference computation burden while ensuring fusion accuracy, achieving synergistic optimization of accuracy and efficiency.

[0030] 3. Significantly improved boundary detail preservation and segmentation continuity: An independent edge assist module (EA) is introduced to extract multi-scale edge information from low-level optical features and inject it into the decoding process through an edge-semantic fusion mechanism. This effectively enhances the recognition ability of small targets, slender features and category boundary areas, and significantly improves the continuity, clarity and positioning accuracy of segmentation boundaries.

[0031] 4. Strong adaptability and high robustness: The method of this invention exhibits stable segmentation performance in complex terrain scenes, coexistence of multi-scale targets, and heterogeneous sensor conditions. Experiments on public datasets show that it outperforms existing mainstream methods in terms of overall accuracy, average intersection-union ratio, and class specificity, and has good generalization ability and practical application value.

[0032] 5. Supports end-to-end training and efficient inference: The entire network structure is clear and highly modular, supporting end-to-end joint training. During training, semantic segmentation and edge output are co-optimized, avoiding error accumulation caused by multi-stage training. The inference stage supports image block processing and seamless stitching, making it suitable for efficient processing of large-scale remote sensing images. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method provided in an embodiment of the present invention.

[0034] Figure 2 This is a diagram of the edge-guided cross-modal interactive fusion network structure provided in an embodiment of the present invention.

[0035] Figure 3 This is a MAFI structure diagram provided in an embodiment of the present invention.

[0036] Figure 4 This is a structural diagram of the LFF module provided in an embodiment of the present invention.

[0037] Figure 5 This is a structural diagram of EA provided in an embodiment of the present invention.

[0038] Figure 6This is a comparison chart of land cover classification visualization results provided in an embodiment of the present invention. In the chart: (a1), (a2), (a3), and (a4) are optical images; (b1), (b2), (b3), and (b4) are SAR images; (c1), (c2), (c3), and (c4) are ground truth labels (GT); (d1), (d2), (d3), and (d4) are segmentation results using the SA-Gate method; (e1), (e2), (e3), and (e4) are segmentation results using the VFuseNet method; (f1), (f2), and (f3) are segmentation results using the VFuseNet method. (f4) represents the segmentation results of the CFNet method; (g1), (g2), (g3) and g4 represent the segmentation results of the MCANet method; (h1), (h2), (h3) and (h4) represent the segmentation results of the CMGFNet method; (i1), (i2), (i3) and (i4) represent the segmentation results of the CMX method; (j1), (j2), (j3) and (j4) represent the segmentation results of the AFNet method; (k1), (k2), (k3) and (k4) represent the segmentation results of the CatNet method; (l1), (l2), (l3) and (l4) represent the segmentation results of the GBFN method; (m1), (m2), (m3) and (m4) represent the segmentation results of the FtransUnet method; (n1), (n2), (n3) and (n4) represent the segmentation results of the method of the present invention. Detailed Implementation

[0039] The technical solution of this application will be further described in detail below with reference to specific embodiments.

[0040] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0041] Please see Figure 1 In one embodiment of the present invention, an edge-guided cross-modal interactive fusion land cover classification method includes the following steps: Step 1: Acquire optical images and synthetic aperture radar (SAR) images of the same area to construct an optical-SAR multimodal remote sensing image dataset; perform unified spatial reference and pixel alignment on the optical images and SAR images, crop the remote sensing images and labels using the same non-overlapping step size, and perform data augmentation through random rotation, random flipping, and random scaling.

[0042] Specifically, step 1 includes the following sub-steps: Step 1.1: Acquire optical images and SAR images of acceptable quality and complete unified spatial resolution processing.

[0043] Step 1.2: Perform pixel-level annotations based on optical and SAR images to generate land cover category label maps corresponding to the images.

[0044] Step 1.3: Perform synchronous cropping processing on the optical image and SAR image pairs to obtain several pairs of sub-image blocks with consistent length and width; discard image blocks whose length and width are less than the preset input size.

[0045] Step 1.4: Organize and filter the sub-image pairs and corresponding label files to construct a multimodal remote sensing image semantic segmentation standard dataset containing training and validation sets. At the same time, multi-scale data augmentation operations are performed using random rotation, random flipping, and random scaling of 0.5, 0.75, 1, 1.25, 1.5, and 1.75 times.

[0046] Step 2: Construct an edge-guided cross-modal interaction fusion network.

[0047] Specifically, step 2 includes the following sub-steps: Step 2.1: The edge-guided cross-modal interactive fusion network designed in this invention adopts an encoder-decoder structure, such as... Figure 2 As shown, the encoder uses two branches with non-shared weights to extract multi-level features from optical and SAR images. The optical and SAR images used contain 4 / 3 channels and 1 channel, respectively, with sub-image sizes of 512×512 or 256×256. For different optical and SAR input channels, the first layer weights of the network's pre-trained weights are modified to correspond to the input. Each bidirectional self-attention block includes an embedding layer and a two-layer routing attention module; the two-layer routing attention module divides the input features into two-dimensional windows of m×m and performs automatic filling and output cropping restoration when the size is not divisible. It performs QKV linear projection on the windowed features and performs two-dimensional downsampling on the key values ​​to control complexity. Through window-level segmentation unit routing, it filters the relevant window indices and weights of the segmentation units based on the average features of the windows, and calculates multi-head attention on the key value set of the filtered windows at the pixel level to achieve cross-window sparse global interaction. At the same time, it introduces a depthwise separable convolutional local enhancement term to perform residual compensation on the attention output. The encoder has four stages with channel settings of 64, 128, 256, and 512 respectively. The number of channels in the optical and SAR feature maps output by the encoder also corresponds one-to-one with the number of channels in the four scales of the encoder.

[0048] Between two adjacent stages in the backbone network, MAFI utilizes spatial and channel interactions for cross-modal information exchange, and then feeds the multi-level features resulting from feature interaction into LFF for cross-modal feature fusion. The edge branches employ ResNet18 to extract multi-scale edge features from optical images and use EA to aggregate multi-level edge information, generating edge detail information and boundary contour maps. Finally, the fused multi-level semantic features and edge detail information are input together into a multi-level decoder, and the final result is generated through progressive upsampling and feature aggregation. Before training, the network input needs to be set, with the input and output dimensions of the multi-branch fusion network determined based on the length, width, and number of channels of the images in the optical and SAR datasets.

[0049] To enhance the consistency of spectral characteristics and geometric positions of the same region across different modalities in a scene, this invention utilizes channel and spatial self-attention mechanisms and adaptive parameters to design MAFI, such as... Figure 3 As shown, this module's progressive, multi-interaction mechanism enables cross-modal interaction, thereby utilizing complementary cross-modal information and enhancing intermodal correlation. The MAFI operation formula is as follows.

[0050] 1) In the channel interaction stage, features are received from two different branches of the two-stream semantic segmentation encoder, with each set of features containing four different scales. One-dimensional global average pooling, global standard deviation pooling, and global max pooling are performed along the spatial dimension during the channel interaction stage. The number of channels in the compressed feature map remains unchanged, and its size is compressed to 1×1. It is then multiplied with the input features and summed through residual connections. The size of the output feature after channel interaction remains consistent with the input feature size.

[0051]

[0052] The MLP, as shown in equation (2), contains two linear layers and a ReLU activation function; This represents a linear layer. Represents input and output channels. Represents the ReLU activation function. , , These refer to global average pooling, global max pooling, and global standard deviation pooling, respectively. This represents the Sigmoid activation function, F. OPT and F SAR These represent the original feature maps extracted by the optical branch and the SAR branch at the current layer, respectively. , These are learnable parameters, initialized to 0, and adjusted during backpropagation to control the degree of interaction between different modalities in order to achieve the optimal fusion ratio.

[0053] 2) Spatial Interaction Stage: This stage receives features from two different branches of channel interaction, each containing four different scales. During the channel interaction stage, one-dimensional global average pooling, global standard deviation pooling, and global max pooling are performed along the channel dimension. The compressed feature map spatial size remains unchanged, and the number of channels is compressed to 1. It is then multiplied with the input features and summed through residual connections. The output feature size after channel interaction remains consistent with the input feature size.

[0054]

[0055] in, , , These refer to global average pooling, global max pooling, and global standard deviation pooling, respectively. Represents a 7×7 convolution. This represents the Sigmoid activation function. Wherein... , These are learnable parameters, initialized to 0, and adjusted during backpropagation to control the degree of interaction between different modalities in order to achieve the optimal fusion ratio.

[0056] To achieve noise suppression and efficient multimodal information fusion, this invention designs an LFF module that combines neuronal attention and multi-scale feature extraction, such as... Figure 4 As shown, this module captures a wider range of information by suppressing noise in the original image and employing depthwise separable convolutions at different scales, thereby fully fusing multimodal features. The multi-scale features include four different scales: 5, 7, 9, and 11, with the corresponding input and output channel numbers remaining unchanged. These features are then aggregated through pointwise convolutions of size 1×1 with a stride of 1. The LFF module operation formula is as follows.

[0057]

[0058] In the formula, k represents the kernel size. This represents a 1×1 PW convolution. The subscript indicates the size of the convolution kernel used. This represents the SimAM operation. BN represents the normalization operation.

[0059] To effectively capture small, dense targets and complex boundaries in remote sensing images, this invention designs an edge-aiding module EA (e.g., based on multi-scale edge information and an EAM weighting mechanism). Figure 5(As shown). EA receives multi-scale edge features of 1 / 4, 1 / 8, and 1 / 16 scale from the edge branch encoder of the original image. After upsampling and aligning each scale feature to the 1 / 4 scale, it is fused. EAM weights the edge contributions at each scale, enhancing clear and stable ground truth boundaries. Furthermore, the multi-scale weighted edge features are input into a 3×3 convolution with a stride of 1, maintaining the original resolution to obtain a global representation. One-dimensional average pooling is then performed along the X and Y directions to obtain local features in those directions. Finally, cross-attention multiplication yields the final edge refinement features. Additionally, the edge refinement features are passed through a 3×3 convolution with the required number of classes in the output channel to output an edge probability map. This allows EA to provide more accurate boundary cues for slender targets and class boundaries, and the obtained edge features are fused with the semantic features of the decoder as constraint information, thereby improving the boundary continuity and localization accuracy of the segmentation results. The EA operation formula is as follows.

[0060]

[0061] In the formula, This means upsampling the feature map to a size of H×W. This involves dividing the feature map into equal-sized segments along the channel dimension. "Group" represents a feature grouping operation. and This represents average pooling in the X direction and average pooling in the Y direction. Gn is group normalization. This represents element-wise multiplication. This represents the normalization function. `Concat` represents concatenation along the channel dimension. `Conv()` represents the convolution operation, with the index indicating the kernel size.

[0062] Step 2.2: Based on Step 2.1, set the parameter values ​​required for network training according to the optical and SAR land cover classification training datasets, mainly including optimizer, learning rate strategy, training epochs and loss function.

[0063] Step 3: Train and optimize the edge-guided cross-modal interaction fusion network using the land cover classification dataset.

[0064] Specifically, step 3 includes the following sub-steps: Step 3.1: Set the parameter values ​​required for network training according to the multimodal remote sensing image semantic segmentation dataset, mainly including the initial learning rate, optimizer and loss function, maximum number of training iterations, etc.; the loss function adopts a comprehensive loss function that combines edge and semantic loss, wherein the edge loss adopts dice loss to avoid generating excessively thick edges, and the semantic loss adopts multi-class loss and dice loss, which improves the inter-class discriminability while alleviating class non-smoothness.

[0065] Step 3.2: Input the training dataset into the edge-guided cross-modal fusion network for training, and validate it on the validation dataset; during training, update the network parameters through backpropagation so that the semantic segmentation output and edge output converge simultaneously, thereby achieving the best learning effect; after the network training is completed, save the network parameters that perform best in the validation set for model testing.

[0066] Specifically, step 4 includes the following sub-steps: Step 4.1: Obtain the optical and SAR image pairs to be processed, wherein the spatial resolution and number of channels of the image pairs are consistent with the training data; and simultaneously crop the image pairs according to the input size set in step 2 to obtain several pairs of sub-image blocks; and sort them, and fill in the sub-image blocks that are not large enough; finally forming a multimodal remote sensing image test dataset.

[0067] Step 4.2: Input the paired images from the multimodal remote sensing image test dataset into the trained edge-guided cross-modal fusion network; obtain the land cover classification result, i.e., the pixel-level segmentation result of the multimodal remote sensing image test dataset.

[0068] Step 4.3: The detection results of the multimodal remote sensing image test dataset are stitched together using the cropping method described in Step 4.1, and the fill parts of the insufficient-sized image pairs in Step 4.1 are deleted to obtain the complete land cover classification results.

[0069] The effectiveness of the embodiments of the present invention can be further verified through the following experiments.

[0070] 1) Experimental environment.

[0071] The proposed model is implemented in PyTorch and trained using a single NVIDIA RTX 4090 GPU.

[0072] 2) Experiment content.

[0073] The method of this invention, namely an edge-guided cross-modal interactive fusion land cover classification method, is used to perform pixel-level semantic segmentation on publicly available optical and SAR multimodal datasets. To verify the advantages of this method, it is compared with a variety of multimodal semantic segmentation comparison models.

[0074] 3) Accuracy evaluation.

[0075] To quantitatively evaluate the effectiveness of this invention, overall accuracy (OA), mean intersection-over-union ratio (mIoU), and F1 score are used as evaluation indicators. It includes seven land cover categories: farmland, city, village, water body, forest, road, and others.

[0076] Table 1. Accuracy Evaluation of the Results of the Method of the Present Invention and Other Methods

[0077]

[0078] 4) Analysis of experimental results.

[0079] Figure 6 The land cover classification results were converted into a pseudo-color label map, where red represents cities, brown represents farmland, yellow represents villages, blue represents water bodies, green represents forests, light blue represents roads, and purple represents other categories. Figure 6 The experimental results in Table 1 show that, compared with other methods, the edge-guided cross-modal interactive fusion proposed in this invention yields land cover segmentation results that are closer to the ground truth map, demonstrating stronger differentiation and generalization capabilities for complex land cover categories. It also better maintains the boundary continuity of linear targets such as roads, and its quantitative evaluation indicators are significantly superior. Therefore, the method proposed in this invention significantly improves land cover classification performance in multiple aspects.

[0080] This edge-guided, cross-modal interactive fusion land cover classification method constructs a pseudo-twin multi-stream encoder to perform hierarchical feature representation of optical and SAR data. A multimodal adaptive feature interaction module (MAFI) is introduced between adjacent stages of the backbone network. This achieves accurate exchange, dynamic alignment, and feature space recalibration of cross-modal information in two cascaded stages: channel interaction and spatial interaction, fundamentally mitigating fusion bias caused by modal heterogeneity and local spatial mismatch. Furthermore, a lightweight feature fusion module (LFF) is proposed, fusing neuron-level attention and multi-scale deep separable convolutions. This significantly suppresses optical salt-and-pepper noise and SAR speckle noise while performing context aggregation, achieving a synergistic optimization of improved accuracy and reduced computational cost. In addition, an edge-assisted module (EA) is introduced to mine multi-scale boundaries from low-level features and adaptively refine them, jointly decoding them with semantic features to significantly enhance boundary details and small target recognition capabilities. This method can still output high-precision pixel-level land cover segmentation results under complex scenes and heterogeneous sensor conditions, making it suitable for applications such as urban refined management, infrastructure inspection, and disaster emergency mapping.

[0081] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these should also be considered within the scope of protection of the present invention. These will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. An edge-guided cross-modal interactive fusion land cover classification method, characterized in that, Includes the following steps: S1. Acquire optical images and synthetic aperture radar (SAR) images of the same area to construct an optical-SAR multimodal remote sensing image dataset; perform unified spatial reference and pixel alignment on the optical images and SAR images, crop the remote sensing images and labels using the same non-overlapping step size, and perform data augmentation through random rotation, random flipping, and random scaling. S2. Construct an edge-guided multimodal semantic segmentation network; the network includes an optical branch encoder and a SAR branch encoder, a cross-modal adaptive feature interaction module (MAFI), a lightweight feature fusion module (LFF), an edge-assisted module (EA), and a decoder; wherein MAFI performs dynamic reweighting and residual correction on optical features and SAR features at multiple scales to suppress modal feature distribution shifts and local spatial misalignments caused by differences in imaging mechanisms; EA extracts boundary priors from shallow detail features and performs edge-guided refinement on the fused semantic features; S3. Use the dataset constructed in step S1 to train and validate the network described in step S2. Through backpropagation joint optimization, the semantic segmentation output and the edge guidance output converge in tandem to obtain the trained semantic segmentation model. S4. Acquire the optical and SAR image pairs to be segmented and perform preprocessing and synchronous block segmentation consistent with the training phase. Input the segmented image pairs into the trained semantic segmentation model, output pixel-level land cover classification results, and seamlessly stitch the block results according to the cropping rules to obtain a complete classification map. The cross-modal adaptive feature interaction module (MAFI) includes: Channel interaction unit: Global average pooling, global max pooling, and global standard deviation pooling are performed on optical features and SAR features respectively to obtain a global description of multiple statistics. The global description is then used to generate corresponding channel attention weights via nonlinear mapping. The channel attention weights dynamically reweight the features of another modality in a cross-modal interaction manner and are updated using residuals to obtain the channel interaction features. This strengthens the complementary features of the two modalities and suppresses modal noise interference without introducing global attention. Spatial Interaction Unit: Based on the channel interaction features, further extract fusion spatial saliency information, generate spatial attention weights that include spatial location and channel semantic coupling, and introduce trainable scaling factors to adaptively adjust the cross-modal spatial correction intensity, so as to achieve pixel-level alignment correction of cross-modal local misalignment and boundary uncertainty areas, thereby significantly reducing the impact of imaging mechanism differences and geometric offset on subsequent fusion and decoding segmentation. The edge assist module (EA) upsamples and stitches together the multi-scale shallow features of the edge branches to form a multi-scale boundary representation. It also adaptively integrates local edge details with global semantic information through the edge and semantic fusion unit, so that the boundaries of ground features and complex backgrounds can be more clearly separated. The edge feature map and edge probability map output by the edge assistance module are fused with the corresponding scale features of the semantic decoder to provide stable boundary constraints for small targets, slender targets and similar border regions, thereby further improving the continuity of segmentation contours and the accuracy of boundary positioning.

2. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S11. Select optical images and SAR images that meet the quality standards, complete unified resolution processing and strict spatial registration, and make the two modes correspond to the same points. S12. Perform pixel-level annotation on the registered image pairs to obtain land feature category labels, and use the Canny operator to generate boundary labels. S13. Perform simultaneous cropping on optical and SAR images to obtain sub-image block pairs with consistent size, and use filling or discarding strategies for boundary regions with insufficient size to ensure consistent sample scale. S14. Organize the image pair files and corresponding change label files to construct a multimodal remote sensing image semantic segmentation standard dataset containing training and test sets.

3. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, The multimodal semantic segmentation network based on edge guidance described in step S2 takes cropped optical and SAR images of the same size as input. The optical image is input into the semantic segmentation network and the edge branch, and the SAR image is input into the semantic segmentation network. The semantic segmentation network includes two sets of independent encoders for feature extraction. Each encoder consists of four bidirectional self-attention blocks, and each bidirectional self-attention block includes an embedding layer and a two-layer routing attention module. The dual-layer routing attention module divides the input features into two-dimensional windows of size m×m and performs automatic filling and output cropping restoration when the size is not divisible. It performs QKV linear projection on the windowed features and performs two-dimensional downsampling on the key values ​​to control complexity. The window-level segmentation unit routing filters the relevant window indices and weights of the segmentation unit based on the average features of the window, and calculates multi-head attention on the key value set of the filtered window at the pixel level to achieve sparse global interaction across windows. At the same time, it introduces a depthwise separable convolutional local enhancement term to perform residual compensation on the attention output. Finally, the output features are of the same size as the input and are used for subsequent feature input and decoding output.

4. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, The edge-guided multimodal semantic segmentation network further includes an edge branch, which uses ResNet-18 as the backbone encoder and contains n residual convolutional blocks. The edge branch takes an optical image as input and sequentially extracts initial features through convolutional layers, batch normalization layers, and nonlinear activation layers. It then performs stepwise downsampling encoding through the n residual convolutional blocks to obtain multi-scale edge representations. Each residual convolutional block includes at least two convolutional operations and an identity residual connection. The residual connection is used to sum the input features with the convolutional output residuals to mitigate gradient vanishing and enhance detail preservation. When the feature dimension or resolution changes at adjacent scales, the downsampling branch performs channel matching and scale alignment on the input features. The edge branch outputs edge information after the features pass through the edge assistance module. The edge information includes an edge feature map and an edge probability map. This edge information is then injected into the corresponding scale semantic features of the semantic segmentation decoder through upsampling and feature fusion operations to enhance the boundary continuity, clarity, and localization accuracy of the segmentation results.

5. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, The Lightweight Feature Fusion Module (LFF) include: Original feature enhancement path: Introducing a neuronal attention mechanism before and after fusion to suppress unstable noise responses in optical and SAR features, thereby improving the reliability of cross-modal fusion input; Multi-scale feature extraction path: After concatenating the features of two modalities, four sets of depthwise separable convolutions with different scales of 5, 7, 9, and 11 are used to extract cross-modal contextual information in parallel, so as to cover the scale differences of remote sensing targets and keep the computational cost controllable. Fusion and reconstruction path: Multi-scale features are aggregated and reconstructed through pointwise convolution and standard convolution, and the global representation is further weighted by the weight maps generated by the two modalities. This achieves the fusion of global complementary information and modality discrimination information under low complexity conditions, improving the ability to preserve details and the robustness against noise in high-resolution scenes.

6. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S31. Set the parameter values ​​required for network training based on the multimodal remote sensing image semantic segmentation dataset, mainly including the initial learning rate, optimizer and loss function, maximum number of training iterations, etc. The loss function adopts a comprehensive loss function that combines edge and semantic loss. The edge loss adopts dice loss to avoid generating excessively thick edges, and the semantic loss adopts multi-class loss and dice loss to improve the discriminability between segmentation classes while alleviating class non-smoothness. S32. Input the training dataset into the edge-guided cross-modal fusion network for training, and validate it on the validation dataset. During training, update the network parameters through backpropagation so that the semantic segmentation output and the edge output converge simultaneously, thereby achieving the best learning effect. After the network training is completed, save the network parameters that perform best on the validation set for model testing.

7. The edge-guided cross-modal interactive fusion land cover classification method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S41. Obtain the optical and SAR image pairs to be processed, wherein the spatial resolution and number of channels of the image pairs are consistent with the training data; and simultaneously crop the image pairs according to the input size set in step S2 to obtain several pairs of sub-image blocks; and sort them, and fill in the sub-image blocks that are not large enough; finally forming a multimodal remote sensing image test dataset. S42. Input the paired images from the multimodal remote sensing image test dataset into the trained edge-guided cross-modal fusion network; The land cover classification results are obtained, namely the pixel-level segmentation results of the multimodal remote sensing image test dataset; S43. The detection results of the multimodal remote sensing image test dataset are stitched together according to the cropping method in step S41, and the parts of the images with insufficient size in step S41 that are filled in are deleted, and finally the complete land cover classification results are obtained.

Images