Land utilization classification method based on joint semantic segmentation

Abstract

The invention relates to a land utilization classification method based on joint semantic segmentation. The method comprises the following steps: constructing a joint semantic segmentation model; the joint semantic segmentation model comprises an image encoder, a feature fusion block and a feature decoder. And performing feature extraction on the optical image and the SAR image by using an image encoder to obtain a multi-scale feature of the optical image and a multi-scale feature of the SAR image. And fusing the multi-scale features of the optical image and the multi-scale features of the SAR image by using a multi-scale feature fusion module and a dynamic gating feature screening and fusion module in the feature fusion block to obtain fused features. And decoding the fused features by using a feature decoder to obtain an output image with a classification region. By adopting the method, the land classification precision can be improved.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to a land use classification method based on joint semantic segmentation. Background Art

[0002] Given the increasingly severe global environmental changes, accurate depiction and monitoring of land features has become a critical task. Semantic segmentation using remote sensing imagery is a fundamental component of urban planning and environmental monitoring. Furthermore, remote sensing images obtained from a single data source often provide a limited perspective on scene features. Optical remote sensing images offer rich spectral information due to their different imaging mechanisms, but are susceptible to interference from climate and lighting conditions. Conversely, synthetic aperture radar (SAR) can capture images under all-weather conditions, but these images are significantly affected by strong coherence speckle noise. As the most commonly used types of remote sensing images, optical and SAR images are recognized for their complementarity and structural consistency. Effective fusion of salient features from optical and SAR images can mitigate the low accuracy associated with reliance on a single data source. The semantic segmentation process of optical and SAR image fusion typically involves three main stages: feature extraction, feature fusion, and feature decoding. The first stage is feature extraction, where a dual-branch network architecture is used to independently extract features from the optical and SAR images before applying the fusion algorithm. This approach mitigates potential feature extraction biases due to the different radiometric characteristics of the two image types. The second stage is feature fusion. Based on the different features of optical and SAR images and specific task requirements, various algorithms, such as linear fusion, attention mechanism fusion, and gated fusion, are used. The goal is to obtain valuable and coherent features to address the inherent complexity of image fusion. The third stage is feature decoding. Traditional methods recover fused features through techniques such as deconvolution and extended convolution. Analysis of optical and SAR image features reveals that in large-scale remote sensing images, optical images provide rich local details, while SAR images, affected by coherent speckle noise, contain sparse local information. This contrast is particularly evident in architectural feature extraction; optical sensors are unaffected by changes in the incident angle and excel at capturing structural details. However, due to spectral similarity between categories, optical images may face challenges in distinguishing global features, leading to redundancy. Furthermore, cloud and fog coverage further obscures global information, hindering classification accuracy. SAR is unaffected by cloud cover, reliably conveys scene structure, and provides robust global information. Therefore, optical images exhibit strong local information and weak global information, while SAR images provide strong global structure but sparse local details, limiting their support for classification inference.

[0003] Feature extraction, as the first step in the entire classification process, lays the foundation for fusion, but is often overlooked. In most existing fusion algorithms, simple feature extractors with the same structure are used to extract features from optical and SAR images, which actually violates the heterogeneity of heterogeneous images. Typically, the role of feature extractor is played by convolutional neural networks (CNNs). CNNs consist of convolutional modules that take the entire image as input and obtain local features through convolution operations. Clearly, CNNs are very effective at extracting local features, but this is not necessarily suitable for global feature extraction from SAR images. However, Vision Mamba has recently attracted attention in the remote sensing field, offering efficient global information capture with minimal memory consumption. However, using the same feature extractor in both data sources carries the risk of information loss and fails to fully utilize the different strengths of optical and SAR data. This oversight in feature extraction can introduce bias in the initial stage, affecting the quality of downstream feature fusion. Furthermore, in land use classification tasks using fused optical and SAR images, most methods rely on CNNs as feature extractors without evaluating the strengths and limitations of different extractors or their applicability to different image sources. Meanwhile, feature fusion in this field typically employs one of four main methods: linear fusion, self-attention, gated fusion, or feature alignment. Relying on a single method without considering the unique features of each image source often hinders the network from achieving comprehensive multi-level feature integration, leading to information loss from individual sources and reduced classification accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a land use classification method based on joint semantic segmentation that can improve the accuracy of image partitioning, in order to address the aforementioned technical problems.

[0005] A land use classification method based on joint semantic segmentation, the method comprising: Acquire optical and SAR images of the land to be classified; construct a joint semantic segmentation model; the joint semantic segmentation model includes an image encoder, a feature fusion block, and a feature decoder; Feature extraction is performed on optical and SAR images using an image encoder to obtain multi-scale features of optical and SAR images. In the feature fusion block, the multi-scale feature fusion module and the dynamic gating feature filtering and fusion module are used to fuse the multi-scale features of the optical image and the multi-scale features of the SAR image to obtain the fused features. The fused features are decoded using a feature decoder to obtain an output image with classification regions.

[0006] The aforementioned land use classification method based on joint semantic segmentation first utilizes CNN to capture useful local spatial details from optical images and VMamba to capture global structural information from SAR images. Then, a multi-scale dynamic feature fusion method is used to select effective features and filter out redundant features. Simultaneously, adding snake convolutions to VMamba effectively captures structural information from SAR images. To maximize the utilization of complementary image information, this application designs a global-local dynamic gating fusion module. Before feature selection and fusion, multi-scale feature extraction and self-attention mechanisms are used to capture as complete image information as possible, and then a dynamic gating mechanism is used to learn the complementary information of optical and SAR images. In the decoding stage, shallow and deep features are simultaneously merged. Through fusion upsampling, spatial and contextual details across feature levels are preserved to the maximum extent, improving land classification accuracy. Experimental evaluations on datasets of different resolutions confirm the robustness of this application and its superior accuracy compared to existing fusion methods, especially in medium and high resolution scenes. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a land use classification method based on joint semantic segmentation in one embodiment; Figure 2 This is a framework diagram of a joint semantic segmentation model in one embodiment; Figure 3 This is a schematic diagram of the SS2D module in one embodiment; Figure 4 Here is a structural diagram of module S6 in one embodiment; Figure 5 This is a schematic diagram of the VSS block in another embodiment; Figure 6 This is a schematic diagram of the dynamic snake convolution process in one embodiment; Figure 7 This is a schematic diagram of the feature fusion process in one embodiment; Figure 8 This is a schematic diagram of the decoder structure in one embodiment. DETAILED DESCRIPTION

[0008] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0009] In one embodiment, Figure 1 As shown, a land use classification method based on joint semantic segmentation is provided, including the following steps: Step 102: Obtain optical and SAR images of the land to be classified; construct a joint semantic segmentation model; the joint semantic segmentation model includes an image encoder, a feature fusion block, and a feature decoder.

[0010] This application proposes a joint semantic segmentation model, OSHFNet, aimed at land use classification by fusing optical and SAR imagery. Figure 2 As shown, OSHFNet consists of three main components: an image encoder, a feature fusion block, and a feature decoder, each containing multiple image encoders, feature fusion blocks, and feature decoders. The encoder employs a dual-branch network for feature extraction, where a CNN processes optical images to effectively capture rich local features, while a VMamba network structure encodes SAR images, capturing their unique global structural features. Given that remote sensing images often contain elongated structures such as rivers and roads, snake convolutions are embedded in the VMamba structure to enhance the capture of these complex global patterns and improve classification accuracy. For image decoding, a multi-level fusion decoding method is used to preserve different features across different dimensions. The feature fusion module includes two key components. The first is a global-local multi-scale feature fusion module, which integrates a self-attention mechanism for capturing global features and a multi-scale convolution module for local feature extraction. This combined approach enriches the spatial context and enhances the network's ability to identify targets at multiple scales. The second component is a dynamically gated fusion module, which adaptively learns gating weights to selectively filter and fuse complementary features from optical and SAR images, further improving classification performance.

[0011] Step 104: Use an image encoder to extract features from the optical image and the SAR image to obtain multi-scale features of the optical image and the SAR image.

[0012] By utilizing a pre-trained ResNet18 as a feature extractor for optical remote sensing images, the pre-trained model can achieve better feature representations from the outset, avoiding lengthy training from scratch and accelerating convergence. It provides hierarchical feature representations from low-level edges to high-level semantic information, which is valuable for target recognition in remote sensing. By leveraging these pre-trained networks, the remote sensing model inherits refined feature extraction capabilities, facilitating accurate object and detail identification in complex scenes.

[0013] Feature extraction from SAR images is performed using a selectively structured state-space model (SSM). The SSM can be VMamba. VMambaV first divides the input image into blocks instead of flattening it into a 1-D sequence, preserving the image's two-dimensional structure, thus obtaining a feature map of size H / 4 × W / 4. Then, several VSS blocks are applied to this feature map in the initial "first stage." The hierarchical representation in VMamba is constructed by downsampling the feature map through patch merging, forming successive "stages" with gradually decreasing resolution: H / 8 × W / 8, H / 16 × W / 16, and H / 32 × W / 32, creating a multi-level feature hierarchy.

[0014] VMamba's unique cross-scan mechanism is at the heart of its design, implementing a four-way scan strategy that captures information from all four corners of the feature map. This approach allows each element to integrate information from all locations and directions, forming a global receptive field while maintaining linear computational complexity. The SS2D module is central to this process, performing feature extraction in three steps—scan expansion, S6 block formation, and scan merging—as follows: Figure 3 As shown, this structure ensures efficient and comprehensive feature extraction. The structure of the S6 block is as follows: Figure 4 As shown. The structure of each VSS block is as follows. Figure 5 As shown, starting from the linear embedding layer, the output is split into two streams. One stream enters the SS2D module through a 3×3 depth convolutional layer with SiLU activation. After processing in SS2D, this stream undergoes layer normalization before being merged with the SiLU-activated outputs of other streams, thus producing the final output of the VSS block. The S6 block is as follows. Figure 5 As shown, the matrix 、 and All of them are from the input data This is derived from the S6 algorithm. This means that S6 knows the contextual information embedded in the input, ensuring the dynamic nature of the weights in the mechanism. These sequences are then processed separately by the SSM. Finally, the obtained features are combined to generate a complete two-dimensional feature map.

[0015] In remote sensing image segmentation, elongated structures are often overlooked. To enhance the flexibility of convolution kernels in capturing the complex geometric details of these targets, this application replaces the standard convolution structure in the VMamba VSS module with snake convolution. Unlike standard convolution, snake convolution introduces deformation offset. ,like Figure 6As shown. Standard convolution only focuses on pixels within the kernel, while dilated convolution expands the receptive field by filling in surrounding pixels, but these methods still cannot capture features of elongated structures. Dynamic snake convolution linearizes the kernel along the x and y axes. Consider a kernel of size 9. Taking the x-axis as an example, the specific location of K in each grid is represented as follows: Where c = 0, 1, 2, 3, 4 represents the horizontal distance from the center grid. Each grid position is selected on the convolution kernel K. It is a cumulative process. From the central position Initially, the position of the grid away from the center depends on the position of the previous grid: Increased Relative to The offset is calculated by gradually accumulating the offset and using an iterative strategy to select the next target location, ensuring consistent attention without overexpanding the sensing range. This allows VMamba to capture elongated, complex geometric features while preserving the global structural information of SAR images.

[0016] Step 106: In the feature fusion block, the multi-scale feature fusion module and the dynamic gating feature filtering and fusion module are used to fuse the multi-scale features of the optical image and the multi-scale features of the SAR image to obtain the fused features.

[0017] like Figure 7 As shown, the feature fusion block—the GLDG module—mainly consists of two parts: a multi-scale feature fusion module and a dynamic gating feature filtering and fusion module. The global-local feature learning module in GLDG has two branches. The first branch learns global features through a multi-head self-attention mechanism, while the second branch uses multi-kernel convolution to capture multi-scale local features. Once processed by the global-local feature learning module, the gating weights derived from the self-attention mechanism are applied to selectively filter and fuse features from optical and SAR images, respectively.

[0018] and These represent the feature map outputs from the RGB and SAR encoders, respectively. First, a self-attention mechanism and a multi-scale feature extraction mechanism are used to fuse features obtained from the optical and SAR image feature extractors, respectively. This yields the optical multi-scale fused features. and SAR multi-scale fusion characteristics To dynamically fuse multi-scale features from optical and SAR images, convolutional layers are used to adjust the number of channels for optical and SAR features to maintain consistency, given the different feature extractors. These features are then concatenated along a dimension with dim=1. A gating threshold in the 0-1 range is obtained using convolutional layers, ReLU activation function, batch normalization layer, and sigmoid function. Finally, this gating filter fuses the optical and SAR multi-scale features to obtain the final feature.

[0019] Step 108: Use the feature decoder to decode the fused features to obtain an output image with classification regions.

[0020] The structure of the decoder is as follows Figure 8 As shown, from the highest level of final features Initially, the feature is input to decoder 4. Here, feature recombination is performed through a multi-scale fusion module in the first half of the convolutional layer and GLDG fusion module. The recombined feature is then merged with the final feature and output after passing through an MLP layer and a droppath layer. Decoder 3 receives the output of decoder 4 and the fused feature. At this stage, the output features of Decoder4 are processed similarly, and then... Bilinear interpolation fusion is performed to obtain the output of Decoder3. Decoder2 and Decoder1 follow the same process as Decoder3. Finally, the final upsampling layer generates an output image with land classification regions.

[0021] The aforementioned land use classification method based on joint semantic segmentation first utilizes CNN to capture useful local spatial details from optical images and VMamba to capture global structural information from SAR images. Then, a multi-scale dynamic feature fusion method is used to select effective features and filter out redundant features. Simultaneously, adding snake convolutions to VMamba effectively captures structural information from SAR images. To maximize the utilization of complementary image information, this application designs a global-local dynamic gating fusion module. Before feature selection and fusion, multi-scale feature extraction and self-attention mechanisms are used to capture as complete image information as possible, and then a dynamic gating mechanism is used to learn the complementary information of optical and SAR images. In the decoding stage, shallow and deep features are simultaneously merged. Through fusion upsampling, spatial and contextual details across feature levels are preserved to the maximum extent, improving land classification accuracy. Experimental evaluations on datasets of different resolutions confirm the robustness of this application and its superior accuracy compared to existing fusion methods, especially in medium and high resolution scenes.

[0022] In one embodiment, an image encoder is used to extract features from optical and SAR images to obtain multi-scale features of the optical and SAR images, including: Pre-trained ResNet18 is used to extract features from optical images to obtain multi-scale features of optical images; Feature extraction of SAR images is performed using a selective structured state-space model, dividing the SAR images into blocks to obtain initial feature maps. Multi-level feature extraction is then performed on the initial feature maps by merging patches based on multiple VSS modules embedded with dynamic serpentine convolutions, resulting in multi-scale features of the SAR images.

[0023] In one embodiment, the VSS module includes a normalization layer, a linear layer, a dynamic snake convolutional layer, and an SS2D module; based on the multiple VSS modules, the initial feature map is patched and merged to perform multi-level feature extraction, resulting in multi-scale features of the SAR image, including: The initial feature map is normalized in the normalization layer. Then, the normalized feature map is split into two paths. One path passes through a linear layer and then enters a dynamic snake convolution layer, causing the convolution kernel to move along... x shaft and y Linearization encodes the normalized feature map to obtain global structural information features, and then uses the SS2D module to perform feature mapping on the global structural information features to obtain a feature map; in another path, a linear layer is used to perform linear processing on the normalized feature map to obtain a linear feature map. After normalizing the feature map, it is combined with the linear features to obtain the multi-scale features of the SAR image.

[0024] In one embodiment, the SS2D module is used to perform feature mapping on the global structural information features to obtain a feature mapping map, including: The SS2D module is used to perform feature mapping on the global structural information features, resulting in the following feature mapping map.

[0025] in, Represents global structural information characteristics. ∈ ={1,2,3,4} represents four different scan directions, and expand() and merge() represent the scan expansion and scan merging operations in the code. S6 represents the feature sequence after the input is scanned in four directions. It is a selective scan spatial state sequence model. This represents the output sequence after different scan sequences are processed by the S6 module. This represents the feature map.

[0026] In one embodiment, a multi-scale feature fusion module and a dynamic gating feature filtering and fusion module are used to fuse the multi-scale features of the optical image and the multi-scale features of the SAR image to obtain the fused features, including: In the multi-scale feature fusion module, the self-attention mechanism and the multi-scale feature extraction mechanism are used to fuse the multi-scale features of optical images and SAR images to obtain optical multi-scale fusion features and SAR multi-scale fusion features. In the dynamic gating feature filtering and fusion module, convolutional layers, ReLU activation function, normalization layer and sigmoid function are used to obtain gating thresholds in the range of 0-1. Based on the gating thresholds, the spliced ​​multi-scale fusion features are selectively filtered and fused to obtain the final features.

[0027] In one embodiment, a self-attention mechanism and a multi-scale feature extraction mechanism are used to fuse the multi-scale features of the optical image and the multi-scale features of the SAR image to obtain optical multi-scale fused features and SAR multi-scale fused features, including: This paper utilizes a self-attention mechanism and a multi-scale feature extraction mechanism to fuse multi-scale features from optical and SAR images. The number of channels for the multi-scale features from both images is adjusted in the convolutional layer, and then they are concatenated along the dimension with dim=1 to obtain the optical and SAR multi-scale fused features.

[0028] in, Represents the multi-scale features of optical images. Indicates the splicing module. Indicates a convolutional layer. This represents the features obtained by the optical self-attention mechanism. Indicates the kernel size. The range is 1, 3, 5. Indicates the fusion dimension. This refers to optical multi-scale fusion features, which fuse multi-scale features of optical images and features obtained by optical self-attention mechanisms at a dimension of 1. Represents the multi-scale features of SAR images. This refers to SAR multi-scale fusion features, which fuse multi-scale features of SAR images and features obtained by SAR self-attention mechanism at a dimension of 1. This represents the SAR image features acquired by VMamba, i.e., the initial feature map. This represents a self-attention mechanism model. This indicates a dynamic gating module.

[0029] In one embodiment, the stitched multi-scale fusion features are selectively filtered and fused according to a gating threshold to obtain the final features, including: Based on the gating threshold, selective filtering and fusion are performed on the stitched multi-scale fused features to obtain the final features.

[0030] in, This indicates the characteristics of optical multi-scale fusion. This indicates the multi-scale fusion characteristics of SAR. This indicates a dynamic gating module.

[0031] In one embodiment, the feature decoder includes multiple decoders; the feature decoders are used to decode the fused features to obtain an output image with a classification region, including: The final features of the fourth feature fusion block are input into the fourth decoder, and the final features are reconstructed at multiple scales to obtain the reconstructed features. The reconstructed features are then merged with the final features and output after passing through an MLP layer and a droppath layer. After the output of the fourth decoder is input into the third decoder, the output of the fourth decoder is recombined in the manner described above, then bilinear interpolation upsampling is performed, and then fused with the final features of the third feature fusion block for output; the decoding process in the second and first decoders is the same as that in the third decoder, and the output of the first decoder is an output image with a classification region.

[0032] In specific embodiments, this application conducted multiple experiments. First, the WHU-OPT-SAR dataset was acquired. This dataset provides images with a 5-meter resolution, including 100 pairs of optical and SAR images, each 5556×3704 pixels in size, covering an area of ​​approximately 50,000 square kilometers in Province A (30°N-33°N, 108°E-117°E). This dataset encompasses different terrain and vegetation types, providing the pixel-level annotations necessary for deep learning-based land use classification. The YYX-OPT-SAR image set contains 150 pairs of 512×512 images, covering urban, suburban, and mountainous areas. The SAR images were collected by UAVs, with a geographic range of 110.135°E-110.160°E and latitude of 34.593°N-34.626°N. The corresponding optical images of the area were obtained from mapping software, with a data resolution of 0.5m.

[0033] First, WHU-OPT-SAR experiments were conducted, selecting several methods for quantitative and qualitative comparisons on the 5-meter resolution WHU-OPT-SAR dataset. Two sets of experiments were performed using U-Net and DeepLabv3+: one using only optical images, and the other using both optical and SAR images. These experiments demonstrate to some extent that for medium-resolution remote sensing image segmentation tasks, simply using linear fusion methods can improve classification accuracy. MCANet introduced a self-focused cross-fusion module for land use classification. CMGFNet uses a gated fusion mechanism to fuse multimodal information for classification, while DCSANet uses a dynamic convolutional self-focused mechanism to fuse multi-source information for classification.

[0034] Meanwhile, to verify the effectiveness of the heterogeneous branches, the fusion module was kept unchanged, and the feature extraction network was replaced with CNN or VMamba.

[0035] The results show that this application achieves the highest OA (81.65%), kappa (73.96%), and mIOU (46.56%), significantly exceeding the values ​​obtained by other methods. Compared to the latest fusion methods, this application improves OA and mIoU by approximately 1% and 3.0%, respectively. Analysis of classification results for individual categories reveals that this application effectively improves the accuracy of categories with strong global structural features (such as water bodies and roads). Specifically, the accuracy for water bodies reaches 78.55%, approximately 3% higher than other fusion methods, and the accuracy for roads is 45.12%, approximately 15% higher than other methods. Furthermore, since CNNs can acquire local information from optical images, they can effectively compensate for the shortcomings of SAR image-based classification. The accuracy of this application in villages is approximately 2% higher than other methods.

[0036] The reason this application achieves such excellent results is as follows: The heterogeneous dual-branch network in this application uses CNN to capture useful local spatial details from optical images and VMamba to capture global structural information from SAR images. Then, a multi-scale dynamic feature fusion method is used to select effective features and remove redundant features. Simultaneously, adding snake convolutions effectively captures structural information in SAR images. At this point, the network can fully learn the multi-level complementary features and structural consistency features of optical and SAR images.

[0037] Optical and SAR images, as the two most widely used modalities of remote sensing imagery, provide complementary information and structural consistency. Therefore, research on optical and SAR image fusion has been increasing year by year. Existing methods all extract features from optical and SAR images using feature encoders with the same structure, then design fusion algorithms to fuse the features extracted by the isomorphic network and finally classify them. However, optical and SAR images differ significantly in their image representation, and feature extraction, as the most fundamental and crucial step, suffers from significant information loss due to simple isomorphic networks. To overcome this problem, this application starts with an analysis of the representational forms of optical and SAR images, adopting a heterogeneous dual-branch network framework to learn the rich local information of optical images and the rich global structural information of SAR images. Furthermore, to maximize the utilization of complementary information, this application designs a global-local dynamic gating fusion module. Before feature selection and fusion, multi-scale feature extraction and self-attention mechanisms are used to capture as much complete image information as possible, and then a dynamic gating mechanism is used to learn the complementary information of optical and SAR images. This application has been tested on medium-resolution and high-resolution image datasets, and it outperforms other state-of-the-art methods. More importantly, this application can adapt to data of different resolutions simultaneously. Experimental results demonstrate the validity of this application.

[0038] In addition, YYX-OPT-SAR experiments were conducted, performing quantitative and qualitative comparisons of the same method on the 0.5-meter high-resolution YYX-OPT-SAR dataset. Similarly, two sets of experiments were performed using U-Net and DeepLabv3+, one using only optical images and the other using both optical and SAR images. However, the results for 0.5-meter high-resolution images yielded the opposite conclusion compared to medium-resolution images. These two sets of experiments suggest, to some extent, that simple fusion alone cannot fully utilize multimodal information and may even degrade model performance. Compared to other fusion methods, this application achieved the highest OA (84.92%), Kappa (70.26%), and mIOU (50.89%), quantitatively exceeding the same values ​​obtained by several other methods. Compared to state-of-the-art fusion methods, this application improved OA and mIoU by approximately 2% and 3%, respectively. It can be seen that this application outperforms other methods on high-resolution images compared to medium-resolution images. Second, the classification accuracy of individual categories was observed. The OSHFNet proposed in this application was found to have high accuracy in each category, improving overall accuracy without sacrificing the accuracy of individual categories.

[0039] To better demonstrate the necessity and efficiency of each part used in this application, ablation analysis is described in detail in this embodiment. All cases were trained and tested using the same dataset: 1. Effectiveness of the Heterogeneous Two-Branch Encoder: To verify the effectiveness of the heterogeneous two-branch network in 0.5-meter high-resolution images, this application was compared with OSCNN and OSMamba. OSCNN is a two-branch CNN network that preserves the fusion module, and OSMamba is a single-branch Mamba network that preserves the fusion module. OA for each category, as well as the average OA, mIOU, and KAPPA results, were verified.

[0040] First, compared to homogeneous two-branch networks, this application achieves the highest accuracy across all metrics, demonstrating the effectiveness of heterogeneous two-branch networks. Second, ablation experiments were conducted on different modules to verify their effectiveness. To better illustrate the effectiveness and necessity of heterogeneous two-branch networks in this application, a detailed ablation analysis is provided. All experiments were performed using the same dataset. Two sets of experiments were designed to compare this application with OSCNN and OSMamba. OSCNN is a two-branch homogeneous CNN network that retains the fusion module, while OSMamba is a two-branch Mamba network that also retains the fusion model. The overall accuracy (OA), mean OA, and mean intersection (mIoU) for each class were evaluated. First, regardless of the dataset resolution, this application achieves the highest accuracy across all metrics compared to homogeneous two-branch networks, demonstrating the effectiveness of heterogeneous two-branch networks. Second, after analyzing the experimental results on datasets of different resolutions, it was observed that the accuracy of homogeneous two-branch CNN is significantly lower on the medium-resolution 5-meter dataset, while its performance is significantly better than homogeneous two-branch Mamba on the high-resolution 0.5-meter dataset. This difference can be attributed to the nature of image representation: CNNs primarily capture local information, while local information in medium-resolution SAR images is very limited, leading to adverse effects during the fusion process. Conversely, VMamba captures global information, effectively learning the global structural features of SAR images, thus enhancing the network's ability to learn these structural features. Visualization results also show that two-branch Mamba can clearly depict the global structural contours of an image, while two-branch CNNs tend to produce local confusion. In high-resolution images, optical images contain rich information, and the scattering characteristics of SAR images can also be effectively represented. At this stage, CNNs can extract rich local features, and when combined with multi-scale fusion modules, the network can learn complementary features more effectively.

[0041] 2. Effectiveness of the Fusion Module: Analysis of the module ablation results shows that adding any module can improve classification accuracy. Compared to a single module, the combination of two modules can improve classification accuracy. This application combines three modules to achieve optimal accuracy. This is because it can dynamically and effectively adjust the complementary information of optical and SAR images according to different categories, and the data correlation between modes can improve accuracy. It should be understood that, although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0042] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0043] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0044] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A land use classification method based on joint semantic segmentation, characterized in that: The method comprises: Acquire optical images and SAR images of the land to be classified; construct a joint semantic segmentation model; the joint semantic segmentation model includes an image encoder, a feature fusion block and a feature decoder; Using the image encoder to extract features from the optical image and the SAR image to obtain multi-scale features of the optical image and multi-scale features of the SAR image; In the feature fusion block, a multi-scale feature fusion module and a dynamic gated feature screening and fusion module are used to fuse the multi-scale features of the optical image with the multi-scale features of the SAR image to obtain fused features; The fused features are decoded using the feature decoder to obtain an output image with classified areas.

2. The method according to claim 1, characterized in that The image encoder is used to extract features from the optical image and the SAR image to obtain multi-scale features of the optical image and multi-scale features of the SAR image, including: Using a pre-trained ResNet18 to extract features from the optical image to obtain multi-scale features of the optical image; The SAR image is subjected to feature extraction using a selective structured state space model, and the SAR image is divided into blocks to obtain an initial feature map; and multi-level feature extraction is performed on the initial feature map by patch merging according to a plurality of VSS modules embedded with dynamic snake convolutions to obtain multi-scale features of the SAR image.

3. The method according to claim 1, characterized in that The VSS module includes a normalization layer, a linear layer, a dynamic snake convolution layer and a SS2D module; The initial feature map is subjected to multi-level feature extraction by patch merging according to multiple VSS modules to obtain multi-scale features of the SAR image, including: The initial feature map is normalized in the normalization layer, and then the normalized feature map is divided into two paths. One path passes through the linear layer and enters the dynamic snake convolution layer so that the convolution kernel is along x Axis and y Axis linearization is used to encode the normalized feature map to obtain global structural information features, and then the SS2D module is used to perform feature mapping on the global structural information features to obtain a feature mapping map; In another path, a linear layer is used to perform linear processing on the normalized feature map to obtain a linear feature map; The feature map is normalized and then merged with the linear features to obtain multi-scale features of the SAR image.

4. The method according to claim 3, characterized in that The SS2D module is used to perform feature mapping on the global structure information feature to obtain a feature mapping diagram, including: The SS2D module is used to perform feature mapping on the global structural information features, and the feature mapping diagram is obtained as follows: in, Represents the global structural information features, ∈ ={1,2,3,4} represents four different scanning directions, expand() and merge() represent the scan expansion and scan merging operations, It represents the feature sequence after the input is scanned in four directions. S6 is the selective scanning space state sequence model. It represents the output sequence after different scanning sequences are processed by S6 module. Represents a feature map.

5. The method according to claim 1, characterized in that The multi-scale features of the optical image and the multi-scale features of the SAR image are fused using a multi-scale feature fusion module and a dynamic gated feature screening and fusion module to obtain fused features, including: In the multi-scale feature fusion module, the multi-scale features of the optical image and the multi-scale features of the SAR image are fused by using a self-attention mechanism and a multi-scale feature extraction mechanism to obtain optical multi-scale fusion features and SAR multi-scale fusion features; In the dynamic gating feature screening and fusion module, convolutional layers, ReLU activation functions, normalization layers and Sigmoid functions are used to obtain a gating threshold in the range of 0-1. The spliced ​​multi-scale fusion features are selectively filtered and fused according to the gating threshold to obtain the final features.

6. The method according to claim 5, characterized in that The multi-scale features of the optical image and the multi-scale features of the SAR image are fused by using a self-attention mechanism and a multi-scale feature extraction mechanism to obtain optical multi-scale fusion features and SAR multi-scale fusion features, including: The multi-scale features of the optical image and the multi-scale features of the SAR image are fused by using the self-attention mechanism and the multi-scale feature extraction mechanism. The number of channels of the multi-scale features of the optical image and the multi-scale features of the SAR image are adjusted in the convolution layer, and then they are concatenated in the dimension of dim=1 to obtain the optical multi-scale fusion features and the SAR multi-scale fusion features as follows: in, Represents the multi-scale features of optical images, Represents the splicing module, represents the convolutional layer, represents the features obtained by the optical self-attention mechanism, represents the convolution kernel size, The range is 1, 3, 5, represents the fusion dimension, It represents the optical multi-scale fusion feature that fuses the multi-scale features of the optical image and the features obtained by the optical self-attention mechanism at a level of dimension 1. Represents the multi-scale features of SAR images, It represents the SAR multi-scale fusion feature that fuses the multi-scale features of the SAR image and the features obtained by the SAR self-attention mechanism at a level of dimension 1. It represents the SAR image features acquired by VMamba, that is, the initial feature map. represents the self-attention mechanism model, Represents a dynamic gating module.

7. The method according to claim 5, characterized in that The spliced ​​multi-scale fusion features are selectively filtered and fused according to the gated threshold to obtain the final features, including: According to the gate threshold, the spliced ​​multi-scale fusion features are selectively filtered and fused to obtain the final feature: in, represents the optical multi-scale fusion feature, represents the SAR multi-scale fusion feature, Represents a dynamic gating module.

8. The method according to claim 1, characterized in that The feature decoder includes a plurality of decoders; the fused features are decoded by using the feature decoder to obtain an output image with classified areas, including: After inputting the final features of the fourth feature fusion block into the fourth decoder, the final features are multi-scale reorganized to obtain reorganized features; the reorganized features are combined with the final features and then output after passing through the MLP layer and the droppath layer; The output of the fourth decoder is input into the third decoder and then reorganized in the manner of the previous section, and the features obtained by fusion with the third feature after bilinear interpolation upsampling are fused and output. The decoding process in the second decoder and the first decoder is the same as that in the third decoder, and the output of the first decoder is an output image with classified areas.

Images