Mangrove forest remote sensing image semantic segmentation method and system based on multi-modal data

Abstract

The application provides a mangrove remote sensing image semantic segmentation method and system based on multi-modal data, relates to the technical field of image semantic segmentation, and comprises the following steps: acquiring multi-modal remote sensing images of a target mangrove; performing multi-level double-branch coding on the multi-modal remote sensing images to obtain coding features corresponding to multiple levels, including optical coding features and SAR coding features; fusing the optical coding features and the SAR coding features corresponding to the same level to obtain fusion features corresponding to different levels; decoding the fusion features corresponding to different levels to obtain decoding features; constructing a mangrove remote sensing image semantic segmentation model based on the decoding features, and outputting a remote sensing image semantic segmentation result of the target mangrove through the mangrove remote sensing image semantic segmentation model. The application solves the problem that the existing mangrove remote sensing monitoring method cannot effectively capture multi-scale features and realize deep complementary of multi-modal information.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and more specifically, to a method and system for semantic segmentation of mangrove remote sensing images based on multimodal data. Background Technology

[0002] Mangroves, as one of the most valuable ecosystems in tropical and subtropical coastal zones, play an irreplaceable role in coastal protection, biodiversity maintenance, blue carbon sequestration, and climate change mitigation. However, in recent years, affected by sea-level rise, land reclamation, pollution, and extreme weather, the global mangrove area has been shrinking rapidly, necessitating high-precision, dynamic monitoring of its spatial distribution and health status. Traditional monitoring methods based on field surveys are time-consuming, labor-intensive, and have limited coverage, while high-resolution remote sensing technology offers the possibility of large-scale, long-term mangrove mapping.

[0003] However, existing multimodal fusion segmentation technologies still have many bottlenecks: insufficient multi-scale feature capture, making it difficult to simultaneously address the recognition needs of large-area distribution of mangroves and the fine structure of small patches; insufficient depth of modal information fusion, often remaining at simple weighting or shallow splicing, failing to fully explore the complementary relationship between the two at the semantic and structural levels; prominent issues of boundary ambiguity and omissions, making it difficult to identify the boundary transition zone between mangroves and water / land, and small mangrove patches are easily missed; and weak anti-interference ability, with segmentation accuracy significantly decreasing when facing complex scenarios such as cloud cover and tidal inundation.

[0004] These problems make it difficult for existing technologies to meet the actual needs of high-precision dynamic monitoring of mangroves. There is an urgent need to develop a semantic segmentation method that can achieve deep fusion of multimodal information, accurate capture of multi-scale features, and effective enhancement of boundary details. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic segmentation method and system for mangrove remote sensing images based on multimodal data, in order to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0006] Firstly, this application provides a semantic segmentation method for mangrove remote sensing images based on multimodal data, including:

[0007] Acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images;

[0008] The multimodal remote sensing image is subjected to multi-level dual-branch coding to obtain coding features corresponding to multiple levels. The coding features include optical coding features and SAR coding features.

[0009] The optical coding features and SAR coding features corresponding to the same level are fused together, and combined with channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion, fused features corresponding to different levels are obtained.

[0010] Based on the fusion features corresponding to different levels, feature decoding is performed to obtain decoded features;

[0011] Based on the decoding features, a semantic segmentation model for remote sensing images of mangroves is constructed, and the semantic segmentation results of remote sensing images of the target mangroves are output through the semantic segmentation model for remote sensing images of mangroves.

[0012] Secondly, this application also provides a semantic segmentation system for mangrove remote sensing images based on multimodal data, including:

[0013] The acquisition unit is used to acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images;

[0014] The coding unit is used to perform multi-level dual-branch coding on the multimodal remote sensing image to obtain coding features corresponding to multiple levels, including optical coding features and SAR coding features.

[0015] The fusion unit is used to fuse the optical coding features and the SAR coding features corresponding to the same level, and combine channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion to obtain fused features corresponding to different levels.

[0016] The decoding unit is used to perform feature decoding based on the fusion features corresponding to different levels to obtain decoded features;

[0017] The semantic segmentation unit is used to construct a semantic segmentation model for mangrove remote sensing images based on the decoded features, and output the semantic segmentation result of the target mangrove remote sensing image through the mangrove remote sensing image semantic segmentation model.

[0018] The beneficial effects of this invention are as follows:

[0019] (1) This invention achieves deep complementarity of optical and SAR modal information and accurate capture of multi-scale features through the innovative combination of dual-branch encoder and multimodal pyramid fusion module (MPF). The optical coding branch focuses on extracting the spectral and texture features of mangroves, while the SAR coding branch efficiently captures structural and scattering information. The combination of the two avoids the problem of information loss caused by cloud and tide in optical images and makes up for the shortcomings of insufficient spectral discrimination in SAR images. The MPF uses a layered boundary enhancement mechanism. The shallow layer uses the boundary perception module to explicitly extract edge features, while the deep layer uses the boundary-guided fusion module to guide the fusion of edge and semantic features, effectively solving the problems of blurred boundaries and missed small patches in mangrove segmentation.

[0020] (2) This invention employs a dual-path fusion strategy in the spatial and frequency domains. In the spatial domain, it uses shallow weighted summation to balance modal contributions and deep cross-modal attention to mine semantic associations. In the frequency domain, it separates high- and low-frequency information through wavelet transform and fuses them accordingly, fully leveraging the complementary advantages of the two modalities. Simultaneously, the decoder uses a top-down feature aggregation approach to ensure the full utilization of multi-scale information, significantly improving the identification ability of small mangrove patches and the differentiation accuracy of subtidal mangroves. Therefore, it can improve the robustness and segmentation accuracy of the semantic segmentation model for mangrove remote sensing images, thus providing reliable technical support for high-frequency dynamic monitoring, ecological protection, and management of mangroves, and has significant theoretical and practical value.

[0021] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the semantic segmentation method for mangrove remote sensing images based on multimodal data as described in this embodiment of the invention;

[0024] Figure 2 This is a schematic diagram of the semantic segmentation model for mangrove remote sensing images in an embodiment of the present invention;

[0025] Figure 3This is a schematic diagram of the multimodal pyramid fusion module in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram comparing the model segmentation results in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] Example 1:

[0030] This embodiment provides a semantic segmentation method for mangrove remote sensing images based on multimodal data.

[0031] It should be noted that existing remote sensing monitoring methods for mangroves are unable to effectively capture multi-scale features and achieve deep complementarity of multi-modal information when faced with challenges such as complex spectral characteristics, highly heterogeneous spatial patterns, continuous cloud cover, and periodic tidal inundation. This results in low segmentation accuracy, blurred boundaries, missed small patches, and insufficient ability to identify subtidal mangroves.

[0032] Optical and SAR modes are naturally complementary in terms of information content and imaging mechanism, and their combined use has become an inevitable trend in mangrove remote sensing mapping.

[0033] Specifically, while high-resolution multispectral optical imagery can provide rich spectral and textural information about vegetation, cloud cover in tropical and subtropical regions where mangroves are distributed is as high as 60%–90% year-round, and canopy shadows and tidal inundation further exacerbate the information loss in optical imagery. In contrast, synthetic aperture radar (SAR) has all-weather, all-time imaging capabilities and is highly sensitive to vegetation canopy structure, roughness, and underlying surface moisture. It can effectively penetrate clouds and obtain structural and moisture information that is difficult to capture in optical imagery.

[0034] See Figure 1 The figure shows that the method includes steps S1, S2, S3, S4 and S5.

[0035] Step S1: Acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images;

[0036] In this step, the optical image is a 10-band multispectral optical image, and the SAR image is a 3-channel image. Simultaneously, both the optical and SAR images in the multimodal remote sensing imagery have undergone preprocessing and spatiotemporal registration, and their modalities have been unified into the same geographic coordinate system.

[0037] Step S2: Perform multi-level dual-branch coding on the multimodal remote sensing image to obtain coding features corresponding to multiple levels. The coding features include optical coding features and SAR coding features.

[0038] In this step, Figure 2 The image shown is a semantic segmentation model for mangrove remote sensing images, where MPF represents the multimodal pyramid fusion module, and Output represents the output of the mangrove remote sensing image semantic segmentation model. Indicates the width of the feature map. Indicates the feature map height. This indicates the number of feature map channels. It should be noted that all features mentioned in this embodiment are feature maps.

[0039] like Figure 2 As shown, the multi-level dual-branch coding uses a dual-branch encoder, which includes an optical coding branch (Optical Path) and a SAR coding branch (SAR Path). Each branch includes a patch embedding module and four levels of coding layers.

[0040] It should be noted that the four levels, from lowest to highest, are Stage 1, Stage 2, Stage 3, and Stage 4. Here, "Stage" indicates the layer level; Stage 1 and Stage 2 are shallow levels, while Stage 3 and Stage 4 are deep levels. Furthermore, the resolutions of the output features corresponding to Stage 1, Stage 2, Stage 3, and Stage 4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively.

[0041] Step S2 includes:

[0042] Step S21: Construct the encoding layer, which is obtained by constructing multiple Swing Transformer modules;

[0043] Step S22: Construct an optical coding branch and a SAR coding branch by sequentially connecting a patch embedding module and multiple coding layers. The optical coding branch and the SAR coding branch have the same structure.

[0044] In this step, the patch embedding module is a 4×4 patch embedding. Taking the optical coding branch as an example, the coding layer of Stage 1 includes one linear embedding module and two Swin Transformer modules connected in sequence; the coding layer of Stage 2 includes one patch embedding module and two Swin Transformer modules connected in sequence; the coding layer of Stage 3 includes one patch embedding module and six Swin Transformer modules connected in sequence; and the coding layer structure of Stage 4 is the same as that of Stage 2, including one patch embedding module and two Swin Transformer modules connected in sequence. Figure 2 In the text, ×2 and ×6 indicate that the number of Swin Transformer modules is 2 and 6, respectively.

[0045] The structure of the SAR coding branch is the same as that of the optical coding branch. The linear embedding module is used for linear embedding, while the Swin Transformer module is the Swin Transformer Block, which is used to capture long-range dependencies through internal window self-attention (W-MSA) and shift window self-attention (SW-MSA) mechanisms.

[0046] It should be noted that the input image size for the optical coding branch is... The size of the input image for the SAR coding branch is .

[0047] Meanwhile, patch embedding is a key step in the Swin Transformer module to convert a two-dimensional image into a serialized feature representation. It segments the input image into fixed-size, non-overlapping patch blocks, and then maps each patch block to a high-dimensional feature space through linear projection. In 4×4 patch embedding, 4×4 means that the input image is segmented into 4-pixel × 4-pixel blocks, and each 4×4 pixel block is regarded as a patch block.

[0048] Step S23: Encode the optical image through the optical coding branch to obtain optical coding features corresponding to multiple levels;

[0049] Step S24: The SAR image is feature-encoded through the SAR coding branch to obtain SAR coding features corresponding to multiple levels.

[0050] It should be noted that the boundaries between mangroves and water / land are often blurred. The dual-branch encoder with a dual-branch architecture captures boundary information from two perspectives. Optical imagery provides the high reflectivity characteristics of mangroves in the near-infrared band, making it easier to distinguish mangroves from other vegetation types; SAR imagery provides the backscattering characteristics of mangrove trunks and branches, which can penetrate clouds and vegetation canopies, and still provide effective information under adverse weather conditions.

[0051] Therefore, the dual-branch architecture can provide richer feature representations. The optical coding branch specifically extracts the spectral and texture features of mangroves, preserving spectral details, while the SAR coding branch specifically extracts the structural and geometric features of mangroves, preserving structural details. Simultaneously, the complementary and detailed information from the two branches can mutually enhance each other during subsequent fusion, improving the detection capability of small mangrove patches through multi-scale feature extraction using the dual-branch architecture.

[0052] Meanwhile, when encoding features of optical images and SAR images through corresponding branches, it is necessary to crop the optical images and SAR images and divide them into 256×256 pixel input image blocks.

[0053] Step S3: Perform feature fusion on the optical coding features and SAR coding features corresponding to the same level, and combine channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion to obtain fused features corresponding to different levels;

[0054] In this step, multiple multimodal pyramid fusion modules (MPFs) are used for feature fusion. For each MPF, feature fusion is performed on optically coded features and SAR-coded features at one level. Each MPF includes a channel alignment module, a hierarchical boundary enhancement module, a spatial domain fusion module, a frequency domain fusion module, and a refinement module.

[0055] It should be noted that the complex background of the Gaoqiao Mangrove Nature Reserve presents challenges such as the mixing of various land cover types, spectral similarity, and structural complexity. MPF utilizes mechanisms such as multi-scale fusion, boundary enhancement, and spatial-frequency dual-path fusion to fully leverage complementary information from optical and SAR technologies, thereby improving segmentation accuracy in complex backgrounds.

[0056] Therefore, MPF independently performs multimodal feature fusion at each level (Stage 1-4). Stage 1 fuses large-scale features to identify mangrove regions, Stage 2 fuses medium-scale features to identify mangrove patches, and Stages 3-4 fuse semantic features to understand mangrove categories. This progressive fusion strategy ensures that multi-scale information is fully utilized, gradually enhancing feature representations from shallow to deep layers.

[0057] In step S3, obtaining the fusion features corresponding to different levels includes:

[0058] Step S31: Align the optical coding features and SAR coding features corresponding to the same level through channels to obtain aligned optical coding features and SAR coding features. The level includes low-level level and deep-level level.

[0059] In this step, such as Figure 3 As shown, through the channel alignment module, that is, through Convolution maps the number of channels of optically coded features and SAR-coded features to a unified value. Eliminate dimensional differences between modalities. Indicates the embedding dimension.

[0060] Step S32: The aligned optical coding features and SAR coding features are respectively enhanced by the hierarchical boundary enhancement module to obtain optical edge enhancement features and SAR edge enhancement features at different levels;

[0061] In this step, such as Figure 3 As shown, the Hierarchical Boundary Enhancement (HBE) module comprises a Boundary Awareness (BAM) module and a Boundary Guided Fusion (BGF) module connected sequentially. In the case of a shallow layer, the output features of BAM are used as the output features of the HBE module to obtain the corresponding optical edge enhancement features and SAR edge enhancement features. Specifically, BAM extracts edge features through a gated channel transformation mechanism combined with depthwise separable convolution, explicitly extracting edge information from high-frequency textures for subsequent fusion.

[0062] When dealing with deep hierarchical structures, the output features of BGF are used as the output features of the hierarchical boundary enhancement module to obtain the corresponding optical edge enhancement features and SAR edge enhancement features. Specifically, a two-step boundary enhancement strategy is adopted: first, BAM is used to extract edge features (optical edge features and SAR edge features) from the original semantic features (aligned optical coding features and SAR coding features); then, BGF is applied. This module receives the original semantic features and the extracted edge features as dual inputs, and through depthwise separable convolution, coordinate attention mechanism, and thinning fusion, edge-guided semantic feature fusion is achieved, effectively preventing boundary blurring during downsampling and improving the accuracy of segmentation boundaries.

[0063] In step S32, the specific steps for feature enhancement performed by the hierarchical boundary enhancement module are as follows:

[0064] Step S321: Obtain the hierarchy of the input features of the hierarchical boundary enhancement module;

[0065] Step S322: The input features of the hierarchical boundary enhancement module are processed by the boundary perception module to obtain edge features, which are the output features of the lower-level hierarchical boundary enhancement module.

[0066] In this step, the input features of the layer boundary enhancement module are aligned optically coded features or SAR coded features.

[0067] Step S322, the feature edge detection through the boundary awareness module includes:

[0068] Step A1: Perform gated channel transformation on the input features of the boundary sensing module to obtain channel weights;

[0069] In this step, global context modeling is performed on the input features of BAM, channel statistics are calculated using global average pooling, and then channel weights are generated through a gating mechanism to recalibrate the feature channels and highlight important channels.

[0070] The formula for calculating channel weight is:

[0071] ;

[0072] In the formula, Indicates channel weight, This represents the Sigmoid activation function. and Both indicate the former. Represents the ReLU activation function. Indicates global average pooling. This represents the input features of the boundary sensing module.

[0073] Step A2: Generate high-frequency spatial edge features from the input features of the boundary awareness module through depthwise separable convolution;

[0074] In this step, depthwise separable convolutions are used in parallel to process the input features of BAM, and their lightweight characteristics are used to extract spatial high-frequency edge features.

[0075] Step A3: Adjust the channel weights of the spatial high-frequency edge features using the channel weights to obtain the output features of the boundary perception module.

[0076] In this step, the channel weights of the spatial high-frequency edge features are adjusted by channel weighting, and residual connections are performed with the input features of BAM to obtain the output features of BAM:

[0077] ;

[0078] In the formula, This represents the output features of the boundary sensing module. Indicates channel weight, This represents depthwise separable convolution. This represents the input features of the boundary sensing module.

[0079] Step S323: The edge features and the input features of the hierarchical boundary enhancement module are fused by the boundary-guided fusion module to obtain the edge fusion features, which are the output features of the deep-level hierarchical boundary enhancement module.

[0080] In step S323, the edge-guided semantic feature fusion via the boundary-guided fusion module includes:

[0081] Step B1: After concatenating the first and second input features of the boundary-guided fusion module, perform depthwise separable convolution to obtain the first fused feature. The first input feature is the input feature of the boundary-aware module.

[0082] In this step, the edge features output by BAM are used as the second input features of BGF. BGF is deployed in a deep network to fuse the edge features extracted by BAM back into the semantic features, preventing boundary blurring caused by downsampling.

[0083] Specifically, the input features of BAM and BAM output characteristics Preliminary fusion was performed, and the first fusion feature was obtained:

[0084] ;

[0085] In the formula, Indicates the first fusion feature, This represents depthwise separable convolution. This indicates splicing along the channel dimension. This represents the output features of the boundary sensing module. This represents the input features of the boundary sensing module.

[0086] Step B2: Perform average pooling on the first fused feature in both the horizontal and vertical directions to generate horizontal pooling features and vertical pooling features;

[0087] In this step, along the horizontal direction ( Direction) and vertical direction ( Average pooling is performed to obtain the corresponding horizontal pooling features and vertical pooling features:

[0088] ;

[0089] ;

[0090] In the formula, Indicates at altitude Location, passage The horizontal pooling vector at that position, Indicates the width of the feature map. express In the passage ,high and width Pixel value at that location, Indicates the width Location, passage The vertical pooling vector at that point. Indicates the feature map height. express aisle ,high and width The pixel value at that location.

[0091] pass and Obtain the corresponding horizontal pooling features and vertical pooling features .

[0092] Step B3: After concatenating the horizontal pooling features and the vertical pooling features, generate horizontal intermediate features and vertical intermediate features through the Split operation;

[0093] Step B4: Generate corresponding horizontal attention weights and vertical attention weights using the horizontal intermediate features and the vertical intermediate features;

[0094] Step B5: Multiply the horizontal attention weights and the vertical attention weights pixel by pixel to obtain the attention map;

[0095] In this step, the formula for obtaining the attention map is:

[0096] ;

[0097] ;

[0098] In the formula, and These represent the horizontal and vertical intermediate features, respectively. This indicates a Split operation, which splits a feature along the channel dimension into two equal-dimensional features. Represents the ReLU activation function. This represents the convolution operation. This represents an attention map. This represents the Sigmoid activation function. and These represent horizontal convolution operations and vertical convolution operations, respectively.

[0099] Step B6: After performing pixel-by-pixel multiplication between the attention map and the first fused feature, refine the result through convolution to obtain the second fused feature;

[0100] Step B7: After performing residual connection between the second fused feature and the first input feature, the output feature of the boundary-guided fusion module is obtained.

[0101] In this step, the output features of the boundary-guided fusion module are obtained as follows:

[0102] ;

[0103] In the formula, This indicates the output characteristics of the boundary-guided fusion module. express convolution, Indicates the first fusion feature, This represents a pixel-wise dot product operation. This represents an attention map. This represents the input features of the boundary sensing module.

[0104] Step S33: Spatial domain fusion of the optical edge enhancement features and the SAR edge enhancement features to obtain spatial domain fusion features at different levels;

[0105] In this step, spatial domain fusion is performed through the spatial domain fusion module.

[0106] Step S33 includes:

[0107] Step S331: For the shallow layer, the optical edge enhancement features and the SAR edge enhancement features are weighted and summed to obtain the spatial domain fusion features under the shallow layer.

[0108] In this step, a weighted summation is used for the shallow layers, and the formula is:

[0109] ;

[0110] In the formula, This indicates the spatial domain fusion characteristics at the shallow level. Indicates the first fusion weight. Indicates optical edge enhancement features, This indicates SAR edge enhancement features.

[0111] Step S332: For deep layers, a cross-modal attention mechanism is introduced to fuse the optical edge enhancement features and the SAR edge enhancement features to obtain spatial domain fusion features at deep layers.

[0112] In this step, we aim to preserve the rich color and spectral semantics of the optical imagery as the main structure. Therefore, we use optical edge enhancement features to calculate query features, representing the query intent. SAR imagery, on the other hand, is sensitive to texture and geometric structure (high penetration) and is used as a reference frame to match blurred areas in the optical features. Therefore, we use SAR edge enhancement features to calculate index features, representing index information. Simultaneously, content features also originate from SAR edge enhancement features, representing content information—complementary details that are actually extracted and injected into the optical features (such as terrain structures obscured by clouds and fog).

[0113] Specifically, for deeper layers, edge enhancement features are calculated using optical edge enhancement features and SAR edge enhancement features. , and :

[0114] ;

[0115] ;

[0116] ;

[0117] In the formula, , and These represent query features, index features, and content features, respectively. Indicates optical edge enhancement features, Indicates SAR edge enhancement features, , and Both represent the first projection weight.

[0118] To preserve the integrity of the original optical features and prevent network degradation, residual connections are also performed to obtain spatial domain fusion features at deeper levels:

[0119] ;

[0120] ;

[0121] In the formula, This indicates the spatial domain fusion characteristics at deeper levels. Indicates optical edge enhancement features, This represents cross-modal attention. Represents the normalized exponential function, express transpose, , and These represent query features, index features, and content features, respectively. This represents the first scaling factor.

[0122] Understandably, spatial domain integration also employs different integration strategies at different levels.

[0123] For shallow layers, element-wise addition and convolutional normalization are used directly. For deep layers, cross-modal attention machines are used for learning, focusing on complementary information from different modalities in two directions to ensure that information from both modalities is fully utilized, and feature representation capabilities are enhanced through bidirectional fusion.

[0124] Step S34: The optical edge enhancement feature and the SAR edge enhancement feature are fused in the frequency domain to obtain frequency domain fused features at different levels;

[0125] In this step, such as Figure 3 As shown, frequency domain fusion is performed through the frequency domain fusion module.

[0126] Step S34 includes:

[0127] Step S341: At each level, the optical edge enhancement feature and the SAR edge enhancement feature are subjected to discrete wavelet decomposition to obtain the low-frequency approximate components and high-frequency components under the optical mode and the SAR mode, respectively. The high-frequency components include horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components.

[0128] In this step, the two-dimensional Haar wavelet transform (DWT) is used to decompose the optical edge enhancement features and SAR edge enhancement features into four frequency band sub-maps, namely the low-frequency approximate components ( ), horizontal high-frequency components ( ), vertical high-frequency components ( ) and diagonal high-frequency components ( ), and the dimensions of each frequency band sub-map after decomposition are all .

[0129] Step S342: Perform cross-modal attention fusion using the low-frequency approximation components in the optical and SAR modes to obtain low-frequency fusion features;

[0130] In this step, the low-frequency approximation component contains the main semantic structure and energy of the image. Therefore, the low-frequency approximation component in the optical mode is used as the main component, and the low-frequency approximation component in the SAR mode is used to supplement the occluded semantic information, and cross-modal attention fusion is performed:

[0131] ;

[0132] ;

[0133] In the formula, Indicates low-frequency fusion characteristics, This represents the low-frequency approximate component in the optical mode. This represents the low-frequency approximate component under the SAR mode. Represents the normalized exponential function, , and These represent low-frequency query characteristics, low-frequency index characteristics, and low-frequency content characteristics, respectively. , and Both represent the second projection weights. Indicates the second scaling factor. express The transpose of .

[0134] Step S343: Take the maximum value of each high-frequency component under the optical mode and SAR mode to obtain the corresponding horizontal high-frequency features, vertical high-frequency features and diagonal high-frequency features;

[0135] In this step, the high-frequency components retain a lot of edge details, so a maximum value strategy is adopted to retain the most significant texture and boundary response in both modes.

[0136] Step S344: The low-frequency fusion feature, the horizontal high-frequency feature, the vertical high-frequency feature, and the diagonal high-frequency feature are reconstructed using two-dimensional inverse discrete wavelet transform to obtain frequency domain fusion features at different levels.

[0137] In this step, reconstruction is performed using the two-dimensional inverse discrete wavelet transform (IDWT), and the size of the reconstructed frequency domain fused features is restored to its original value. It includes the semantic consistency of optics and the structural texture details of SAR.

[0138] Frequency domain fusion features are:

[0139] ;

[0140] In the formula, Indicates frequency domain fusion characteristics, This represents the two-dimensional inverse discrete wavelet transform. , , and These represent low-frequency fusion characteristics, horizontal high-frequency characteristics, vertical high-frequency characteristics, and diagonal high-frequency characteristics, respectively.

[0141] Step S35: Perform feature fusion on the spatial domain fusion feature and the frequency domain fusion feature to obtain fusion features corresponding to different levels.

[0142] In this step, feature fusion is performed through the refinement module to obtain fused features:

[0143] ;

[0144] In the formula, Indicates fusion characteristics, express convolution, Indicates spatial domain fusion characteristics, Indicates frequency domain fusion characteristics, This represents the second fusion weight.

[0145] Understandably, if fusion is performed only in the spatial domain, there is a problem of limited information representation. However, combining spatial and frequency domain fusion achieves a synergistic effect, significantly improving the accuracy and effectiveness of semantic segmentation of multimodal remote sensing images through multi-dimensional and multi-level information fusion, particularly in boundary accuracy and detail preservation. Specifically, the combination of spatial and frequency domain fusion can simultaneously capture both spatial and frequency information, resulting in richer and more comprehensive feature representations, and the fusion of information from different dimensions enhances model robustness.

[0146] Step S4: Decode the features based on the fusion features corresponding to different levels to obtain the decoded features;

[0147] In this step, the feature decoding is performed using the decoder of the Feature Pyramid Network (FPN).

[0148] Understandably, the MPF outputs fused features at four levels, namely... , , and Corresponding to Stages 1-4, with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, and all having the same number of channels. .

[0149] The decoder uses a top-down path to start from the deepest layer. Starting from, for Bilinear interpolation upsampling is performed to increase the resolution from 1 / 32 to 1 / 16, resulting in the fourth upsampled feature corresponding to Stage 4. Then and After element-wise addition, smoothing (3×3 smooth convolution) is performed to eliminate aliasing effects (such as jagged edges) caused by upsampling, resulting in the third fusion feature corresponding to Stage 3. .

[0150] right Bilinear interpolation upsampling is performed, increasing the resolution to 1 / 8, to obtain the third upsampled feature corresponding to Stage 3. Then and After element-wise addition, smoothing (3×3 smooth convolution) is performed to eliminate aliasing effects (such as jagged edges) caused by upsampling, resulting in the second fused feature corresponding to Stage 2. .

[0151] right Bilinear interpolation upsampling is performed, increasing the resolution to 1 / 4, to obtain the second upsampled feature corresponding to Stage 2. Then and After element-wise addition, smoothing (3×3 smooth convolution) is performed to eliminate aliasing effects (such as jagged edges) caused by upsampling, resulting in the first fused feature corresponding to Stage 1. .

[0152] Then to Multiple rounds of bilinear interpolation upsampling are performed to match the resolution of the original remote sensing image, resulting in decoded features. As the output feature of the decoder, each layer is fused and then convolved for smoothing to eliminate aliasing.

[0153] Step S5: Construct a semantic segmentation model for mangrove remote sensing images based on the decoded features, and output the semantic segmentation result of the target mangrove remote sensing image through the mangrove remote sensing image semantic segmentation model.

[0154] In step S5, constructing a semantic segmentation model for mangrove remote sensing images based on the decoded features includes:

[0155] Step S51: Construct a dual-branch encoder that takes multimodal remote sensing images as input and multiple levels of coding features as output;

[0156] Step S52: Construct a multimodal pyramid fusion module that takes multi-level encoded features as input and multi-level fused features as output;

[0157] Step S53: Construct a decoder that takes multi-level fused features as input and decoded features as output;

[0158] Step S54: Construct a prediction module that takes decoded features as input and remote sensing image semantic segmentation results as output;

[0159] In this step, the prediction module will convert the input... Applying 1×1 convolution, the number of channels is reduced from... Compressed to the number of categories This yields an unnormalized fractional graph.

[0160] The unnormalized fractional map is normalized along the channel dimension using the Softmax function to obtain a category probability map with the same resolution as the original image. Then, the category probability map is transformed into a pixel-level segmentation map by a threshold, thus obtaining the semantic segmentation result of the remote sensing image.

[0161] Step S55: A semantic segmentation model for mangrove remote sensing images is constructed using the dual-branch encoder, the multimodal pyramid fusion module, the decoder, and the prediction module.

[0162] Therefore, the mangrove remote sensing image semantic segmentation model of this invention (MPF-Swin model for short) adopts an end-to-end architecture of dual-stream coding, multi-level fusion, and unified decoding. The MPF-Swin model can simultaneously receive multispectral optical images (10 bands) and SAR images (3 channels). A dual-branch encoder with two parallel coding branches independently extracts multi-scale features from both modalities. Simultaneously, the core MPF is constructed, performing dual fusion of spatial and frequency domains and boundary enhancement at each feature level. Finally, a feature pyramid network is used for top-down feature aggregation to generate the final pixel-level segmentation map.

[0163] Simultaneously, before performing semantic segmentation of mangrove remote sensing images, the MPF-Swin model needs to be trained. During training, the training data for the two modalities used need to be preprocessed. First, the raw data, namely SAR radar data and optical data, are obtained. After processing the SAR radar data and optical data, optical images and SAR images that can be directly input into the MPF-Swin model can be obtained.

[0164] For SAR radar data, terrain correction is required. Due to the characteristics of radar side-looking imaging, terrain undulations can cause severe geometric and radiometric distortions. Terrain correction achieves correction based on a volumetric model, obtaining the basic scattering characteristics. Specifically, a digital elevation model is used to calculate the slope, and the angle is converted to radians. Then, a simplified volumetric scattering model is applied for correction, dividing the original VV and VH bands by the cosine of the slope to generate the corrected VV and VH bands. Based on the above processing, extremely small slope values ​​(<0.01) are corrected to 0.01 to prevent division by zero errors.

[0165] Based on the corrected VV and VH bands, enhanced ground features including multiple radar indices are constructed, such as depolarization rate, ratio index, radar vegetation index, and normalized difference index.

[0166] Then, based on the corrected bands, spatial texture features were extracted using GLCM (Gray-Level Co-occurrence Matrix), which is very effective in distinguishing uniform regions from textured regions caused by SAR speckle noise. Specifically, floating-point data was converted to Int16 integers. Then, texture was calculated using three window sizes, and various texture indices were calculated for different bands, including angular second moment, contrast, correlation, variance, inverse moment, entropy, and dissimilarity.

[0167] By integrating data from the entire year, time-series statistics of basic scattering features, enhanced ground features, and spatial texture features are calculated, including minimum, maximum, mean, standard deviation, percentiles, range, and interquartile median.

[0168] The basic scattering features, enhanced ground features, and spatial texture features, along with their temporal statistics, are combined into a complete feature set. Then, through mean synthesis denoising and feature selection and condensation, one type of core stable information is extracted from each of the three types of features, ultimately forming a 3-channel SAR image, with each channel corresponding to one type of core feature.

[0169] For optical data, clean, cloud-free periodic mosaics are generated through multi-source data coordination, cloud removal, and quarterly time-series synthesis.

[0170] Specifically, the vector file of the study area is loaded and converted to the WGS84 coordinate system. Then, the complex polygons in the vector file are merged to clarify the scope of the study area and avoid confusion in the edge data.

[0171] Secondly, based on the merged data, images with a cloud cover ratio exceeding 30% are filtered out. Then, using the QA60 band (quality assessment band) of the images, opaque clouds and cirrus clouds are identified through bitwise operations, retaining only pixels with both bits set to 0 (i.e., cloudless areas). By numerically scaling the images obtained after cloud removal, their pixel values ​​are converted to reflectance values ​​between 0 and 1.

[0172] Based on the scaled image, 10 core bands were selected, covering different spectral ranges, including visible light bands (band B2, band B3, band B4), red edge bands (band B5, band B6, band B7), near-infrared bands (band B8, band B8A) and short-wave infrared bands (band B11, band B12).

[0173] For each quarter's imagery, a median algorithm is used to synthesize a single quarterly image, removing residual cloud shadows, noise, and outliers to generate a cleaner base map than the average composite. Finally, the quarterly images are cropped using vector files to obtain corresponding 10-band optical images.

[0174] It should be noted that in practical applications, data preprocessing is also required. After obtaining optical and SAR images, they are then input into the MPF-Swin model for image segmentation.

[0175] In this embodiment, multimodal mangrove remote sensing data from a specific region (Gaoqiao Mangrove Nature Reserve) were also used for experiments, and the results are as follows: Figure 4 As shown in Table 1, this diagram compares the segmentation results of the MPF-Swin model with those of other models, and provides a comparison table of experimental results. Figure 4 In this context, "optical" represents the optical mode, and "SAR" represents the SAR mode.

[0176] Table 1 Comparison of experimental results for mangrove datasets

[0177]

[0178] Table 1 shows a comparison of different models in terms of accuracy, mean intersection-over-union ratio (mIoU), and specific indicators for background and mangrove land cover.

[0179] Compared to single-modal methods, the MPF-Swin model, which incorporates multimodal data, achieves significant improvements across all metrics. In particular, compared to the single-modal baseline Swin-Transformer model with the same architecture, the MPF-Swin model's mIoU increases from 85.02% to 90.59%, validating the necessity of fusing SAR and optical data.

[0180] The MPF-Swin model performs particularly well in the key mangrove category. Its IoU for mangroves reaches 85.24%, significantly higher than the U-Net model's 58.14% and also significantly better than the Swin-Transformer model's 76.96%. This demonstrates that the MPF-Swin model greatly improves the issues of missed and false detections of mangroves.

[0181] Secondly, in the comparison within the multimodal method group, the MPF-Swin model also outperformed the DDHRNet, MCANet, and RGB-X models. In particular, it achieved an F1 score of 92.03% for mangroves, demonstrating the architectural advantages of the MPF-Swin model in handling class imbalance and complex feature extraction.

[0182] and Figure 4 The visualization results intuitively demonstrate the mangrove extraction performance of each model in different scenarios. Figure 4 The samples in columns 3 and 4 show that mangroves grow in a complex intertidal zone with fragmented edges and small channels. The results from the U-Net and Deeplabv3+ models exhibit significant adhesion, failing to distinguish subtle tidal channels and displaying severe jagged edges. While the Swin-Transformer model outperforms convolutional networks, it still suffers from blurring at mangrove boundaries. The MPF-Swin model's extraction results are highly consistent with the Ground Truth model. It successfully preserves the fine water system structure within the mangroves (i.e., the gaps in the mangrove patches), with smooth and clear edges, demonstrating a strong ability to capture detail.

[0183] Meanwhile, in the first column of samples, background areas (such as water bodies or non-vegetated areas) are easily interfered with by light reflection from optical images. Single-modal models are prone to generating scattered false detection noise. The MPF-Swin model combines the sensitivity of SAR data to texture and structure, effectively eliminating these false targets, and generating more dense and uniform patches with reduced fragmentation and voids.

[0184] Therefore, the MPF-Swin model designed in this invention exhibits higher robustness in mangrove identification and complex background processing. Experimental results comparing different unimodal and multimodal methods show that the MPF-Swin model outperforms existing technologies in mangrove segmentation, further validating its advantages in specific application scenarios.

[0185] Example 2:

[0186] This embodiment provides a semantic segmentation system for mangrove remote sensing images based on multimodal data. The system includes:

[0187] The acquisition unit is used to acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images;

[0188] The coding unit is used to perform multi-level dual-branch coding on the multimodal remote sensing image to obtain coding features corresponding to multiple levels, including optical coding features and SAR coding features.

[0189] The fusion unit is used to fuse the optical coding features and the SAR coding features corresponding to the same level, and combine channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion to obtain fused features corresponding to different levels.

[0190] The decoding unit is used to perform feature decoding based on the fusion features corresponding to different levels to obtain decoded features;

[0191] The semantic segmentation unit is used to construct a semantic segmentation model for mangrove remote sensing images based on the decoded features, and output the semantic segmentation result of the target mangrove remote sensing image through the mangrove remote sensing image semantic segmentation model.

[0192] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0193] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic segmentation method for mangrove remote sensing images based on multimodal data, characterized in that, include: Acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images; The multimodal remote sensing image is subjected to multi-level dual-branch coding to obtain coding features corresponding to multiple levels. The coding features include optical coding features and SAR coding features. The optical coding features and SAR coding features corresponding to the same level are fused together, and combined with channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion, fused features corresponding to different levels are obtained. Based on the fusion features corresponding to different levels, feature decoding is performed to obtain decoded features; Based on the decoding features, a semantic segmentation model for remote sensing images of mangroves is constructed, and the semantic segmentation results of remote sensing images of the target mangroves are output through the semantic segmentation model for remote sensing images of mangroves.

2. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 1, characterized in that, The step of performing multi-level dual-branch coding on the multimodal remote sensing image to obtain coding features corresponding to multiple levels includes: A coding layer is constructed, which is obtained by building multiple Swin Transformer modules; An optical coding branch and a SAR coding branch are constructed by sequentially connecting patch embedding modules and multiple coding layers, and the optical coding branch and the SAR coding branch have the same structure. The optical image is feature-encoded through the optical coding branch to obtain optical coding features corresponding to multiple levels; The SAR image is feature-encoded through the SAR coding branch to obtain SAR coding features corresponding to multiple levels.

3. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 1, characterized in that, The obtained fusion features corresponding to different levels include: The optical coding features and SAR coding features corresponding to the same level are aligned by channel to obtain aligned optical coding features and SAR coding features. The level includes low-level level and deep-level level. The aligned optically coded features and SAR coded features are respectively enhanced by the hierarchical boundary enhancement module to obtain optical edge enhancement features and SAR edge enhancement features at different levels; The optical edge enhancement features and the SAR edge enhancement features are spatially fused to obtain spatially fused features at different levels. The optical edge enhancement features and the SAR edge enhancement features are fused in the frequency domain to obtain frequency domain fused features at different levels. The spatial domain fusion feature and the frequency domain fusion feature are fused to obtain fusion features corresponding to different levels.

4. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 3, characterized in that, The specific steps for feature enhancement performed by the hierarchical boundary enhancement module are as follows: Obtain the hierarchy of the input features for the hierarchical boundary enhancement module; The input features of the hierarchical boundary enhancement module are processed by the boundary perception module to obtain edge features, which are the output features of the lower-level hierarchical boundary enhancement module. The edge features and the input features of the layered boundary enhancement module are fused by the boundary-guided fusion module to obtain edge fusion features, which are the output features of the deep-level layered boundary enhancement module.

5. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 4, characterized in that, The feature edge detection through the boundary awareness module includes: The input features of the boundary sensing module are subjected to gated channel transformation to obtain channel weights; The input features of the boundary awareness module are used to generate high-frequency spatial edge features through depthwise separable convolution; The channel weights of the spatial high-frequency edge features are adjusted by the channel weights to obtain the output features of the boundary perception module.

6. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 4, characterized in that, The edge-guided semantic feature fusion via the boundary-guided fusion module includes: After concatenating the first and second input features of the boundary-guided fusion module, a depthwise separable convolution is performed to obtain the first fused feature, where the first input feature is the input feature of the boundary-aware module. The first fused feature is average pooled in both the horizontal and vertical directions to generate horizontal pooling features and vertical pooling features. After concatenating the horizontal pooling features and the vertical pooling features, a split operation is used to generate horizontal intermediate features and vertical intermediate features; The horizontal and vertical attention weights are generated using the horizontal and vertical intermediate features. An attention map is obtained by multiplying the horizontal attention weights and the vertical attention weights pixel by pixel. After performing a pixel-by-pixel multiplication between the attention map and the first fused feature, the second fused feature is obtained by convolution. After performing a residual connection between the second fusion feature and the first input feature, the output feature of the boundary-guided fusion module is obtained.

7. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 3, characterized in that, The step of spatially fusing the optical edge enhancement features and the SAR edge enhancement features to obtain spatial domain fused features at different levels includes: For the shallow layer, the optical edge enhancement features and the SAR edge enhancement features are weighted and summed to obtain the spatial domain fusion features at the shallow layer. For deeper layers, a cross-modal attention mechanism is introduced to fuse the optical edge enhancement features and the SAR edge enhancement features to obtain spatial domain fusion features at deeper layers.

8. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 3, characterized in that, The step of fusing the optical edge enhancement features and the SAR edge enhancement features in the frequency domain to obtain frequency domain fused features at different levels includes: At each level, the optical edge enhancement features and the SAR edge enhancement features are subjected to discrete wavelet decomposition to obtain low-frequency approximate components and high-frequency components in the optical mode and SAR mode, respectively. The high-frequency components include horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components. Low-frequency fusion features are obtained by cross-modal attention fusion of low-frequency approximate components under optical and SAR modes; For each high-frequency component, the maximum value under the optical mode and SAR mode is taken to obtain the corresponding horizontal high-frequency characteristics, vertical high-frequency characteristics and diagonal high-frequency characteristics; The low-frequency fusion features, the horizontal high-frequency features, the vertical high-frequency features, and the diagonal high-frequency features are reconstructed using two-dimensional inverse discrete wavelet transform to obtain frequency domain fusion features at different levels.

9. The semantic segmentation method for mangrove remote sensing images based on multimodal data according to claim 1, characterized in that, The construction of a semantic segmentation model for mangrove remote sensing images based on the decoded features includes: Construct a dual-branch encoder that takes multimodal remote sensing images as input and multiple levels of coding features as output; Construct a multimodal pyramid fusion module that takes multi-level encoded features as input and multi-level fused features as output; Construct a decoder that takes multi-level fused features as input and decoded features as output; Construct a prediction module that takes decoded features as input and remote sensing image semantic segmentation results as output; A semantic segmentation model for mangrove remote sensing images is constructed using the dual-branch encoder, the multimodal pyramid fusion module, the decoder, and the prediction module.

10. A semantic segmentation system for mangrove remote sensing images based on multimodal data, characterized in that, include: The acquisition unit is used to acquire multimodal remote sensing images of the target mangrove forest, including optical images and SAR images; The coding unit is used to perform multi-level dual-branch coding on the multimodal remote sensing image to obtain coding features corresponding to multiple levels, the coding features including optical coding features and SAR coding features; The fusion unit is used to fuse the optical coding features and the SAR coding features corresponding to the same level, and combine channel alignment, hierarchical boundary enhancement, spatial domain fusion and frequency domain fusion to obtain fused features corresponding to different levels. The decoding unit is used to perform feature decoding based on the fusion features corresponding to different levels to obtain decoded features; The semantic segmentation unit is used to construct a semantic segmentation model for mangrove remote sensing images based on the decoded features, and output the semantic segmentation result of the target mangrove remote sensing image through the mangrove remote sensing image semantic segmentation model.

Images