A multi-modal remote sensing image semantic segmentation method based on double-layer wavelet hybrid expert and cross-branch state fusion

Abstract

This invention discloses a multimodal remote sensing image semantic segmentation method using a two-layer wavelet hybrid expert and cross-branch state fusion approach, comprising the following steps: acquiring spatially registered optical and auxiliary modal images to form a dual-modal input; extracting four-stage features via a two-branch encoder, performing channel recalibration, shared mapping, and residual enhancement on the first three stages; aligning and stitching the features of the second to fourth stages to generate gating weights, which are then cross-gated and fused to obtain unified deep features; executing two-layer wavelet hybrid expert and state-space alternating modulation, with each layer performing wavelet decomposition on the input features to generate three types of cues and driving corresponding expert weighted fusion, which is then multi-directional state-space bidirectional modulation with the first-stage enhanced features, and outputting to the next layer; decoding and fusing the second-layer output to obtain the semantic segmentation result. This remote sensing segmentation method can improve the segmentation accuracy of target boundaries, modal anomaly regions, small-scale features, and shared backgrounds in complex scenes.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent processing of remote sensing images, and in particular to a multimodal remote sensing image semantic segmentation method that combines two-layer wavelet hybrid expert and cross-branch state fusion. Background Technology

[0002] Semantic segmentation of remote sensing images refers to assigning corresponding land cover category labels to each pixel in a remote sensing image, thereby obtaining the spatial distribution results of targets such as roads, buildings, water bodies, vegetation, and vehicles. It has significant application value in land cover analysis, urban planning, agricultural monitoring, environmental monitoring, and disaster assessment. With the development of Earth observation technology, multi-source remote sensing data, such as optical images, synthetic aperture radar images, digital surface models, and thermal infrared images, are becoming increasingly abundant.

[0003] Existing semantic segmentation methods for remote sensing images still have many shortcomings. While existing multimodal segmentation schemes can cross-fuse optical and auxiliary modal images, they typically focus on the interaction of salient features at a certain level and have not yet implemented stage-by-stage consistency constraints on the modal distribution differences already existing in shallow and mid-level features. This makes it difficult to eliminate modal shifts while preserving the unique local structures of each modality. Existing wavelet transform-based feature processing methods mainly use preset weights to fuse different frequency bands. They cannot dynamically select appropriate feature processing paths for different pixels based on the spatial differences of target boundary regions, modal inconsistency regions, and shared background regions, thus limiting the adaptability of wavelet transform in complex remote sensing scenarios. Furthermore, conventional skip connections usually fuse shallow and deep features by splicing or element-wise addition. Deep semantics are insufficient in suppressing redundant information in shallow details, and the reverse correction of shallow structure for the localization of deep semantic boundaries is also insufficient, making it difficult to balance edge fineness and regional consistency in the segmentation results.

[0004] To address the aforementioned problems, this invention provides a remote sensing image segmentation method based on dual-layer wavelet hybrid expert and cross-branch state space alternating modulation. This method improves the ability to differentiate target boundaries, modal anomaly regions, small-scale features, and shared backgrounds in complex scenes. Summary of the Invention

[0005] In view of the problems existing in the above-mentioned remote sensing image semantic segmentation methods, this invention is proposed.

[0006] Therefore, the purpose of this invention is to provide a multimodal remote sensing image semantic segmentation method that combines two-layer wavelet hybrid expert and cross-branch state fusion. The aim is to achieve differentiated and accurate segmentation of target boundaries, modal anomaly regions, small-scale features and shared backgrounds in multimodal remote sensing images under complex scenes by using shallow and mid-layer stage-by-stage modal alignment, wavelet decomposition-driven pixel-by-pixel expert routing, and bidirectional interaction between deep semantics and shallow details.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution, including the following steps:

[0008] Acquire spatially registered optical remote sensing images and auxiliary modal remote sensing images, adjust them to the same spatial size, and form a dual-modal input.

[0009] The dual-modal input is fed into two branch encoders to extract four-stage hierarchical features. The first three stages of features are then subjected to channel recalibration, shared parameter mapping, and intramodal residual enhancement to obtain dual-modal enhanced features.

[0010] The second to fourth stage features of the two branches are scale aligned and spliced ​​projection to form multi-scale aggregated features of each modality. Spatial gating weights and modal gating weights are generated based on the multi-scale aggregated features. Unified deep features are obtained through cross-gating fusion.

[0011] The unified deep features are subjected to dual-layer wavelet hybrid expert enhancement and cross-branch state space alternating modulation, including two layers of cascaded processing units. Two-dimensional discrete wavelet decomposition is performed on the input features of each layer. Based on the decomposition results, structural cues, abnormal residual cues, and shared background cues are generated. The three types of cues drive the corresponding three experts to perform weighted fusion to obtain expert-enhanced features. The expert-enhanced features are then subjected to cross-branch multi-directional state space bidirectional modulation with the first-stage dual-modal enhancement features to obtain the current layer modulation features, which are then input to the next layer.

[0012] The modulation features output by the second-layer processing unit are decoded and fused to output pixel-level semantic segmentation results.

[0013] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the following steps are performed: Each layer of input features is decomposed into low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands, and diagonal high-frequency sub-bands using two-dimensional discrete wavelet decomposition; a low-frequency calibration mask is generated based on the low-frequency sub-bands using lightweight channel projection and a Sigmoid activation function; the low-frequency calibration mask is multiplied element-wise with the low-frequency sub-bands to obtain low-frequency residuals; and the low-frequency residuals are superimposed on the low-frequency sub-bands to obtain calibrated low-frequency features; channel stitching and convolutional projection are performed on the three high-frequency sub-bands to obtain high-frequency aggregated features.

[0014] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the calibration low-frequency features and the high-frequency aggregated features are input into the inverse discrete wavelet transform for spatial reconstruction, and the reconstruction result is channel aligned by the first pointwise convolution. The channel aligned result is then locally spatially mixed by the depthwise convolution with a kernel size of 7×7. Finally, wavelet enhancement features are obtained through the second pointwise convolution and normalization processing.

[0015] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the following steps are performed: the high-frequency aggregated features are convolved pointwise to obtain the structural cues; the low-frequency subbands are convolved and projected to obtain shared baseline features; the element-wise absolute difference between the wavelet enhancement features and the shared baseline features is calculated; and the abnormal residual cues are obtained by sequentially performing pointwise convolution, depthwise convolution, and channel attention processing; and the shared background cues are obtained by performing pointwise convolution on the shared baseline features.

[0016] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the three experts include a structural expert, an anomaly residual expert, and a shared background expert; the structural expert includes a 3×3 depthwise convolutional branch and a 5×5 depthwise convolutional branch set in parallel, and the outputs of the two branches are spliced ​​and projected and added to the residual of the wavelet enhancement feature; the anomaly residual expert injects the anomaly residual cue into the wavelet enhancement feature; the shared background expert splices the 7×7 depthwise convolution result of the wavelet enhancement feature with the shared reference feature, and after pointwise convolution and channel recalibration, adds it to the residual of the wavelet enhancement feature.

[0017] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the wavelet enhancement features, the structural cue, the anomaly residual cue, and the shared background cue are concatenated into channels, and spatial routing values ​​corresponding to three experts are generated sequentially through pointwise convolution, 3×3 depthwise convolution, and pointwise convolution; the spatial routing values ​​are normalized at each pixel position using a Softmax function with a temperature parameter; and the outputs of the three experts are weighted and summed pixel by pixel according to the normalized routing weights to obtain adaptive expert fusion features.

[0018] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the adaptive expert fusion feature is concatenated and convolved with the wavelet enhancement feature for thinning, and then added to the residual of the current layer input feature to obtain the expert enhancement feature.

[0019] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the cross-branch multi-directional state space bidirectional modulation includes: taking the expert enhancement features as deep branches and the first-stage bimodal enhancement features as shallow branches; performing layer normalization and linear projection on the two branches respectively, so that they have the same spatial size and channel dimension; performing depth convolution on the two branches respectively and unfolding them into one-dimensional sequences along four directions, performing state space recursion processing and then aggregating them to obtain shallow state features and deep state features.

[0020] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the method comprises: generating shallow gating coefficients based on the deep branch input after layer normalization; generating deep gating coefficients based on the shallow branch input after layer normalization; multiplying the shallow gating coefficients element-wise with the shallow state features and the deep gating coefficients element-wise with the deep state features; performing linear output projection on the multiplication results and adding the residuals of the corresponding branch inputs; and concatenating and convolving the features obtained by adding the residuals of the two branches to obtain the current layer modulation features.

[0021] As a preferred embodiment of the multimodal remote sensing image semantic segmentation method of the present invention, the decoding fusion includes: convolving and refining the modulation features output by the second layer processing unit, and generating a semantic category prediction map consistent with the spatial size of the optical remote sensing image through a pixel classification prediction layer.

[0022] The beneficial effects of this invention are as follows: This method extracts bimodal four-stage hierarchical features through a dual-branch encoder, and performs channel recalibration, shared parameter mapping, and intramodal residual enhancement on the first three stages of features. It eliminates modal distribution differences at the shallow and mid-level feature levels while preserving the unique local structure of each modality. This method performs multi-scale aggregation on the second to fourth stages of features from both branches, and generates spatial gating weights and modal-level gating weights for cross-gating fusion, achieving an adaptive fusion strategy to suppress redundant responses and highlight key modal information. This method uses dual-layer wavelet-guided hybrid expert enhancement. In each layer, the input features are decomposed into two-dimensional discrete wavelet decomposition. Based on low-frequency and high-frequency sub-bands, structural cues, abnormal residual cues, and shared background cues are generated, driving the corresponding three experts to perform pixel-by-pixel route-weighted fusion. Based on the spatial differences between target boundaries, anomalies, and background regions, it dynamically selects processing paths for different pixels, overcoming the limitations of traditional wavelet transform-based preset weight fusion. This method alternates between wavelet hybrid expert enhancement and cross-branch state-space modulation in each layer, enabling local frequency band refinement and global context modeling to progress layer by layer and support each other. Furthermore, it balances edge refinement and regional consistency through bidirectional modulation of deep semantics and shallow details. This invention can effectively improve the differential segmentation accuracy of target boundaries, modal anomaly regions, small-scale features, and shared backgrounds in multimodal remote sensing images in complex scenes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A schematic diagram of the overall process of the remote sensing segmentation method with two-layer wavelet hybrid expert and cross-branch state fusion is shown;

[0025] Figure 2 A schematic diagram of the wavelet decomposition and reconstruction process is shown.

[0026] Figure 3 The diagram illustrates the three types of prompt generation and hybrid expert routing processes.

[0027] Figure 4 A schematic diagram of the cross-branch state space fusion decoding process is shown;

[0028] Figure 5 A schematic diagram of a four-directional two-dimensional state space scanning process is shown;

[0029] Figure 6 This diagram illustrates the differences in shallow and mid-level modalities, spatial regional heterogeneity, disconnect between shallow details and deep semantics, and limitations of fixed-frequency processing in the existing multimodal remote sensing image semantic segmentation process.

[0030] Figure 7 This is a schematic diagram of the overall structure of the multimodal remote sensing image semantic segmentation network of the present invention;

[0031] Figure 8 This diagram illustrates a comparison of the segmentation results of the method of this invention with different semantic segmentation models on multimodal remote sensing images. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0033] The terminology used in this invention is that which is currently widely used in the art in consideration of the function of the invention; however, these terms may vary according to the intent of those skilled in the art, precedent, or new technology in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the invention. Therefore, the terms used in this specification should not be construed as simple names, but rather based on their meanings and the overall description of the invention.

[0034] Example 1, referring to Figures 1-8 The first embodiment of the present invention provides a multimodal remote sensing image semantic segmentation method that combines two-layer wavelet hybrid expert and cross-branch state fusion. This method includes steps 100-500.

[0035] Step 100: Acquire spatially registered optical remote sensing images and auxiliary modal remote sensing images, adjust them to the same spatial size, and form a dual-modal input. The auxiliary modal remote sensing image is selected from digital surface model images, thermal infrared images, or synthetic aperture radar images. The optical remote sensing image and the auxiliary modal remote sensing image have been registered with geographic coordinates. After cropping and scaling them to the same spatial size, they are stitched together along the channel dimension to form a dual-modal input. Each training sample includes at least one optical remote sensing image, one auxiliary modal remote sensing image spatially registered with the optical remote sensing image, and one pixel-level semantic label image. Before entering the network, the optical image and the auxiliary modal image undergo geographic range matching, pixel size unification, and spatial position calibration to ensure that the same coordinate position in the two modalities corresponds to the same ground area.

[0036] Specifically, let the optical remote sensing image be... The auxiliary modal remote sensing image is ,in, The number of channels represents the auxiliary mode; and These represent the height and width of the input image, respectively. The auxiliary modality uses digital surface model images, thermal infrared images, or synthetic aperture radar images, which are adjusted to the same spatial size and then stitched together along the channel dimension to form a dual-modal input. Each training sample includes at least one optical remote sensing image, one auxiliary modality remote sensing image spatially registered with the optical remote sensing image, and one pixel-level semantic label map. The end-to-end semantic segmentation process is represented as follows:

[0037]

[0038] in, This represents the multimodal semantic segmentation network to be trained; Indicates the number of semantic categories to be identified; Indicates network output Channel pixel-level category score map or probability map.

[0039] Step 200: The dual-modal input is fed into the two-branch encoder to extract four-stage hierarchical features. The first three stages of features are subjected to channel recalibration, shared parameter mapping, and intramodal residual enhancement to obtain dual-modal enhanced features.

[0040] Specifically, the two-branch encoders receive optical images and auxiliary modal images respectively. Each encoding branch extracts four-stage features by progressively decreasing spatial resolution and progressively increasing the number of channels, represented as follows:

[0041]

[0042] in, Indicates modal index; Indicates optical modes; Indicates auxiliary modes; Indicates the first The encoder corresponding to each mode; This represents the input image for this modality; Indicates the first The mode in the th ... The features output by each encoding stage, among which The spatial resolutions of the features in the first three stages are one-half, one-quarter, and one-eighth of the original image, respectively. For the third stage... The modality of the first Characteristics of the stage Perform channel recalibration:

[0043]

[0044] in, , and They represent the first The number of channels, height, and width of the stage features; Indicates global average pooling; Represents the channel description vector; Indicates the kernel size as One-dimensional local cross-channel convolution; express Activation function; Indicates the value to be to Channel weights; This represents element-wise multiplication; This indicates the characteristics of the channel after recalibration.

[0045] Then, shared parameter mapping and intramodal residual enhancement are performed:

[0046]

[0047] in, Indicates the first Phase-based shared parameter adapter; This represents the fitting result obtained by using optical features as the main input and auxiliary features as the conditional input. This represents the fitting result obtained by using auxiliary features as the main input and optical features as the conditional input. This represents the third version of deformable convolution; Representation layer normalization; and These represent the enhanced optical features and auxiliary modal features, respectively. The fourth-stage features already contain strong abstract semantics, therefore let... , The aforementioned enhancements will no longer be performed to prevent deep semantics from being repeatedly transformed.

[0048] Step 300: Perform scale alignment and splicing projection on the second to fourth stage features of the two branches respectively to form multi-scale aggregated features of each modality. Generate spatial gating weights and modal-level gating weights based on the multi-scale aggregated features. Obtain unified deep features through cross-gating fusion.

[0049] Specifically, the features of the second, third, and fourth stages are aligned to the spatial size of the second stage, and then stitched and convolutionally projected within each modality:

[0050]

[0051] in, and These represent the multi-scale aggregation features of optical modes and auxiliary modes, respectively; and Indicates convolution projection; Indicates channel splicing; and They represent Double Double upsampling.

[0052] Pixel-wise spatial gating and modality-level global gating are generated based on the two-modal aggregation features:

[0053]

[0054] in, and Spatial gating diagrams representing optical modes and auxiliary modes, respectively; express convolution; and These represent modal features filtered through local space; This represents a multilayer perceptron; and These represent the global gating vectors for the two modes, respectively.

[0055] Applying global gating from another modality to the current modality and fusing them yields unified deep features. The formula is:

[0056]

[0057] in, and These represent the optical characteristics and auxiliary modal characteristics after cross-gating, respectively; Indicates fused convolution; This represents a unified deep feature.

[0058] Step 400: Perform two-layer wavelet hybrid expert enhancement and cross-branch state space alternating modulation on the unified deep features, including two layers of cascaded processing units. Perform two-dimensional discrete wavelet decomposition on the input features of each layer. Generate structure cue, anomalous residual cue, and shared background cue based on the decomposition results. The three types of cue drive the corresponding three experts to perform weighted fusion to obtain expert enhanced features. Perform cross-branch multi-directional state space bidirectional modulation on the expert enhanced features and the first-stage dual-modal enhanced features to obtain the current layer modulation features and input them to the next layer.

[0059] Specifically, a two-layer wavelet hybrid expert enhancement and cross-branch state space alternating modulation are performed on the unified deep features, including two layers of cascaded processing units. In each layer, a two-dimensional discrete wavelet transform is first performed on the input features of the current layer to obtain low-frequency subbands and three high-frequency subbands in three directions. The low-frequency components are subjected to residual calibration through gating masks, and the three high-frequency components are spliced ​​and convolutionally projected to form high-frequency aggregated features. The initial reconstructed features are obtained through inverse wavelet transform, and then the final wavelet enhanced features are obtained through deep convolution and normalization. Structural cues are generated based on the high-frequency aggregated features. Anomaly residual cues (modality-specific anomaly residual cues) are generated based on the difference between the final wavelet enhanced features and the shared low-frequency baseline features. Shared background cues (modality-specific anomaly residual cues) are generated based on the shared low-frequency baseline features. The three types of cues drive the structural expert, the anomaly residual expert, and the shared background expert, respectively. The outputs of the three experts are weighted and fused and residual corrected through pixel-by-pixel routing weights to obtain the expert enhanced features of the current layer. Then, the expert-enhanced features are used as deep branches, and the first-stage bimodal enhancement features are used as shallow branches. Multi-directional two-dimensional state space scanning is performed on both branches. Gating coefficients are generated from the opposing branches to achieve bidirectional modulation, obtaining the current layer's modulation features, which are then input to the next layer. First layer output. Second layer output ,in, and These represent the first and second layer wavelet-guided hybrid expert modules, respectively.

[0060] Step 500: Decode and fuse the modulation features output by the second-layer processing unit to output pixel-level semantic segmentation results.

[0061] Specifically, the modulation features output by the second-layer processing unit are decoded and fused, and a semantic category prediction map with the same spatial size as the original optical remote sensing image is output by the pixel classification prediction layer. During training, the semantic segmentation loss, low-frequency regularization loss, boundary supervision loss, and expert equalization loss are combined into a total loss, expressed as:

[0062]

[0063] in, Indicates the total loss; Indicates semantic segmentation loss; This indicates low-frequency regularization loss; Indicates boundary surveillance loss; This represents the expert equilibrium loss; , and This represents the weight of each auxiliary loss.

[0064] The semantic segmentation loss consists of multi-class Dice loss and cross-entropy loss. Softmax is applied to the network output along the class dimension to obtain the predicted probability. The ground truth labels are converted to one-hot encodings, and the sum of the two losses is calculated after excluding ignored regions. Low-frequency regularization loss applies L1 norm constraints to the low-frequency residual amplitudes of the two-layer wavelet hybrid expert modules. Boundary supervision loss generates boundary labels based on the class differences between adjacent pixels according to the ground truth semantic labels. The structural expert output or structural cues are connected to the boundary prediction head, and the sum of the Dice loss and cross-entropy loss between the boundary prediction and the boundary label is calculated. Expert equalization loss calculates the average usage ratio of each expert in a batch and calculates the KL divergence between this routing distribution and the uniform distribution. The above four losses are combined according to preset weights to form the total training loss, which is used to perform backpropagation and network parameter updates. After each training epoch, forward inference is performed on the validation set, and the segmentation performance index is calculated. When the current model's overall segmentation performance is better, the parameters of each network module are saved as model parameters for subsequent testing and actual inference. During inference, the optical remote sensing image to be processed and the auxiliary modal remote sensing image are read. Spatial registration check, size adjustment and numerical normalization are completed in the same way as in the training phase. The processed dual-modal image is input into the trained network and sequentially passes through dual-branch coding and enhancement, multi-scale gating fusion, dual-layer wavelet hybrid expert enhancement and cross-branch state space alternating modulation and decoding fusion to obtain the probability of each pixel belonging to each semantic category. For each pixel, the category index corresponding to the maximum category probability is selected to form a semantic category index map, and a visual semantic segmentation map is generated according to the preset color mapping.

[0065] In one specific implementation, each layer of input features is decomposed into low-frequency subbands, horizontal high-frequency subbands, vertical high-frequency subbands, and diagonal high-frequency subbands using two-dimensional discrete wavelet decomposition. A low-frequency calibration mask is generated based on the low-frequency subbands using lightweight channel projection and a Sigmoid activation function. The low-frequency calibration mask is multiplied element-wise with the low-frequency subbands to obtain low-frequency residuals, and these residuals are superimposed onto the low-frequency subbands to obtain calibrated low-frequency features. Channel splicing and convolutional projection are performed on the three high-frequency subbands to obtain high-frequency aggregated features.

[0066] Specifically, the two-dimensional discrete wavelet transform uses Haar wavelet basis functions to independently decompose each channel of the current layer input feature F into four sub-bands:

[0067]

[0068] This represents the input features of the current wavelet hybrid expert layer. Represents a two-dimensional discrete wavelet transform, low-frequency subband. The size is half that of the input feature, the number of channels remains the same, and the horizontal high-frequency subband... Vertical high-frequency subband and diagonal high-frequency subband The size is also half that of the input features, and the number of channels remains unchanged.

[0069] When performing residual gating calibration on the low-frequency subband, a lightweight channel projection with a convolution kernel size of 1×1 is used. The number of channels in the low-frequency subband is compressed to one-quarter of the original number, and the compressed result is activated by the Sigmoid function and used as a low-frequency calibration mask. The calibration mask is applied element-wise to the original low-frequency subband to obtain the low-frequency residual. Then, the low-frequency residual is added element-wise to the original low-frequency subband to obtain the calibrated low-frequency characteristics. The dimensions of the calibrated low-frequency features are the same as those of the original low-frequency subbands. The three high-frequency subbands are then concatenated, resulting in three times the number of channels of the original high-frequency subbands, and passed through a 3×3 convolutional layer. Projecting the splicing result restores the number of channels to the same level as the original high-frequency subbands, yielding the high-frequency aggregation features. .in, This indicates low-frequency lightweight channel projection; Indicates a low-frequency calibration mask; Indicates low-frequency residual; This indicates the low-frequency characteristics after calibration; Indicates high-frequency convergent projection; This indicates high-frequency aggregation characteristics.

[0070] In one specific implementation, the low-frequency and high-frequency aggregated features are input into the inverse discrete wavelet transform for spatial reconstruction. The reconstruction result is then channel-aligned using a first pointwise convolution, and the channel-aligned result is locally spatially mixed using a depthwise convolution with a kernel size of 7×7. Finally, wavelet-enhanced features are obtained through a second pointwise convolution and normalization.

[0071] Specifically, the calibration low-frequency characteristics will be used. Aggregation features corresponding to the three high-frequency subbands Arranged according to their spatial positions during wavelet decomposition, the input is reconstructed using inverse discrete wavelet transform. The reconstructed spatial resolution is restored to the spatial resolution of the current layer's input features F, and the number of channels is four times that of the current layer's input features. The reconstruction result is first reduced to twice the number of channels of the current layer's input features through a first pointwise convolution. Then, local spatial blending is performed using depthwise convolution with a kernel size of 7×7: In depthwise convolution, the number of input channels equals the number of output channels, and each channel undergoes spatial convolution independently. The locally blended features are further reduced to the same number of channels as the input features of the current layer through a second pointwise convolution. Finally, wavelet-enhanced features are obtained through batch normalization. This makes the numerical distribution of the wavelet transform reconstructed features and the subsequent expert processing features more consistent. Among them, Indicates the inverse discrete wavelet transform; Indicates the initial reconstructed features; express Depthwise convolution; Indicates normalization; This represents the final wavelet enhancement feature.

[0072] In one specific implementation, high-frequency aggregated features are convolved pointwise to obtain structural cues; low-frequency subbands are convolved and projected to obtain shared baseline features; the element-wise absolute difference between wavelet enhanced features and shared baseline features is calculated; and abnormal residual cues are obtained by sequentially processing pointwise convolution, depthwise convolution, and channel attention; and shared background cues are obtained by convolving the shared baseline features pointwise.

[0073] Specifically, structural hints are obtained through high-frequency aggregation features. Pointwise convolution yields: Pointwise convolution does not change the spatial size of the features, but maps the number of channels to a preset number. Shared baseline features are achieved through low-frequency subbands. Convolution projection yields: The projection uses a 1×1 convolution kernel to adjust the number of channels in the low-frequency subband to match the wavelet enhancement feature. The same applies. The abnormal residual indication is generated as follows: first, the absolute difference between the wavelet enhanced feature and the shared baseline feature at each channel in the same spatial location is calculated. The absolute difference results are compressed through pointwise convolution, and then spatial features are extracted using depthwise convolution with a kernel size of 7×7. The extracted results are input into the channel attention module, which generates attention weights for each channel and recalibrates the channels. The calibration results serve as a warning of abnormal residuals. Shared background hints are obtained by performing pointwise convolutions on shared baseline features: The spatial dimensions of structural cues, anomalous residual cues, and shared background cues are all the same as those of wavelet enhancement features. Among them, Indicates a high-frequency spatial structure indication; This represents the shared low-frequency reference feature obtained by projecting from the low-frequency sub-band; This represents the absolute difference between wavelet-enhanced features and shared baseline features; This represents the residual features after local convolution thinning; Indicates residual channel attention; This indicates modality-specific anomaly residuals; This indicates a shared background notification.

[0074] In one specific implementation, the three experts include a structural expert, an anomaly residual expert, and a shared background expert. The structural expert includes a 3×3 depthwise convolutional branch and a 5×5 depthwise convolutional branch set in parallel. The outputs of the two branches are spliced ​​and projected, and then added to the wavelet enhancement feature residual. The anomaly residual expert injects anomaly residuals into the wavelet enhancement feature. The shared background expert splices the 7×7 depthwise convolution result of the wavelet enhancement feature with the shared baseline feature, performs pointwise convolution and channel recalibration, and then adds it to the wavelet enhancement feature residual.

[0075] Specifically, the structural expert consists of two parallel branches. The first branch uses a 3×3 depthwise convolution to extract the local boundary response, and the second branch uses a 5×5 depthwise convolution to extract the structural response within a larger receptive field.

[0076]

[0077]

[0078] The outputs of the two branches are concatenated along the channel dimension and then projected through pointwise convolution. The projection result is then added element-wise to the wavelet enhancement feature: Anomaly residual experts inject anomalous residual clues into wavelet enhancement features through element-wise addition: In the shared background expert model, wavelet enhancement features are first spatially blended using a 7×7 depthwise convolution with a stride of 1 and padding of 3 to maintain the spatial dimensions. The blended result is then concatenated with the shared baseline feature F_base along the channel dimension. After concatenation, the number of channels is adjusted to match the wavelet enhancement feature through pointwise convolution and channel recalibration. The adjusted result is then element-wise added to the wavelet enhancement feature.

[0079]

[0080] in, Indicates the core size is The depthwise convolution output; This represents pointwise convolution; Indicates batch normalization; This represents the feature extraction branch of the structural expert; This represents the output from the structural expert. This indicates the expert output for abnormal residuals; Indicates the channel recalibration unit; This indicates shared background expert output.

[0081] In one specific implementation, wavelet enhancement features, structural cues, anomalous residual cues, and shared background cues are concatenated into channels, and spatial routing values ​​corresponding to the three experts are generated sequentially through pointwise convolution, 3×3 depthwise convolution, and pointwise convolution. The spatial routing values ​​are normalized at each pixel position using a Softmax function with a temperature parameter. The outputs of the three experts are then weighted and summed pixel by pixel according to the normalized routing weights to obtain adaptive expert fusion features.

[0082] Specifically, wavelet enhancement features, structural cues, anomalous residual cues, and shared background cues are concatenated along the channel dimension: The number of channels after concatenation is the sum of the number of channels from the four features. The concatenated features are sequentially processed through a first pointwise convolution to compress the number of channels to one-quarter of the original number, followed by spatial blending through a 3×3 depthwise convolution, and then expanded to three channels through a second pointwise convolution. The three channels correspond to the spatial routing values ​​of the structural expert, the anomaly residual expert, and the shared background expert, respectively. The initial value of the spatial routing value at each pixel position is an unnormalized real number. The three routing values ​​are then normalized at the same pixel position using a Softmax function with a temperature parameter. , The temperature parameter τ is set as a learnable scalar during model training. The output features of the three experts are multiplied by their corresponding routing weights at the same pixel location, and then the three weighted features are summed element-wise. This yields adaptive expert fusion features. express , , and The channel splicing result; This represents the three expert space scores output by the routing header; express Temperature parameters; , and These represent the routing weights of the structural expert, the anomaly residual expert, and the shared background expert at each pixel location, respectively. This represents the expert features after pixel-by-pixel weighted fusion.

[0083] In one specific implementation, the adaptive expert fusion features and wavelet enhancement features are concatenated and refined by convolution, and then added to the residual of the current layer input features to obtain the expert enhancement features.

[0084] Specifically, adaptive expert fusion features With wavelet enhancement features The features are concatenated along the channel dimension, and then refined using a 3×3 convolutional layer. The number of output channels of the convolutional layer is the same as that of the wavelet-enhanced features. The refined result is added to the residual of the current layer's input feature F to form the current layer's output feature. The addition of residuals ensures the smooth propagation of gradients during forward and backward propagation. The two-layer wavelet-guided hybrid expert enhancement consists of a cascaded first wavelet hybrid expert layer. Second wavelet hybrid expert layer The first layer receives unified deep features. As input for the current layer: The output features of the first layer are directly used as the input of the current layer of the second layer: The second layer does not change the number of channels or spatial size of the features. The internal structure of the two layers is exactly the same, but the network parameters are trained independently.

[0085] In one specific implementation, cross-branch multi-directional state space bidirectional modulation includes: using expert enhancement features as deep branches and first-stage bimodal enhancement features as shallow branches; performing layer normalization and linear projection on the two branches respectively, so that they have the same spatial size and channel dimension; performing depth convolution on the two branches respectively and unfolding them into one-dimensional sequences along four directions, performing state space recursion processing and then aggregating them to obtain shallow state features and deep state features.

[0086] Specifically, the dual-layer enhanced features output by the second wavelet hybrid expert layer are... The features are upsampled to an intermediate spatial resolution, which is twice the current feature space resolution, using bilinear interpolation. The upsampled features are then passed through a 1×1 convolutional layer to adjust the number of channels, resulting in the input for the deep branch. Shallow branch inputs enhance features through the first stage of the optical branch encoder. Enhanced features in the first stage of the auxiliary modal branch encoder The channel is stitched together, and then projected through a convolutional layer with a kernel size of 1×1 to obtain: The number of channels after projection is the same as the number of channels in the deep branch input, and the spatial size is the same as the deep branch input. Both the deep and shallow branch inputs are standardized by layer normalization to make the features of each channel zero and the variance one. After standardization, they are adjusted to the same channel dimension and spatial size through a linear projection layer.

[0087] in, Indicates input for a deep branch; Indicates convolution thinning; express Double upsampling; Indicates shallow branch input; Indicates shallow fusion projection; and Indicates the linear projection parameters; and This represents the shallow and deep features after projection onto the common feature space.

[0088] In one specific implementation, shallow gating coefficients are generated based on the deep branch input after layer normalization, and deep gating coefficients are generated based on the shallow branch input after layer normalization. The shallow gating coefficients are multiplied element-wise with the shallow state features, and the deep gating coefficients are multiplied element-wise with the deep state features. The multiplication results are linearly projected and added to the residuals of the corresponding branch inputs. The features obtained by adding the residuals of the two branches are concatenated and fused by convolution to obtain the modulation features of the current layer.

[0089] Specifically, shallow branch input and deep branch input Spatial features are extracted using depthwise convolutions with a kernel size of 3×3. After convolution, the features at each location form a two-dimensional feature map along the channel dimension. These two-dimensional features are then unfolded into a one-dimensional sequence along four complementary directions. The four-directional two-dimensional state space scan is represented as follows:

[0090]

[0091] Where U represents the two-dimensional feature to be scanned, q represents the scanning direction number, and S_q(.) represents the unfolding operation in the q-th direction. This indicates the corresponding reverse scan recovery operation. This represents one-dimensional state space processing; This represents the two-dimensional state space output after the four-directional results are recovered and summed.

[0092] When expanding from left to right, each row of the 2D feature map is concatenated from left to right to form a 1D sequence; when expanding from right to left, each row is concatenated from right to left; when expanding from top to bottom, each column is concatenated from top to bottom; and when expanding from bottom to top, each column is concatenated from bottom to top. The 1D sequences in each of the four directions are input into the state space recursion module. For any direction, a 1D sequence of length L is obtained. The discretization of state-space parameters and the hidden state update process are represented as follows:

[0093]

[0094] in, , , and This represents the parameter matrix in a continuous state-space model. This represents the discretization step size, which is determined by the input. and This represents the discretized state transition matrix and input matrix; Indicates the first in the sequence Input at each position; Indicates the first The hidden state of each location; Indicates the corresponding output; Represents the identity matrix; Indicates the sequence length.

[0095] After the recursion in each direction is completed, the one-dimensional sequence is restored to its original two-dimensional spatial position according to its unfolding order through the corresponding inverse scan operation. The restoration results in the four directions are added and aggregated element by element at the same spatial position. The aggregated result is then processed by layer normalization to obtain shallow state features and deep state features:

[0096] in, and These represent the features of shallow and deep branches after local convolution, four-way state scanning, and normalization, respectively.

[0097] In one specific implementation, the decoding fusion includes: convolving and refining the modulation features output by the second-layer processing unit, and generating a semantic category prediction map with the same spatial size as the optical remote sensing image through a pixel classification prediction layer.

[0098] Specifically, the shallow gating coefficients are obtained by performing pointwise convolution on the normalized deep branch input and then activating it with the Sigmoid function: The deep gating coefficients are obtained by performing pointwise convolution on the shallow branch input after layer normalization and then activating it with the Sigmoid function. The shallow gating coefficients are multiplied element-wise with the shallow state features, thereby modulating the shallow state features with deep semantic information in a gating manner. The deep gating coefficient is multiplied element-wise with the deep state features, thereby modulating the deep state features in a gating manner using shallow structural information. .in, and Indicates the gated projection parameters; This represents the gating coefficient generated by the deep branch and applied to the shallow branch; This represents the gating coefficient generated by shallow branches and applied to deep branches; and This represents the two-branch features after the interaction enhancement.

[0099] The multiplication results are then processed through linear output projection layers with adjusted channel numbers. After projection, they are added element-wise to the corresponding branch inputs to form residual connections. , The features obtained by adding the residuals of the two branches are concatenated along the channel dimension and then fused through a convolutional layer with a kernel size of 3×3. The modulation features of the current layer are obtained. The modulation features output by the second processing unit are refined by a 3×3 convolutional layer, and a semantic category prediction map with the same spatial size as the original optical remote sensing image is output by a pixel classification prediction layer. , .in, and This represents the output projection parameters of the two branches; and This indicates the shallow and deep features after residual compensation; Indicate the original Figure 1 Intermediate decoding features at half resolution; This indicates the decoding features used to restore the image to its original resolution. Indicates branch fusion projection; This represents the pixel classification prediction layer.

[0100] The remote sensing segmentation method provided by this invention, which combines dual-layer wavelet hybrid expert and cross-branch state fusion, extracts four-stage hierarchical features from optical and auxiliary modal images using a dual-branch encoder. It then performs channel recalibration, shared parameter mapping, and intra-modal residual enhancement on the first three stages of features. This eliminates modal distribution differences at the shallow and mid-level features while preserving the unique local structural information of each modality, providing a well-aligned feature foundation for subsequent cross-modal fusion. By aggregating the second to fourth stage features from both branches at multiple scales and generating spatial and modal-level gated weights for cross-gated fusion, the method adaptively adjusts the fusion strategy according to the importance of different regions and modalities in the image, suppressing redundant responses and highlighting key modal information. This invention employs a two-layer wavelet-guided hybrid expert enhancement approach. In each layer, the input features are decomposed into two-dimensional discrete wavelet decomposition. Structural cues, anomalous residual cues, and shared background cues are generated based on low-frequency and high-frequency subbands, respectively. These three types of cues drive three corresponding experts to perform pixel-by-pixel route-weighted fusion. This approach dynamically selects appropriate processing paths for different pixels based on spatial differences in target boundary regions, modal inconsistency regions, and shared background regions, overcoming the limitations of traditional wavelet transforms that rely on preset weighted fusion bands. By alternating wavelet hybrid expert enhancement with cross-branch state-space modulation in each layer, a progressive and mutually supportive tight coupling relationship is established between local frequency band refinement and global context modeling. Simultaneously, the modulation of shallow details by deep semantics and the reverse modulation of deep semantics by shallow structure balance edge refinement and regional consistency. This invention effectively improves the differential segmentation accuracy of target boundaries, modal anomaly regions, small-scale features, and shared backgrounds in multimodal remote sensing images under complex scenes.

[0101] It is important to note that the constructions and arrangements of this application shown in several different exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible (e.g., changes in the size, dimensions, structure, shape, and proportions of various elements, as well as parameter values ​​(e.g., temperature, pressure, etc.), mounting arrangements, use of materials, color, orientation, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in this application). For example, an element shown as integrally formed may be composed of multiple parts or elements, the position of elements may be inverted or otherwise altered, and the nature or number or position of discrete elements may be changed or altered. Therefore, all such modifications are intended to be included within the scope of the invention. The order or sequence of any process or method steps may be changed or rearranged according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and not only structurally equivalent but also equivalent in structure. Other substitutions, modifications, alterations, and omissions may be made in the design, operation, and arrangement of the exemplary embodiments without departing from the scope of the invention. Therefore, the present invention is not limited to the specific embodiments, but extends to various modifications that still fall within the scope of the appended claims.

[0102] Furthermore, to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features not relevant to the currently considered best mode for carrying out the invention, or those features not relevant to implementing the invention) may be omitted. It should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention and not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal remote sensing image semantic segmentation method using a two-layer wavelet hybrid expert and cross-branch state fusion, characterized in that, Includes the following steps: Acquire spatially registered optical remote sensing images and auxiliary modal remote sensing images, adjust them to the same spatial size, and form a dual-modal input; The dual-modal input is fed into two branch encoders to extract four-stage hierarchical features. The first three stages of features are then subjected to channel recalibration, shared parameter mapping, and intramodal residual enhancement to obtain dual-modal enhanced features. The second to fourth stage features of the two branches are scale aligned and spliced ​​projection to form multi-scale aggregated features of each modality. Spatial gating weights and modal gating weights are generated based on the multi-scale aggregated features. Unified deep features are obtained by cross-gating fusion. The unified deep features are subjected to two-layer wavelet hybrid expert enhancement and cross-branch state space alternating modulation, including two layers of cascaded processing units. Two-dimensional discrete wavelet decomposition is performed on each layer of input features. Based on the decomposition results, structural cue, abnormal residual cue, and shared background cue are generated. The three types of cue drive the corresponding three experts to perform weighted fusion to obtain expert-enhanced features. The expert enhancement features and the first-stage bimodal enhancement features are subjected to cross-branch multi-directional state space bidirectional modulation to obtain the current layer modulation features and input to the next layer. The modulation features output by the second-layer processing unit are decoded and fused to output pixel-level semantic segmentation results.

2. The multimodal remote sensing image semantic segmentation method according to claim 1, characterized in that, Each layer of input features is decomposed into low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands, and diagonal high-frequency sub-bands using two-dimensional discrete wavelet decomposition. A low-frequency calibration mask is generated based on the low-frequency sub-bands using lightweight channel projection and a Sigmoid activation function. The low-frequency calibration mask is multiplied element-wise with the low-frequency sub-bands to obtain low-frequency residuals. The low-frequency residuals are then superimposed on the low-frequency sub-bands to obtain the calibrated low-frequency features. High-frequency aggregated features are obtained by channel splicing and convolution projection of the three high-frequency subbands.

3. The multimodal remote sensing image semantic segmentation method according to claim 2, characterized in that, The calibration low-frequency features and the high-frequency aggregation features are input into the inverse discrete wavelet transform for spatial reconstruction. The reconstruction result is then channel-aligned using a first pointwise convolution, and the channel-aligned result is locally spatially mixed using a depthwise convolution with a kernel size of 7×7. Finally, wavelet enhancement features are obtained through a second pointwise convolution and normalization.

4. The multimodal remote sensing image semantic segmentation method according to claim 3, characterized in that, The high-frequency aggregated features are convolved pointwise to obtain the structural cue; the low-frequency subbands are convolved and projected to obtain shared baseline features; the element-wise absolute difference between the wavelet enhancement features and the shared baseline features is calculated; and the abnormal residual cue is obtained by sequentially processing pointwise convolution, depthwise convolution, and channel attention; the shared background cue is obtained by convolving the shared baseline features pointwise.

5. The multimodal remote sensing image semantic segmentation method according to claim 3, characterized in that, The three experts include a structural expert, an anomaly residual expert, and a shared background expert; the structural expert includes a 3×3 depthwise convolutional branch and a 5×5 depthwise convolutional branch set in parallel, and the outputs of the two branches are spliced ​​and projected and added to the wavelet enhancement feature residual; the anomaly residual expert injects the anomaly residual cue into the wavelet enhancement feature; The shared background expert concatenates the 7×7 depth convolution result of the wavelet enhancement feature with the shared baseline feature, and adds it to the residual of the wavelet enhancement feature after pointwise convolution and channel recalibration.

6. The multimodal remote sensing image semantic segmentation method according to claim 5, characterized in that, The wavelet enhancement features, structural cues, anomalous residual cues, and shared background cues are concatenated into channels, and spatial routing values ​​corresponding to the three experts are generated sequentially through pointwise convolution, 3×3 depthwise convolution, and pointwise convolution. The spatial routing values ​​are normalized at each pixel position using a Softmax function with a temperature parameter. The three expert outputs are then weighted and summed pixel by pixel according to the normalized routing weights to obtain adaptive expert fusion features.

7. The multimodal remote sensing image semantic segmentation method according to claim 6, characterized in that, The adaptive expert fusion feature is concatenated and convolved with the wavelet enhancement feature for thinning, and then added to the residual of the current layer input feature to obtain the expert enhancement feature.

8. The multimodal remote sensing image semantic segmentation method according to claim 7, characterized in that, The cross-branch multi-directional state space bidirectional modulation includes: taking the expert enhancement features as deep branches and the first-stage bimodal enhancement features as shallow branches; performing layer normalization and linear projection on the two branches respectively, so that they have the same spatial size and channel dimension; performing depth convolution on the two branches respectively and unfolding them into one-dimensional sequences along four directions, performing state space recursion processing and then aggregating them to obtain shallow state features and deep state features.

9. The multimodal remote sensing image semantic segmentation method according to claim 8, characterized in that, Shallow gating coefficients are generated based on the normalized deep branch input, and deep gating coefficients are generated based on the normalized shallow branch input. The shallow gating coefficients are multiplied element-wise with the shallow state features, and the deep gating coefficients are multiplied element-wise with the deep state features. The multiplication results are linearly projected and added to the residuals of the corresponding branch inputs. The features obtained by adding the residuals of the two branches are concatenated and fused by convolution to obtain the modulation features of the current layer.

10. The multimodal remote sensing image semantic segmentation method according to claim 1, characterized in that, The decoding fusion includes: convolving and refining the modulation features output by the second-layer processing unit, and generating a semantic category prediction map with the same spatial size as the optical remote sensing image through a pixel classification prediction layer.

Images