Multi-source remote sensing image segmentation model, method and system based on VMama and pseudo twin network

Abstract

The invention discloses a multi-source remote sensing image segmentation model, method and system based on VMama and a pseudo twin network. The multi-source remote sensing image segmentation model comprises a coding network, a feature enhancement reconstruction network and a decoding network, the coding network comprises an optical coding branch and an SAR coding branch, the optical coding branch comprises a first Patch Embedding layer and n layers of VSSLayers which are sequentially connected in series, and the SAR coding branch comprises a second Patch Embedding layer and n layers of first Re-SVSSLayers which are sequentially connected in series; the output end of the last layer of the optical coding branch and the output end of the last layer of the SAR coding branch are added and connected to the input end of the decoder; the decoding network comprises n second Re-SVSSLayer layers and a Final Projection layer, wherein the n second Re-SVSSLayer layers and the Final Projection layer are sequentially connected in series; the feature enhancement reconstruction network comprises n feature enhancement reconstruction modules, the output ends of the n feature enhancement reconstruction modules are in jump connection with the input ends of n second Re-SVSSLayers in a one-to-one correspondence mode, and the input ends of the n feature enhancement reconstruction modules are connected with the output ends of the first n layers of the optical coding branch and the SAR coding branch in a one-to-one correspondence mode. The multi-source remote sensing image semantic segmentation method effectively improves the precision of multi-source remote sensing image semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a multi-source remote sensing image segmentation model, method and system based on VMamba and a pseudo twin network. BACKGROUND

[0002] Optical remote sensing images are the most widely used form in remote sensing technology and one of the most intuitive means of obtaining information about the Earth's surface. However, optical images are easily disturbed by factors such as clouds or adverse weather conditions during image acquisition, which can hinder their effectiveness in meeting the needs of practical applications. In contrast, SAR (Synthetic Aperture Radar) sensors can function under all weather conditions and can penetrate certain types of ground cover, producing detailed geometric information about ground objects. In specific applications, optical images and SAR images have a high degree of complementarity, and optical image and SAR image fusion is considered a promising strategy to improve classification accuracy. Nevertheless, optical image and SAR image segmentation in the field of remote sensing poses greater challenges than multi-modal segmentation in natural images.

[0003] With the rapid development of deep learning, numerous studies have provided evidence that multi-modal fusion can enhance semantic segmentation performance. According to the location where fusion occurs, multi-modal fusion techniques can be divided into three main types: data fusion, early fusion, and late fusion. Data fusion involves directly merging the original input images, which is a straightforward technique often used for homogeneous data. Early fusion, also known as feature fusion, occurs after feature extraction. Late fusion, also known as decision-level fusion, directly fuses the segmentation results obtained from the two modalities. Early fusion and late fusion are commonly used techniques when dealing with heterogeneous data, such as optical and SAR. Simple feature fusion methods, such as concatenation or direct addition of features from different modalities along the channel dimension, do not adequately consider semantic gaps, further limiting the utilization of complementary information. In recent years, to overcome these limitations, attention-based fusion models have been proposed to effectively integrate unique features obtained from optical and SAR images. These models can fully capture significant features by focusing on areas of interest. In addition, CFNet, MFNet, CMGFNet, and CEGFNet utilize gate mechanisms to eliminate redundant features and suppress interference, improving the effectiveness of feature fusion. Furthermore, Transformers, which have strong capabilities to capture long-range dependencies, have been introduced into the field of multi-modal segmentation to improve the extraction or fusion of modality-specific features.

[0004] Although these methods improve the segmentation accuracy to some extent, segmentation networks based on CNN architecture are often limited in segmentation accuracy due to the insufficient ability of CNN to extract global information, while segmentation networks based on Transformer architecture have strong global information extraction ability, but the high computational complexity brought by the self-attention mechanism makes it difficult to be applied in practice.

[0005] Most of the current multi-source remote sensing image segmentation methods basically adopt a pseudo-twin architecture, that is, two branches of encoders are used to extract optical image features and SAR image features respectively, but they all ignore the heterogeneous nature of optical image data and SAR data and directly use the same architecture to extract features from optical images and SAR images, which hinders the complete extraction of complementary information and limits the semantic segmentation accuracy.

[0006] Ignoring the semantic gap and directly performing feature fusion makes it difficult for the model to converge and cannot take advantage of complementary advantages to make up for the limitations of a single modality. Although techniques such as SA-Gate and MCANet have introduced attention mechanisms to calculate feature weights between modalities, thus achieving more effective feature selection and fusion. These symmetric fusion frameworks operate under the assumption that key features are consistently shared between the two modalities. Nonetheless, the different imaging mechanisms of optical images and SAR images indicate inherent differences between modalities. Traditional symmetric fusion methods assign the same attention weight to different branches, making the features of the two modalities look more similar. However, this approach hinders the learning of complementary features between different modalities, and not only that, the traditional fusion mechanism will cause the generation of redundant features after fusion, which greatly hinders the performance of the model. SUMMARY

[0007] To solve the technical problems in the background art, the present application proposes a multi-source remote sensing image segmentation model, method and system based on VMamba and pseudo-twin network.

[0008] In a first aspect, the present application proposes a multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network, characterized in that it comprises an encoding network, a feature enhancement reconstruction network and a decoding network.

[0009] The encoding network comprises an optical encoding branch and a SAR encoding branch, the optical encoding branch comprises a first Patch Embedding layer and n layers of VSSLayers connected in series, and the SAR encoding branch comprises a second Patch Embedding layer and n layers of first Re-SVSSLayers connected in series; an input end of the first Patch Embedding layer is an input end of the optical encoding branch, an input end of the second Patch Embedding layer is an input end of the SAR encoding branch, and an output end of the last layer of the optical encoding branch and an output end of the last layer of the SAR encoding branch are connected in addition to an input end of the decoder; wherein n is a positive integer greater than or equal to 1.

[0010] The decoding network comprises n second Re-SVSSLayers and a Final Projiction layer connected in series, an input end of the first second Re-SVSSLayer is an input end of the decoder, and an output end of the Final Projiction layer is an output end of the decoder.

[0011] The feature enhancement reconstruction network comprises n feature enhancement reconstruction modules, the output ends of the n feature enhancement reconstruction modules are connected in jump to the input ends of the n second Re-SVSSLayers in one-to-one correspondence, the input ends of the n feature enhancement reconstruction modules are connected to the output ends of the first n layers of the optical encoding branch in one-to-one correspondence, and the input ends of the n feature enhancement reconstruction modules are also connected to the output ends of the first n layers of the SAR encoding branch in one-to-one correspondence.

[0012] Preferably, the VSSLayer comprises two VSSBlocks connected in series; the first Re-SVSSLayer and the second first Re-SVSSLayer each comprise a SimAM VSSBlock connected in residual connection twice, and the SimAM VSSBlock is a VSSBlock introducing a parameter-free attention mechanism SimAM.

[0013] Preferably, n is 2-6.

[0014] Preferably, n is 4.

[0015] Preferably, the feature enhancement reconstruction module comprises a FEM module and a FRM module; the input end of the FEM module and the input end of the FRM module are shared, and the output end of the FEM module and the output end of the FRM module are connected in element-wise addition, and the element-wise addition connection point is taken as the output end of the feature enhancement reconstruction module.

[0016] Preferably, in the reconstruction process of the feature enhancement reconstruction module, the FEM module is used to one-to-one enhance and fuse the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, to obtain FEM features;

[0017] The FRM module is used to one-to-one enhance the interaction between the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, separate the strong information and the weak information between the optical features and the SAR features, and process the strong information and the weak information respectively, to realize efficient aggregation of the optical features and the SAR features, and obtain FRM features.

[0018] The output of the FEM module and the output of the FRM module are element-wise added, and the element-wise addition result is taken as the final output feature of the feature enhancement reconstruction module.

[0019] Preferably, the FEM module is used to one-to-one enhance and fuse the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, specifically including:

[0020] The corresponding optical features and the corresponding SAR features are respectively passed through a 1x1 convolution to obtain intermediate optical features and intermediate SAR features; the intermediate optical features and the intermediate SAR features are element-wise added to obtain preliminary fusion features; a 1x1 convolution is used to compress the preliminary fusion features into a single channel, a spatial aggregation weight is generated through a sigmoid activation, and the spatial aggregation weight and the preliminary fusion features are element-wise multiplied to obtain a spatial aggregation feature; the spatial aggregation feature is normalized and weighted to obtain a normalized and weighted feature.

[0021] The optical features and the SAR features are respectively sent into two FDMM modules in pairs to maximize the difference between the two different features, to obtain first difference maximization features and second difference maximization features; the first difference maximization features, the second difference maximization features, and the normalized and weighted feature are input into the SMN module to obtain FEM features.

[0022] Preferably, the optical features and the SAR features are respectively sent into two FDMM modules in pairs to maximize the difference between the two different features, to obtain first difference maximization features and second difference maximization features, specifically including:

[0023] The first difference maximization feature of the optical image is obtained by using one FDMM module to perform pixel-by-pixel subtraction between the optical and SAR feature maps, applying global max pooling to the obtained difference feature map, generating global difference weights through sigmoid activation, element-wise multiplying the global difference weights with the optical image, and performing weighted summation on the element-wise multiplication result and the optical image.

[0024] The second difference maximization feature of the SAR image is obtained by using another FDMM module to perform pixel-by-pixel subtraction between the optical and SAR feature maps, applying global max pooling to the obtained difference feature map, generating global difference weights through sigmoid activation, element-wise multiplying the global difference weights with the SAR image, and performing weighted summation on the element-wise multiplication result and the SAR feature map.

[0025] Preferably, the first difference maximization feature, the second difference maximization feature, and the normalized weighted feature are input into the SMN module together to obtain the FEM feature, specifically including:

[0026] The element-wise summation result of the first difference maximization feature and the second difference maximization feature is obtained by performing element-wise summation on the first difference maximization feature and the second difference maximization feature; the mean and the standard deviation of the element-wise summation result of the first difference maximization feature and the second difference maximization feature are calculated; the rich spatial feature is calculated according to the normalized weighted feature and the mean and the standard deviation of the element-wise summation result of the first difference maximization feature and the second difference maximization feature; and the FEM feature is obtained by using a learnable parameter to perform linear transformation on the rich spatial feature.

[0027] Preferably, In the formula, f smn represents the rich spatial feature, f nwf represents the normalized weighted feature, f dif represents the element-wise summation result of the first difference maximization feature and the second difference maximization feature, and mean(·) and std(·) respectively represent the mean and the standard deviation of the element-wise summation result of the first difference maximization feature and the second difference maximization feature.

[0028] Preferably, the process of the linear transformation is represented as In the formula, β and γ represent learnable parameters, represents element-wise summation, represents element-wise multiplication.

[0029] Preferably, the FRM module is used to correspondingly enhance the interaction between the optical features of the output of the first n layers of the optical encoding branch and the SAR features of the output of the first n layers of the SAR encoding branch, separate the strong information and the weak information between the optical features and the SAR features, and process the strong information and the weak information respectively, so as to realize efficient aggregation of the optical features and the SAR features, and obtain FRM features, including:

[0030] The batch normalization processing is respectively performed on the optical features and the SAR features, and the sigmoid function is activated to generate unique weight information of the optical features and weight information of the SAR features at each spatial position; and the optical features and the SAR features are reconstructed in a cross-weighting manner according to the weight information of the optical features and the weight information of the SAR features.

[0031] The reconstructed optical features and the reconstructed SAR features are added to obtain reconstructed addition features; the average pooling and the sigmoid function are used to generate information weights on each channel, and the information weights on each channel are used as feature weight thresholds of each channel.

[0032] The weight information of the optical features and the weight information of the SAR features are compared with the feature weight thresholds of the corresponding channels in the channel dimension, the optical features and the SAR features greater than the feature weight thresholds are classified as strong features, and the optical features and the SAR features less than the feature weight thresholds are classified as weak features; the strong features and the weak features are respectively transformed; and the transformed strong features and the weak features are element-wise added to obtain FRM features.

[0033] Preferably, the strong features and the weak features are respectively transformed, and the transformation specifically includes: the strong features are convoluted by using 1x1 convolution; and the weak features are processed by using a gating module.

[0034] Preferably, T=Sigmoid(AvgPool(Add Feature)); in the formula, T represents the feature weight threshold, AvgPool(·) represents global average pooling, Sigmoid(·) represents an activation function, and Add Feature represents the reconstructed addition features.

[0035] wherein, in the formula, SARFeature represents the SAR features, OPT Feature represents the optical features, represents element-wise summation, represents element-wise multiplication, ω sar represents the weight information of the SAR features, ω opt represents the weight information of the optical features.

[0036] wherein, where BN(·) denotes batch normalization, Conv 1×1 (·) denotes 1x1 convolution.

[0037] In a second aspect, the present application further provides a multi-source remote sensing image segmentation method based on VMamba and pseudo twin networks, comprising:

[0038] obtaining a multi-source remote sensing image segmentation dataset; wherein the multi-source remote sensing image segmentation dataset comprises paired optical remote sensing images and SAR images and corresponding real ground object label maps;

[0039] dividing the multi-source remote sensing image segmentation dataset into a training set and a test set;

[0040] constructing the multi-source remote sensing image segmentation network model as described in any one of the first aspect;

[0041] training and testing the multi-source remote sensing image segmentation network model in turn using the training set and the test set to obtain a trained multi-source remote sensing image segmentation network model;

[0042] segmenting a multi-source remote sensing image to be segmented using the trained multi-source remote sensing image segmentation network model to obtain a multi-class ground object segmentation result.

[0043] In a third aspect, the present application further provides a multi-source remote sensing image segmentation system based on VMamba and pseudo twin networks, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the multi-source remote sensing image segmentation method based on VMamba and pseudo twin networks as described in any one of the second aspect.

[0044] In the present application, the multi-source remote sensing image segmentation model, method and system based on VMamba and pseudo twin networks are constructed, trained and tested, and a multi-source remote sensing image segmentation network model based on Vision Mamba (VMamba) and pseudo twin networks is constructed, which can capture extensive context information, thereby capturing strong remote information and maintaining linear computational complexity, avoiding the lack of the ability to capture remote information due to the limitation of the local receptive field of the CNN model and the high computational burden caused by the quadratic complexity of the image size of the self-attention mechanism of the Transformer structure; and the complementary information between heterogeneous data can be fully extracted, the effective fusion between the two features can be realized, and the feature redundancy can be reduced, so that the features between different modalities can be complementary utilized, thereby effectively improving the accuracy of multi-source remote sensing image semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1A structural schematic diagram of a multi-source remote sensing image segmentation model based on VMamba and pseudo twin network in an embodiment of the present application.

[0046] Figure 2 A structural schematic diagram of VSSLayer and Re-SVSSLayer in an embodiment of the present application.

[0047] Figure 3 A structural schematic diagram of FERBlock in an embodiment of the present application.

[0048] Figure 4 A structural schematic diagram of FEM in an embodiment of the present application; wherein (a) represents a whole structural schematic diagram of the FEM module, (b) represents a structural schematic diagram of the FDMM module, and (c) represents a structural schematic diagram of the SMN module.

[0049] Figure 5 A structural schematic diagram of FRM module in an embodiment of the present application.

[0050] Figure 6 A schematic diagram of a training set in an embodiment of the present application.

[0051] Figure 7 A schematic diagram of a test set in an embodiment of the present application.

[0052] Figure 8 A segmentation result schematic diagram of image segmentation by using a multi-source remote sensing image segmentation method based on VisionMamba (VMamba) and pseudo twin network proposed in the present application. DETAILED DESCRIPTION

[0053] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] In a first aspect, as shown in Figures 1-5 and Figure 8 The present application proposes a multi-source remote sensing image segmentation method based on Vision Mamba (VMamba) and pseudo twin network, which comprises:

[0055] obtaining a multi-source remote sensing image segmentation dataset; wherein the multi-source remote sensing image segmentation dataset comprises paired optical remote sensing images and SAR images and corresponding real ground object label images;

[0056] dividing the multi-source remote sensing image segmentation dataset into a training set and a test set;

[0057] Construct a multi-source remote sensing image segmentation network (Pseudo-twinVMamba Multi-source SegmentationNetwork, PVMSNet) model based on Vision Mamba (VMamba) and pseudo-twin network;

[0058] The multi-source remote sensing image segmentation network model is trained and tested in sequence using the training set and the test set to obtain a trained multi-source remote sensing image segmentation network model;

[0059] The trained multi-source remote sensing image segmentation network model is used to segment the multi-source remote sensing image to obtain multi-category ground object segmentation results.

[0060] The multi-source remote sensing image segmentation network model constructed by the present invention is based on Vision Mamba (VMamba) and pseudo-twin networks. It can capture a wide range of contextual information, thereby capturing powerful long-range information and maintaining linear computational complexity. It avoids the CNN model's lack of ability to capture long-range information due to the limitation of the local receptive field and the Transformer structure's self-attention mechanism requiring quadratic complexity in terms of image size, resulting in a higher computational burden. Moreover, it can achieve full extraction of complementary information between heterogeneous data and effective fusion of two features and reduce feature redundancy, so that features between different modalities can be complementary and utilized, effectively improving the accuracy of semantic segmentation of multi-source remote sensing images.

[0061] like Figure 1 As shown, in this embodiment, the multi-source remote sensing image segmentation network model includes: an encoding network, a feature enhancement and reconstruction network, and a decoding network; the encoding network includes an optical encoding branch and a SAR encoding branch, the optical encoding branch includes a first Patch Embedding layer and n layers of VSSLayer connected in series, and the SAR encoding branch includes a second Patch Embedding layer and n layers of first Re-SVSSLayer connected in series; the input end of the first Patch Embedding layer is the input end of the optical encoding branch, the input end of the second Patch Embedding layer is the input end of the SAR encoding branch, and the output end of the last layer of the optical encoding branch and the output end of the last layer of the SAR encoding branch are added and connected to the input end of the decoder; wherein n is a positive integer greater than or equal to 1;

[0062] The decoding network includes n second Re-SVSSLayer and Final Projiction layers connected in series, the input end of the first second Re-SVSSLayer serves as the input end of the decoder, and the output end of the Final Projiction layer serves as the output end of the decoder; the feature enhancement and reconstruction network includes n feature enhancement and reconstruction modules, the output ends of the n feature enhancement and reconstruction modules are jump-connected to the input ends of the n second Re-SVSSLayer in a one-to-one correspondence, the input ends of the n feature enhancement and reconstruction modules are connected to the output ends of the first n layers of the optical coding branch in a one-to-one correspondence, and the input ends of the n feature enhancement and reconstruction modules are also connected to the output ends of the first n layers of the SAR coding branch in a one-to-one correspondence.

[0063] Among them, the first Patch Embedding layer and the second Patch Embedding layer are the same PatchEmbedding layer, and the first Re-SVSSLayer and the second Re-SVSSLayer are the same second Re-SVSSLayer, such as Figure 1 and Figure 2 shown.

[0064] In specific implementation, the optical coding branch passes through the first Patch Embedding layer and then passes through n series-connected VSSLayers for feature extraction and downsampling; similarly, the SAR coding branch passes through the second Patch Embedding layer and then passes through n series-connected first Re-SVSSLayers for feature extraction and downsampling; the output of the last layer of the optical branch is sent element-by-element to the decoding network, and the output of the first n layers of the encoding network is fused through the feature enhancement and reconstruction network and input into the corresponding decoding network as a skip connection layer. Finally, the output of the decoding network is passed through the FianlProjection layer to obtain the final segmentation result.

[0065] Since the optical image contains more information and the ground feature texture is more detailed, the optical encoding branch directly uses the VSSLayer for feature extraction, retaining more detailed information. However, the SAR data contains more noise information, which will interfere with the discrimination of ground feature types. Therefore, the SAR encoding branch uses the first Re-SVSSLayer for encoding, which can give more attention to ground targets, thereby reducing the influence of noise and reducing the loss of information in the transmission process. Moreover, using the second Re-SVSSLayer as the decoder in the decoding network can further reduce the noise influence of SAR image information in the decoding process and improve the attention to the ground targets to be segmented. In addition, the FERBlock is designed to realize effective fusion between the two features and reduce feature redundancy, so that the features between different modalities can be complementary. Therefore, the multi-source remote sensing image segmentation network model in the embodiment can realize the full extraction of complementary information between heterogeneous data.

[0066] wherein n is 2-6.

[0067] In one specific embodiment, n is 4. In the encoding process, the optical encoding branch uses four VSSLayers in series as the encoder, and the SAR encoding branch uses four first Re-SVSSLayers in series as the encoder. The encoder applies the patch merging operation at the end of the first three stages to reduce the height and width of the input features while increasing the number of channels. Similarly, in the decoding process, four second Re-SVSSLayers are used in series as the decoder, and at the beginning of the last three stages, the patch expanding operation is used to reduce the number of feature channels and increase the height and width. Except for the output of the last layer of the encoder, which uses additive fusion, each layer of features output by the optical encoding branch and the SAR encoding branch is fused by the FERBlock and sent to the corresponding decoding layer through the skip connection.

[0068] In the embodiment, the VSSLayer includes two VSSBlocks in series; the first Re-SVSSLayer and the first Re-SVSSLayer each include a residual connection between two SimAM VSSBlocks, and the SimAM VSSBlock is a VSSBlock with a parameter-free attention mechanism SimAM. The results of the first Re-SVSSLayer and the first Re-SVSSLayer are as follows. Figure 2 the structure of the Re-SVSSLayer in the

[0069] The embodiment takes a VSSBlock based on a VMamba basic block as a basic unit of feature extraction, and concatenates two VSSBlocks to form a basic feature extraction layer VSSLayer. Since the optical image contains more information and the ground feature texture is more detailed, the VSSLayer is directly used for feature extraction to retain more detailed information. However, the noise information contained in the SAR data will interfere with the discrimination of the ground feature type, and therefore, the embodiment introduces a parameter-free attention mechanism SimAM (A Simple, Parameter-Free Attention Module) in the VSSBlock to adjust the VSSBlock, constructs a SimAM VSSBlock, so that it can pay more attention to the ground feature target, thereby reducing the influence of noise, and uses a residual connection to connect two SimAM VSSBlocks to construct a first Re-SVSSLayer and a second Re-SVSSLayer to reduce the loss of information in the transmission process.

[0070] As shown in Figure 4 In the embodiment, the feature enhancement reconstruction module (FERBlock) includes a FEM module (Feature Reconstruction Module) and a FRM module (Feature Reconstruction Module); the feature enhancement reconstruction module includes the FEM module and the FRM module; the input end of the FEM module and the input end of the FRM module are shared, and the output end of the FEM module and the output end of the FRM module are connected in element-by-element addition, and the element-by-element addition connection point is taken as the output end of the feature enhancement reconstruction module.

[0071] In specific implementation, the FEM module is used to enhance and fuse the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch one by one, to obtain FEM features; the FRM module is used to enhance the interaction between the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch one by one, and separate the strong information and the weak information between the optical features and the SAR features, and process the strong information and the weak information respectively, to realize efficient aggregation of the optical features and the SAR features, and obtain FRM features; the output of the FEM module and the output of the FRM module are added in element by element, and the element-by-element addition result is taken as the final output feature Fusion Feature of the feature enhancement reconstruction module.

[0072] The embodiment enhances and fuses the optical feature and the SAR feature through the FEM module, enhances the correlation of the fused feature, solves the problem of poor feature correlation and matching in the traditional fusion strategy, and provides more abundant context information; the interaction between the two features is enhanced through the FRM module, and the strong information and the weak information between the features are separated, and the strong information and the weak information are processed respectively, so that the optical feature and the SAR feature are efficiently aggregated; finally, the semantic deviation of the optical feature and the SAR feature in the fusion is reduced through the reconstruction feature, i.e. element-by-element addition. The embodiment effectively reduces the feature redundancy introduced by fusion and transformation, and enhances the interaction between the two modal information.

[0073] As shown in Figure 4 (a), in the FEM module, the optical feature and the SAR feature are first respectively subjected to a 1x1 convolution to obtain an intermediate optical feature f opt and an intermediate SAR feature f sar to improve the expression ability of the features; then, the intermediate optical feature f opt and the intermediate SAR feature f sar are added element by element to obtain a preliminary fusion feature f ou ; in order to refine the preliminary fusion feature f ou and obtain context perception information, the preliminary fusion feature f ou is compressed into a single channel using a 1x1 convolution to aggregate spatial information; then, a spatial aggregation weight w is generated through a sigmoid activation, and the spatial aggregation weight w and the preliminary fusion feature f ou are multiplied element by element to obtain a spatial aggregation feature f

[0074] In order to enhance the expression ability of the feature map, the embodiment directly normalizes and weights the spatial aggregation feature f sp to obtain a normalized and weighted feature Normalized Weighted Feature; in addition to calculating the normalized and weighted feature, the optical feature and the SAR feature are also sent into two FDMM modules (Feature Difference Maximization Module) to maximize the difference between the two different features, respectively extracting complementary features from each modality, to obtain a first difference maximized feature DiffenceFeature1 and a second difference maximized feature DiffenceFeature2, so as to capture the complementarity between different modalities;

[0075] Finally, the DiffenceFeature1, DiffenceFeature2 and the normalized weighted feature are input into the SMN module (Spatial Mapping Normalization) to obtain the FEM feature of the output of the FEM module.

[0076] In the embodiment,

[0077] In the formula, ω s represents the spatial aggregation weight, f sp represents the spatial aggregation feature, f ou represents the preliminary fusion feature, f opt represents the intermediate optical feature, f sar represents the intermediate SAR feature, represents element-wise summation, represents element-wise multiplication, Sigmoid(·) represents an activation function, Conv 1×1 (·) represents a 1x1 convolution.

[0078] In the process of normalizing and weighting the intermediate feature f sp aggregated in the embodiment, a 1x1 convolution module is used to refine the feature information of adjacent channels, and the numerical stability of the model is improved and the expression ability of the feature map is enhanced through normalization and weighting.

[0079] wherein the normalized and weighted feature is represented as

[0080] ω gm =Sigmoid(GM(Conv 1×1 (f sp )));

[0081]

[0082] In the formula, represents element-wise summation, represents element-wise multiplication, Conv 1×1 (·) represents a 1x1 convolution, GM(·) represents normalization of the feature, represents the normalized weight.

[0083] The FDMM module in the embodiment follows the differential amplification principle, is used for performing pixel-by-pixel subtraction between optical and SAR feature maps to obtain a differential feature map, and applies global maximum pooling to the obtained differential feature map and generates a global differential weight The global difference weight is element-wise multiplied with the optical image or the SAR image, and the element-wise multiplication result is weighted summed with the optical image or the SAR feature map to obtain a difference maximization feature of the optical image or the SAR image.

[0084] As shown in Figure 4 (b), taking Feature1 as an example, the specific process of the FDMM module is represented as

[0085] ω mx = Sigmoid (MaxPool (Feature1-Feature2)) ;

[0086]

[0087] In the formula, represents a global difference weight, Feature1 represents a first feature, i.e., an optical image; Feature2 represents a second feature, i.e., a SAR image, - represents element-wise subtraction, represents element-wise summation, represents element-wise multiplication, MaxPool(·) represents global maximum pooling, and Sigmoid(·) represents an activation function. Similarly, a difference maximization feature DiffenceFeature2 of Feature2 can be obtained.

[0088] The SMN module in the embodiment is used to embed normalized weighted features into difference maximization features, so as to obtain rich spatial features f smn with strong feature correlation and complementarity. Finally, the rich spatial features are linearly transformed using learnable parameters and to obtain FEM features, so as to enhance the expression ability of the features.

[0089] The process of the linear transformation is represented as In the formula, represents element-wise summation, represents element-wise multiplication.

[0090] In the formula, smn the rich spatial features f are represented as In the formula, f smn represents rich spatial features, f nwf NormalizedWeightedFeature represents normalized weighted features, f dif represents the element-wise summation result of DiffenceFeature1 and DiffenceFeature2, mean(·) and std(·) represent mean and standard deviation, respectively.

[0091] Before embedding the normalized weighted features into the difference maximization features, first sum the elements of DiffenceFeature1 and DiffenceFeature2, and obtain the element-by-element summation result f of DiffenceFeature1 and DiffenceFeature2. dif ; Then calculate the element-by-element sum of DiffenceFeature1 and DiffenceFeature2 f dif The mean and standard deviation of dif The mean and standard deviation of the spatial feature f are calculated. smn It can be seen that this embodiment enhances the overall feature representation and improves the information expression capability through the FEM module.

[0092] like Figure 6 As shown, in the FRM module (Feature Reconstruction Module), the optical features and SAR features are first processed by BatchNormalization (BN) respectively, and activated by the sigmoid function to generate unique weight information at each spatial position. and Next, in order to enhance the information interaction between different features, this embodiment uses a cross-weighted approach to reconstruct the optical features and SAR features, that is, the optical features are combined with ω sar Perform element-by-element multiplication, SAR features and ω opt Perform element-by-element multiplication to better achieve complementary utilization between features; then, add the reconstructed optical features and SAR features to obtain the reconstructed additive features (AddFeature); then use average pooling and sigmoid function to generate the information weight on each channel for the reconstructed additive features, and use the information weight on each channel as the feature weight threshold of each channel

[0093] Next, the optical characteristic ω opt The weight information of SAR features and the weight information of SAR features ω sar Compare with the feature weight threshold T of the corresponding channel in the channel dimension, and classify the optical features and SAR features greater than the feature weight threshold as strong features (HighFeature), and classify the optical features and SAR features less than the feature weight threshold as weak features (LowFeature);

[0094] Subsequently, the strong and weak features are transformed separately. Specifically, a 1×1 convolution is used to generate a rich and detailed semantic feature map for the strong features. A gating module is used to process the weak features.

[0095] Finally, the strong features and the weak features are element-wise added to obtain the output features of the FRM module FRM Feature.

[0096] The FRM module in the embodiment greatly enhances the information interaction between the two modalities, and reduces the feature redundancy in the fusion process by separating the strong features and the weak features, thereby improving the expression ability of the model.

[0097] In the process of processing the weak features by using the gating module, the global pooling layer, the depth separable convolution layer and the ReLU function layer are sequentially used to process the weak features, to obtain the channel attention weight; and the channel attention weight and the weak features after the depth separable convolution are element-wise multiplied to obtain the transformed weak features, so as to enhance the important features in the weak features.

[0098] wherein the process of obtaining the information weight T can be represented as

[0099] ω sar = Sigmoid(BN(Conv 1×1 (SAR Feature)));

[0100] ω opt = Sigmoid(BN(Conv 1×1 (OPT Feature)));

[0101]

[0102] T = Sigmoid(AvgPool(Add Feature));

[0103] wherein, denotes element-wise summation, denotes element-wise multiplication, Conv 1×1 (·) denotes 1x1 convolution, AvgPool(·) denotes global average pooling, and Sigmoid(·) denotes an activation function.

[0104] wherein the process of the strong features High Feature and the weak features LowFeature can be represented as

[0105]

[0106]

[0107] wherein, denotes element-wise summation, denotes element-wise multiplication, and Threshold(A, B) denotes comparing the weight A with the threshold B.

[0108] In the embodiment, the multi-source remote sensing image segmentation network model is trained by using the multi-source remote sensing image segmentation dataset, and a trained multi-source remote sensing image segmentation network model is obtained, specifically including: inputting the optical image and the SAR image in the multi-source remote sensing image segmentation dataset into the multi-source remote sensing image segmentation network model in pairs to obtain a predicted segmentation result; calculating a loss function according to the predicted segmentation result and a corresponding real object segmentation map; optimizing the multi-source remote sensing image segmentation network model according to the loss function until the loss of the multi-source remote sensing image segmentation network model converges, and obtaining the trained multi-source remote sensing image segmentation network model.

[0109] In a second aspect, the present application further provides a multi-source remote sensing image segmentation network (PVMSNet) model based on VMamba and pseudo twin network, including: an encoding network, a feature enhancement reconstruction network and a decoding network.

[0110] The encoding network includes an optical encoding branch and a SAR encoding branch, the optical encoding branch includes a first Patch Embedding layer and n layers of VSSLayer connected in series, and the SAR encoding branch includes a second Patch Embedding layer and n layers of first Re-SVSSLayer connected in series; the input end of the first Patch Embedding layer is the input end of the optical encoding branch, the input end of the second Patch Embedding layer is the input end of the SAR encoding branch, and the output ends of the last layer of the optical encoding branch and the last layer of the SAR encoding branch are added and connected to the input end of the decoder; wherein n is a positive integer greater than or equal to 1;

[0111] The decoding network includes n second Re-SVSSLayers and a Final Projiction layer connected in series, the input end of the first second Re-SVSSLayer is the input end of the decoder, and the output end of the Final Projiction layer is the output end of the decoder.

[0112] The feature enhancement reconstruction network includes n feature enhancement reconstruction modules, the output ends of the n feature enhancement reconstruction modules are jump-connected to the input ends of the n second Re-SVSSLayers one by one, the input ends of the n feature enhancement reconstruction modules are connected to the output ends of the first n layers of the optical encoding branch one by one, and the input ends of the n feature enhancement reconstruction modules are also connected to the output ends of the first n layers of the SAR encoding branch one by one.

[0113] In the embodiment, the VSSLayer includes two VSSBlocks connected in series; the first Re-SVSSLayer and the second Re-SVSSLayer each include a residual connection of two SimAM VSSBlocks, and the SimAM VSSBlock is a VSSBlock in which a parameter-free attention mechanism SimAM is introduced.

[0114] wherein n is 2-6. In one specific embodiment, n is 4.

[0115] In the embodiment, the feature enhancement reconstruction module includes an FEM module and an FRM module; the input ends of the FEM module and the FRM module are shared, and the output end of the FEM module and the output end of the FRM module are connected in element-wise addition, and the element-wise addition connection point is taken as the output end of the feature enhancement reconstruction module.

[0116] In the enhancement reconstruction process of the feature enhancement reconstruction module, the FEM module is used to one-to-one correspondingly enhance and fuse the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, to obtain FEM features; the FRM module is used to one-to-one correspondingly enhance the interaction between the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, and separate the strong information and the weak information between the optical features and the SAR features, and process the strong information and the weak information respectively, to realize efficient aggregation of the optical features and the SAR features, to obtain FRM features; the output of the FEM module and the output of the FRM module are added in element-wise, and the element-wise addition result is taken as the final output feature of the feature enhancement reconstruction module.

[0117] In the embodiment, the FEM module is used to one-to-one correspondingly enhance and fuse the optical features of the outputs of the first n layers of the optical encoding branch and the SAR features of the outputs of the first n layers of the SAR encoding branch, specifically including:

[0118] The corresponding optical features and the corresponding SAR features are respectively passed through a 1x1 convolution to obtain intermediate optical features and intermediate SAR features; the intermediate optical features and the intermediate SAR features are added in element-wise to obtain preliminary fusion features; a 1x1 convolution is used to compress the preliminary fusion features into a single channel, a spatial aggregation weight is generated through sigmoid activation, and the spatial aggregation weight and the preliminary fusion features are multiplied in element-wise to obtain a spatial aggregation feature; the spatial aggregation feature is normalized and weighted to obtain a normalized and weighted feature;

[0119] The optical features and the SAR features are respectively input into two FDMM modules in pairs to maximize the difference of two different features, to obtain first difference maximization features and second difference maximization features; the first difference maximization features, the second difference maximization features and the normalized weighted features are input into an SMN module together to obtain FEM features.

[0120] In the embodiment, the optical features and the SAR features are respectively input into two FDMM modules in pairs to maximize the difference of two different features, to obtain first difference maximization features and second difference maximization features, specifically including:

[0121] A FDMM module is used to perform pixel-by-pixel subtraction between the optical and SAR feature maps to obtain a difference feature map, and the obtained difference feature map is applied to global maximum pooling, and a global difference weight is generated through sigmoid activation, the global difference weight is multiplied with the optical image element by element, and the result of the element-by-element multiplication is weighted and summed with the optical image to obtain first difference maximization features of the optical image;

[0122] Another FDMM module is used to perform pixel-by-pixel subtraction between the optical and SAR feature maps to obtain a difference feature map, and the obtained difference feature map is applied to global maximum pooling, and a global difference weight is generated through sigmoid activation, the global difference weight is multiplied with the SAR image element by element, and the result of the element-by-element multiplication is weighted and summed with the SAR feature map to obtain second difference maximization features of the SAR image.

[0123] In the embodiment, the first difference maximization features, the second difference maximization features and the normalized weighted features are input into an SMN module together to obtain FEM features, specifically including:

[0124] The element-by-element sum of the first difference maximization features and the second difference maximization features is obtained, and the mean and standard deviation of the element-by-element sum of the first difference maximization features and the second difference maximization features are calculated;

[0125] The rich spatial features are calculated according to the normalized weighted features and the mean and standard deviation of the element-by-element sum of the first difference maximization features and the second difference maximization features; the rich spatial features are linearly transformed using learnable parameters to obtain FEM features.

[0126] wherein, In the formula, f smn represents the rich spatial features, f nwf represents the normalized weighted features, f difdenotes an element-wise summation result of the first and second difference maximization features, and mean(·) and std(·) denote the mean and standard deviation of the element-wise summation result of the first and second difference maximization features, respectively.

[0127] wherein the process of the linear transformation is represented as wherein β and γ represent learnable parameters, denotes an element-wise summation, denotes an element-wise multiplication.

[0128] wherein the process of obtaining the FRM feature comprises: performing batch normalization on the optical feature and the SAR feature respectively, and generating unique weight information of the optical feature and weight information of the SAR feature at each spatial position through a sigmoid function activation; reconstructing the optical feature and the SAR feature in a cross-weighted manner according to the weight information of the optical feature and the weight information of the SAR feature; adding the reconstructed optical feature and the reconstructed SAR feature to obtain a reconstructed addition feature; and generating information weight on each channel by using average pooling and a sigmoid function on the reconstructed addition feature, and taking the information weight on each channel as a feature weight threshold of each channel;

[0129] the weight information ω sar of the optical feature and the weight information ω opt of the SAR feature are compared with the feature weight threshold T of the corresponding channel in the channel dimension, and the optical feature and the SAR feature greater than the feature weight threshold are classified as strong features, and the optical feature and the SAR feature less than the feature weight threshold are classified as weak features; the strong features and the weak features are respectively transformed; and the transformed strong features and the weak features are element-wise added to obtain the FRM feature. Wherein the strong features and the weak features are respectively transformed, specifically comprising: performing convolution processing on the strong features by using a 1x1 convolution; and performing transformation processing on the weak features by using a gating module.

[0130] wherein T=Sigmoid(AvgPool(Add Feature)); wherein T represents the feature weight threshold, AvgPool(·) represents global average pooling, Sigmoid(·) represents an activation function, and Add Feature represents the reconstructed addition feature;

[0131] wherein, wherein SARFeature represents the SAR feature, and OPT Feature represents the optical feature, denotes an element-wise summation, denotes an element-wise multiplication, ω sar denotes the weight information of the SAR feature, and ω optweight information representing optical features;

[0132] wherein, wherein, BN(·) represents batch normalization, Conv 1×1 (·) represents 1x1 convolution.

[0133] The effects of the present application can be further illustrated by the following simulation.

[0134] 1. Simulation conditions

[0135] The hardware platform is: Intel(R) Core(TM) i7-7700 @ 3.20 GHz, 64.0 GB RAM;

[0136] The software platform is: Pycharm.

[0137] 2. Simulation method

[0138] By using the method of the present application, the multi-source remote sensing image segmentation dataset is segmented; the training set is used to train the PVMSNet, and the trained PVMSNet is used for the test set to test the segmentation effect of the model.

[0139] 3. Simulation content and simulation results

[0140] The multi-source remote sensing image segmentation network obtained by the simulation experiment is selected as the DDHR_xian(no cloud) dataset, which includes a training set and a test set. The training set includes 1501 pairs of optical images and SAR images with a size of 128x128 pixels, and the corresponding groundtruth; the test set includes 1002 pairs of optical images and SAR images with a size of 128x128 pixels, and the corresponding groundtruth. As shown in Figure 6 and Figure 7 As shown in the figures, the dataset mainly includes 6 types of land cover, i.e. farmland, building, water, vegetation, road, and background, and the numbers of each land cover type are represented by {1, 2, 3, 4, 5, 0}. In the groundtruth, the values of the pixel points where the land cover type has been determined correspond to black (0, 0, 0) for background, brown (0, 102, 204) for farmland, red (0, 0, 255) for building, blue (255, 0, 0) for water, green (0, 167, 85) for vegetation, and bright blue (255, 255, 0) for road. Baseline refers to a pseudo-twin network architecture built entirely with VSSLayer, i.e. PVMSNet without Re-VSSLayer and FERBlock.

[0141] The simulation experiment is performed by training the PVMSNet using the training set and using the trained PVMSNet for the test set. The segmentation results of the test set are shown in Table 1.

[0142] Table 1. The IoU (%) scores of each category and the overall mIoU, mF1 scores obtained in the experiment in the simulation

[0143]

[0144] As shown in Table 1, the PVMSNet achieves good segmentation results on the DDHR_xian(no cloud) dataset, with the IoU scores of all categories being above 90%, the mIoU reaching 91.03%, and the mF1 reaching 95.03%. Compared with the Baseline, the mIoU scores of all categories are improved, especially the background, with an improvement of 8.04%, the mIoU improved by 1.74%, and the mF1 improved by 1.18%, which fully proves the effectiveness of the modules designed in the present application. It can be seen that the present application has excellent performance in the task of multi-source remote sensing segmentation. Moreover, the method is designed based on VMamba, which provides another baseline of segmentation model for the field of multi-source remote sensing image segmentation in addition to CNN and Transformer, and provides an effective solution to the existing problems of CNN and Transformer.

[0145] Furthermore, in order to further prove the effectiveness of the modules designed in the present application, an ablation experiment is performed on each designed module, and "√" and "×" respectively represent whether the module is added. The specific results are shown in Table 2 and Table 3.

[0146] Table 2. Ablation results of each designed module of PVMSNet

[0147] Re-SVSS Layer FER Block mloU (%) mF1 (%) × × 89.29 93.85 √ × 89.85 94.28 × √ 90.79 94.88 √ √ 91.03 95.03

[0148] Table 3. Ablation results of each designed module of FERBlock

[0149] FEM FRM mloU (%) mF1 (%) × × 89.85 94.28 √ × 90.45 94.65 × √ 90.59 94.75 √ √ 91.03 95.03

[0150] As can be seen from Table 2, the addition of Re-SVSSLayer makes mIoU increase by 0.56%, and mF1 increase by 0.43%, which fully proves the obvious effect of Re-SVSSLayer in suppressing noise and improving target attention. The addition of FERBlock makes mIoU increase by 1.50%, and mF1 increase by 1.03%, which fully proves the effectiveness of the module in fusing corresponding heterogeneous features and effectively utilizing the complementarity of the two kinds of features. The simultaneous use of Re-SVSSLayer and FERBlock makes mIoU increase by 1.74%, and mF1 increase by 1.18%, which fully proves that the two modules can effectively cooperate and greatly improve the processing ability of the model for multi-modal data, thereby greatly improving the performance of the model.

[0151] As can be seen from Table 3, the addition of FEM makes mIoU increase by 0.60%, and mF1 increase by 0.37%, which fully proves that the FEM module enhances the fused features and improves the expression ability of the fused features. The addition of FRM makes mIoU increase by 0.74%, and mF1 increase by 0.47%, which fully proves that the feature interaction in FRM greatly improves the utilization of complementary information by the fused features, and the feature reconstruction effectively reduces the feature redundancy problem of the fused features.

[0152] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network, characterized by: include: Encoding network, feature enhancement reconstruction network and decoding network; The coding network includes an optical coding branch and a SAR coding branch. The optical coding branch includes a first Patch Embedding layer and n layers of VSSLayer connected in series. The SAR coding branch includes a second Patch Embedding layer and n layers of first Re-SVSSLayer connected in series. The input end of the first Patch Embedding layer is the input end of the optical coding branch, and the input end of the second Patch Embedding layer is the input end of the SAR coding branch. The output end of the last layer of the optical coding branch and the output end of the last layer of the SAR coding branch are added and connected to the input end of the decoder. Wherein, n is a positive integer greater than or equal to 1. The decoding network includes n second Re-SVSSLayer and Final Projection layers connected in series. The input of the first second Re-SVSSLayer serves as the input of the decoder, and the output of the Final Projection layer serves as the output of the decoder. The feature enhancement and reconstruction network includes n feature enhancement and reconstruction modules, the output ends of the n feature enhancement and reconstruction modules are jump-connected to the input ends of the n second Re-SVSSLayers in a one-to-one correspondence, the input ends of the n feature enhancement and reconstruction modules are connected to the output ends of the first n layers of the optical coding branch in a one-to-one correspondence, and the input ends of the n feature enhancement and reconstruction modules are also connected to the output ends of the first n layers of the SAR coding branch in a one-to-one correspondence.

2. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 1, characterized in that: VSSLayer consists of two VSSBlocks connected in series; the first Re-SVSSLayer and the second Re-SVSSLayer both include two SimAM VSSBlocks connected with a residual connection, and SimAM VSSBlock is a VSSBlock that introduces the parameter-free attention mechanism SimAM.

3. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 1, characterized in that: n is 2-6; preferably, n is 4.

4. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 1, characterized in that: The feature enhancement and reconstruction module includes an FEM module and an FRM module; the input end of the FEM module and the input end of the FRM module are shared and serve as the input end of the feature enhancement and reconstruction module, the output end of the FEM module and the output end of the FRM module are connected element by element, and the element-by-element addition connection point is used as the output end of the feature enhancement and reconstruction module; Preferably, in the reconstruction process of the feature enhancement and reconstruction module, the optical features of the output of the first n layers of the optical encoding branch and the SAR features of the output of the first n layers of the SAR encoding branch are enhanced and fused one by one using the FEM module to obtain FEM features; The FRM module is used to enhance the interaction between the optical features of the output of the first n layers of the optical encoding branch and the SAR features of the output of the first n layers of the SAR encoding branch in a one-to-one correspondence. The strong and weak information between the optical and SAR features are separated and processed separately to achieve efficient aggregation of the optical and SAR features to obtain the FRM features. The output of the FEM module and the output of the FRM module are added element by element, and the result of the element-by-element addition is used as the output of the feature enhancement and reconstruction module.

5. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 4, characterized in that: The FEM module is used to enhance and fuse the optical features of the output of the first n layers of the optical encoding branch and the SAR features of the output of the first n layers of the SAR encoding branch one by one. Specifically, the following steps are performed: The corresponding optical features and the corresponding SAR features are respectively subjected to a 1×1 convolution to obtain intermediate optical features and intermediate SAR features; the intermediate optical features and the intermediate SAR features are added element by element to obtain the preliminary fusion features; Use 1×1 convolution to compress the preliminary fusion features into a single channel, and then generate spatial aggregation weights through sigmoid activation, and multiply the spatial aggregation weights and the preliminary fusion features element by element to obtain spatial aggregation features; normalize and weight the spatial aggregation features to obtain normalized weighted features; The optical features and SAR features are respectively sent into two FDMM modules in pairs to maximize the difference between the two different features to obtain the first difference-maximized feature and the second difference-maximized feature; The first differentiated maximization feature, the second differentiated maximization feature and the normalized weighted feature are input into the SMN module together to obtain the FEM feature.

6. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 4, characterized in that: The optical features and SAR features are sent into two FDMM modules in pairs to maximize the difference between the two different features to obtain the first difference-maximized feature and the second difference-maximized feature, specifically including: A FDMM module is used to perform pixel-by-pixel subtraction between the optical and SAR feature maps to obtain a differential feature map. The obtained differential feature map is subjected to global maximum pooling, and a global differential weight is generated through sigmoid activation. The global differential weight is element-wise multiplied with the optical image, and the result of the element-wise multiplication is weighted summed with the optical image to obtain the first difference maximization feature of the optical image. Another FDMM module is used to perform pixel-by-pixel subtraction between the optical and SAR feature maps to obtain a differential feature map. The obtained differential feature map is subjected to global maximum pooling, and a global differential weight is generated through sigmoid activation. The global differential weight is element-wise multiplied with the SAR image, and the result of the element-wise multiplication is weightedly summed with the SAR feature map to obtain the second difference maximization feature of the SAR image.

7. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 4, characterized in that: The first differentiated maximization feature, the second differentiated maximization feature, and the normalized weighted feature are input into the SMN module to obtain the FEM feature, which specifically includes: Element-wise summing of the first difference-maximizing feature and the second difference-maximizing feature to obtain an element-wise summing result of the first difference-maximizing feature and the second difference-maximizing feature; Calculate the mean and standard deviation of the element-by-element summation results of the first difference maximizing feature and the second difference maximizing feature; The enriched spatial features are calculated based on the normalized weighted features and the mean and standard deviation of the element-by-element summation results of the first difference maximization feature and the second difference maximization feature; Use learnable parameters to linearly transform the rich spatial features to obtain FEM features; Preferably, Where, f smn represents rich spatial features, f nwf represents the normalized weighted feature, f dif represents the element-wise sum of the first difference maximizing feature and the second difference maximizing feature, mean(·) and std(·) represent the mean and standard deviation of the element-wise sum of the first difference maximizing feature and the second difference maximizing feature, respectively; Preferably, the linear transformation process is represented as FEM Where β and γ represent learnable parameters. represents element-wise summation, Represents element-wise multiplication.

8. The multi-source remote sensing image segmentation network model based on VMamba and pseudo-twin network according to claim 4, characterized in that: The FRM module is used to enhance the interaction between the optical features of the output of the first n layers of the optical encoding branch and the SAR features of the output of the first n layers of the SAR encoding branch in a one-to-one manner. The strong and weak information between the optical and SAR features are separated and processed separately to achieve efficient aggregation of the optical and SAR features and obtain FRM features. Specifically, the following steps are performed: The optical and SAR features are batch normalized and activated by a sigmoid function to generate unique optical and SAR feature weight information at each spatial location. Based on the optical and SAR feature weight information, the optical and SAR features are reconstructed using a cross-weighted approach. The reconstructed optical features and the reconstructed SAR features are added together to obtain the reconstructed summed features. The reconstructed summed features are then subjected to average pooling and sigmoid function to generate the information weight on each channel, and the information weight on each channel is used as the feature weight threshold of each channel. The weight information of the optical features and the weight information of the SAR features are compared with the feature weight threshold of the corresponding channel in the channel dimension. The optical features and SAR features greater than the feature weight threshold are classified as strong features, and the optical features and SAR features less than the feature weight threshold are classified as weak features. The strong features and weak features are transformed respectively; the transformed strong features and weak features are added element by element to obtain the FRM features; Preferably, the strong features and the weak features are transformed separately, specifically including: performing convolution processing on the strong features using 1×1 convolution; processing the weak features using a gating module; Preferably, T=Sigmoid(AvgPool(Add Feature)); where T represents the feature weight threshold, AvgPool(·) represents global average pooling, Sigmoid(·) represents the activation function, and Add Feature represents the reconstructed additive feature; in, In the formula, SAR Feature represents SAR feature, OPT Feature represents optical feature, represents element-wise summation, represents element-wise multiplication, ω sar Represents the weight information of SAR features, ω opt Representing weight information of optical features; in, Where BN(·) represents batch normalization, Conv 1×1 (·) denotes 1×1 convolution.

9. A multi-source remote sensing image segmentation method based on VMamba and pseudo-twin network, characterized in that: include: Acquire a multi-source remote sensing image segmentation dataset; wherein the multi-source remote sensing image segmentation dataset includes paired optical remote sensing images and SAR images and corresponding ground object real labeling maps; Divide the multi-source remote sensing image segmentation dataset into training set and test set; Constructing a multi-source remote sensing image segmentation network model according to any one of claims 1 to 8; The multi-source remote sensing image segmentation network model is trained and tested in sequence using the training set and the test set to obtain a trained multi-source remote sensing image segmentation network model; The trained multi-source remote sensing image segmentation network model is used to segment the multi-source remote sensing image to obtain multi-category ground object segmentation results.

10. A multi-source remote sensing image segmentation system based on VMamba and pseudo-twin network, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-source remote sensing image segmentation method based on VMamba and pseudo-twin network as claimed in claim 9 are implemented.

Images