Multisource remote sensing image semantic segmentation method based on Transform, Mama and diffusion model

Abstract

The invention discloses a multi-source remote sensing image semantic segmentation method based on Transform, Mama and a diffusion model, belongs to the technical field of electric digital data processing, graphic data reading and image data processing, and is used for extracting image information of a remote sensing image. Comprising the following steps: acquiring remote sensing images, constructing an image information extraction network based on the remote sensing images, and respectively inputting the remote sensing images in a training set, a verification set and a test set into the constructed remote sensing image information extraction network based on Transform, Mama and diffusion models. According to the method, the RGB features and the depth features are subjected to Transform feature extraction, and a cross Mama method is provided to perform cross enhancement on the RGB features and the depth features; according to the method, a row and column pooling Transform module is used for replacing a traditional Transform module, and detail texture information and large-scale structure information of a feature map in the vertical direction and the horizontal direction are extracted at the same time; according to the invention, a cross Mama attention feature fusion module is used to replace a traditional Mama module, and the modeling capability for complex scenes and diversified targets is enhanced.

Description

Technical Field

[0001] The invention relates to a remote sensing image semantic segmentation method, in particular to a remote sensing image method of a diffusion model, Mamba and an attention mechanism, and belongs to the technical fields of electronic digital data processing, graphic data reading and image data processing. Background Art

[0002] Remote sensing images refer to image data of the earth's surface acquired by remote sensing or high-altitude platforms such as aircraft, satellites or drones. They can provide a wide range of spatial and frequency information for applications such as studying the characteristics, changes and classification of objects on the earth's surface. Remote sensing images are widely used in geographic information systems, environmental monitoring, urban planning and agricultural monitoring, providing rich spatiotemporal information for studying surface characteristics, changes and land cover classification. Remote sensing image semantic segmentation is a task that classifies each pixel in a remote sensing image into different object categories, by providing semantic and positioning information clues for ground objects of interest. This segmentation plays a vital role in applications such as road extraction, urban planning and environmental monitoring. The challenges include the large variety of remote sensing images, wide spatial distribution, strong spatial correlation, complex background and large intra-class differences. Traditional methods rely on image processing and machine learning techniques, which usually require manual feature extraction and classifier parameter tuning, which may limit their effectiveness, especially in complex categories and noisy backgrounds in large-scale high-resolution images. In addition, traditional segmentation methods have difficulty in processing the global information of the image, are too computationally intensive and have difficulty in utilizing more additional information.

[0003] Convolutional neural networks play an important role in computer vision and are a mature feature extraction method. Through multi-layer convolution, the texture, shape and spectral features of objects can be extracted from raw pixels, and the local position information relationship between pixels and their surrounding pixels can be effectively captured. For example, the fully convolutional network U-Net, through the encoder-decoder structure design, gradually downsamples the input image and encodes it into a low-dimensional feature representation, and then gradually upsamples and decodes it into a segmentation result of the same size as the input image, thereby achieving accurate segmentation. Another algorithm, DeepLab, introduces a variable spatial expansion rate through a dilated convolution structure to expand the receptive field and capture a wider range of context information. The pyramid pooling network (PSPNet) uses pooling operations of different scales to extract global context information and fuses multi-scale features through a pyramid pooling module. Although convolutional neural networks are widely used in semantic segmentation, they still have some defects. Due to the limitation of the local receptive field, when the intra-class variance of the image is large, the convolutional neural network is weak in extracting global context and spatial correlation information, and the segmentation effect of large-scale object boundaries and details is poor.

[0004] Diffusion models occupy an important position in generation tasks and are one of the effective methods for generating high-quality data. Diffusion models use the process of gradually adding noise and denoising to gradually transform data into noise through a predefined diffusion process, and then generate data from noise through a reverse diffusion process. This bidirectional process can capture the complex distribution characteristics of the data, thereby generating high-fidelity samples. These diffusion models were originally widely used in fields such as image generation, audio synthesis, and text generation. Since the beginning of 2023, the application of diffusion models in semantic segmentation has gradually matured, and researchers have proposed some improvement methods, most of which focus on designing efficient denoising model structures, such as variational autoencoders (VAE), generative adversarial networks (GANs), or Transformer architectures. The success of diffusion models is due to the combination of these advanced network structures, which significantly improves their performance. Through the bidirectional noise addition and denoising process, diffusion models can effectively capture the complex distribution characteristics of data and generate high-fidelity image samples. These models have shown significant advantages in the field of image segmentation, especially in semantic segmentation tasks in fields such as medical images, autonomous driving, and remote sensing images. Although the diffusion model has certain application space in the field of semantic segmentation, it still faces challenges in processing high-dimensional data (such as high-resolution images) and long-range dependencies. In addition, the U-shaped networks used in the diffusion models applied to remote sensing images are all based on convolution operations, which have too many parameters and too much computation.

[0005] Transformer is a deep learning model based on attention mechanism, which can accurately capture the dependencies between different positions in the input sequence. Transformer uses attention mechanism to effectively capture the long-distance dependencies between pixels in the image, so as to better understand the geographic features and spatial structure of remote sensing images. Attention mechanism captures global information by calculating the similarity between all pixels in the image. The position and features of each pixel are compared with the positions and features of all other pixels to determine the dependencies between them. The core advantage of attention mechanism is that it is suitable for parallel computing and has high efficiency. At the same time, it can capture the dependencies between long-distance elements in the sequence, overcoming the limitations of convolutional neural networks and recurrent neural networks in dealing with long-distance dependencies. Attention mechanism is widely used in visual tasks such as image classification, object detection and semantic segmentation. For example, models such as Visual Transformer (ViT) use attention mechanism to achieve excellent performance in image classification and segmentation tasks. Cross-attention mechanism is used to process multimodal data, guiding and adjusting the information of another modality through the information of one modality. However, pure attention mechanism approach may be difficult to fully capture complex spatial and background detail information, especially when dealing with remote sensing images with complex background and large internal differences. In addition, since remote sensing images are usually high-resolution and large-size, semantic segmentation using the attention mechanism may require a lot of computing resources and time, so computational complexity is also a challenge that needs to be faced.

[0006] The advantage of the Laplacian pyramid is that it can decompose images at different scales, thereby retaining the local and global structural information of the image. In image processing and computer vision tasks, the Laplacian pyramid can effectively separate the detailed information of the image through multi-scale representation, which is particularly important for applications such as image enhancement, edge detection, and image compression. In semantic segmentation tasks, the Laplacian pyramid can help the model better capture the hierarchical features of the image, thereby improving the accuracy of the segmentation results. For example, in remote sensing image processing, the edge information of targets such as buildings and roads can be processed more finely using the image decomposed by the Laplacian pyramid, thereby improving the accuracy of semantic segmentation. However, the multi-scale decomposition of the Laplacian pyramid also increases the computational complexity.

[0007] The Mamba model is an advanced model series in the field of computer vision, focusing on visual state space modeling, and aims to improve the performance of image and video analysis tasks. The Mamba model provides long-range dependency modeling capabilities through global receptive field and dynamic weighting, and shows higher computational efficiency in specific tasks compared to traditional convolutional neural networks and Transformers. In addition, compared to convolutional neural networks and Transformers, the Mamba model is good at processing multimodal data and can fuse data from different sensors together to improve segmentation results. In terms of specific applications, Visual Mamba (VMamba) has shown excellent performance in visual state space modeling tasks such as object detection, image denoising, and image super-resolution. The Mamba model has certain advantages in global feature extraction and long-range dependency modeling, but its adaptability and generalization capabilities on different tasks and datasets may sometimes be weak.

[0008] The advantage of the multi-source data method is that it can comprehensively utilize information from different sensors or data sources to provide more comprehensive and accurate analysis results. Adding depth information (DSM) to the data of semantic segmentation of remote sensing images can use multi-source data to reduce the impact of noise and errors in a single data source on the results, enhance the robustness and stability of the model, and multi-source data can provide more detailed background and environmental information, which helps to perform semantic segmentation more accurately and improve the accuracy of segmentation results. The multi-source data method is widely used. For example, in medical image analysis, combining multiple imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI) and ultrasound can improve the accuracy of tumor detection and organ segmentation and the diagnostic effect. In autonomous driving, the use of multi-source data such as cameras, lidar and radar for accurate segmentation of roads, pedestrians and vehicles can enhance the perception ability of autonomous driving systems. However, the use of multi-source data may lead to an increase in computational complexity, so a more lightweight feature extraction and fusion network is needed. Summary of the invention

[0009] The purpose of the present invention is to provide a multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model, so as to solve the problems that the remote sensing semantic segmentation method in the prior art fails to make full use of the directional information, global information and local information of multi-source information, and the multi-source information integration is difficult and the calculation amount is too large, thereby improving the semantic segmentation effect of remote sensing images.

[0010] A multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model, including preparing a data set, adding forward noise, building a network model, training the network model, verifying the network model and outputting semantic segmentation; building a remote sensing image semantic segmentation network model includes designing an encoder-decoder hybrid architecture based on Transformer and Mamba, the encoder is composed of an RGB information branch, a depth information branch and a noisy semantic segmentation information branch in parallel, and the features of the three branches are fused and then the decoder is used to obtain a semantic segmentation map of the remote sensing image; Improve the time step position coding, combine Gaussian position coding and trigonometric function position coding, and embed the time step into the model; Construct a row-column pooling Transformer module, use the row-column attention in the row-column pooling Transformer module to extract RGB information, depth information, and vertical and horizontal detail information and large-scale structure information containing noisy semantic segmentation information, and fuse the information in these directions; Construct a cross-Mamba attention feature fusion module. The RGB information branch and the depth information branch synchronously process the input feature map, and use the Laplacian pyramid to generate high-resolution and low-resolution feature maps. Use the multi-resolution cross-Mamba module to cross-enhance the RGB features and depth features of the same resolution, and merge feature maps of different resolutions. Use the feature weighted mask attention fusion module to perform feature fusion between the RGB information branch and the depth information branch. Construct a multi-resolution cross-Mamba module, and cross-enhance the RGB features and depth features of the same resolution by cross-selectively scanning the Mamba module; Construct a feature weighted mask attention fusion module and introduce feature weighted masks to enhance the effect of feature fusion by giving higher weights to important feature information; Construct a decoder module to achieve feature downsampling.

[0011] The embedding process of the time step includes Gaussian position encoding, trigonometric position encoding, Swish activation function and 2 linear layers: PE=linear(Swish(linear(PE g +λPE s ))); In the formula is the time step position encoding, linear(.) represents the linear layer, Swish represents the Swish activation function, PE g is Gaussian position encoding, PE s is the trigonometric function position encoding, and λ is a custom variable parameter; Gaussian position encoding part: μ T,i =XT+Yi+Z;σ T,i =AT+Bi+C; Where i is the i-th element of the sequence, X, Y, Z, A, B, and C are all adaptive parameters, T is the specific position of the time step t in the sequence, the mean μ represents the center position of the Gaussian distribution, and the standard deviation σ represents the width or extension range of the Gaussian distribution. The mean μ and standard deviation σ are (1, t, d model ), where d model is the embedding dimension of the feature map, PE g is the Gaussian position encoding; Trigonometric function position encoding part: Where T is the specific position of the time step t in this sequence, 2i represents an even number, 2i+1 represents an odd number, and d model is the dimension of the data.

[0012] The row-column pooling Transformer module includes a linear layer, RMS layer normalization, a row-column pooling attention module, a multi-layer perceptron, and a skip connection; For x, the time step position encoding is embedded through nonlinear activation function and RMS layer normalization, and column attention operations are performed by pooling rows and columns of features of different scales respectively, and shortcut connections are added in the row-column pooling Transformer module: A=rcAttention(RMS(SqrtSwish(x),linear(PE))); Y=Sum(A,MLP(RMS(A,linear(PE)))); Wherein, SqrtSwish(x) represents the Swish activation function of the present invention, RMS(.) represents the root mean square layer normalization, The time step position encoding is embedded in the input feature and normalized by the root mean square layer. rcAttention(.) represents the row-column pooling attention module, A is the output feature of the row-column pooling attention module, MLP(.) represents the multi-layer perceptron, and Sum(.) represents the dimension addition.

[0013] Root mean square layer normalization (RMS Norm) replaces the layer normalization layer of the traditional Transformer module. The learnable parameters include not only the scaling factor σ and offset β obtained by input features, but also the scaling factor ω and offset ξ obtained by regressing the embedding vector of time step t; these embedding vectors are processed by a nonlinear activation function (Swish activation function), which weights the input value and outputs the weighted result: Among them, x i represents a vector representation of the input feature, R(x) represents the root mean square of the input feature, represents the root mean square of a vector without scaling and offset, rms(x i ,t) i represents the root mean square after scaling and offset, ε is a small constant for numerical stability, δ and β are normalization parameters adaptively adjusted according to the statistical characteristics of the input features, ω and ξ are the scaling factor and offset obtained by linear regression from the time step position encoding.

[0014] In the row-column pooling attention module, the attention mechanism is divided into two parts: the input features are input to the column pooling attention module and the row pooling attention module respectively, and the time step position encoding is embedded; the row pooling attention module and the column pooling attention module generate scales of and The output feature matrix of is multiplied by the matrix, so as to fuse the output features in the vertical and horizontal directions, so that the model can obtain the direction information and generate the scale The output features of , while downsampling is achieved: RA=RAttention(linear(PE),Conv 1*1 (x));CA=CAttention(linear(PE),Conv 1*1 (x)); A = CA * RA; Where x represents the input feature of the row-column pooling attention module, Conv 1*1 (.) represents 1*1 convolution, It represents the linear change of the time step position encoding, and the time information is added to the row pooling attention module and the column pooling attention module respectively; RAttention(.) and CAttention represent the row pooling attention module and the column pooling attention module respectively, RA and CA are the output features of the two modules, and A is the overall output feature of the row and column pooling attention module; the row pooling attention module and the column pooling attention module are symmetrical modules.

[0015] The row pooling attention module performs row pooling operations on the input features at different scales, generating three inputs x1, x2, and x3 at different scales. Their scales are Use x1, x2, x3 and the time step position encoding to generate the query Q, key K, and value V of the attention mechanism, respectively, with the scale sizes being Then the input x is passed through the linear layer to generate the mapping matrix D, where the number of rows and columns of D are both N. Then the query Q and the mapping matrix D are matrix multiplied to obtain a scale of The result is that the key K and the mapping matrix D are matrix multiplied to obtain a scale of The result is adjusted to the scale by linear interpolation. Finally, they are multiplied by the transpose of the value V and the dimensions are added to generate the scale The output matrix is: x1=MaxPool 2*1 (x); x2 = MaxPool 4*1 (x); x3 = MaxPool 8*1 (x); Q=W Q *(x1+linear(PE(t))); K=W K *(x2+linear(PE(t))); V=W v *(x3+linear(PE(t))); D = linear(x); Where x represents the input feature of the row-column pooling attention module, MaxPool(.) 2*1 、MaxPool(.) 4*1 and MaxPool(.) 8*1 represents the row maximum pooling of 2*1, 4*1, and 8*1 respectively, x1, x2, and x3 are the corresponding pooling results, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and W Q , W K , W V Represent the weight matrices of Q, K, and V respectively, Indicates that the time step position encoding is scaled through the linear layer; the mapping matrix D is generated by the input x through the linear layer; RuLU(.) is the activation function, represents the transpose of the value matrix V and LI represents linear interpolation.

[0016] The Cross-Mamba Attention Feature Fusion Module includes RMS layer normalization, depth-wise separable convolution, Laplacian pyramid module, multi-resolution Cross-Mamba, and feature weighted mask attention fusion module; the input features are RGB features and depth features. This module accepts two features as input and finally generates an output feature: L RH ,L RL =Laplacian(Conv(RMS(x R ,PE));L DH ,L DL =Laplacian(Conv(RMS(x D , PE)); C R ,C D =MCMamba(L RH ,L RL ,L DH ,L DL );Z R ,Z D =Sum(Conv(RMS(C R ,C D )),RMS(x R ,x D )); W = MaskAttention(Z R ,Z D ); In the formula, Respectively, they represent the RMS layer normalization of RGB features and depth features and embedding them into the time step position encoding, Conv(.) represents the depthwise separable convolution, Laplacian(.) represents the Laplacian pyramid module, and L RH , L RL , L DH , L DL They are high-resolution features of RGB features, low-resolution features of RGB features, high-resolution features of depth information, and low-resolution features of depth information. MCMamba(.) represents the multi-resolution cross-Mamba module. R , C D Represents the output features of the multi-resolution cross-Mamba module, and Sum(.) represents dimensional addition. The multi-resolution cross-Mamba module is a four-input dual-output system. Through cross-Mamba, the module realizes cross-enhancement of dual-channel multi-resolution data. MaskAttention(.) represents the feature weighted mask attention fusion module, W is the output of the module, and the feature weighted mask attention fusion module performs deep feature fusion on the cross-enhanced RGB features and depth features.

[0017] The Laplacian pyramid module generates 6 feature maps of different resolutions through Gaussian blur and downsampling, and then processes them into high-resolution and low-resolution feature maps through bilinear transformation and channel splicing: G0=x; L0=x; G i =Reduce(G i-1 ),(i>0);L i =G i -Expand(G i+1 ),(i>0); HC=Channel(L0,BI(L1),BI(L2)); LC=Channel(L3,BI(L4),BI(L5)); HX=Channel(HC,BI(LC)); LX=Channel(MaxPool 8*8 (HC), LC); Where G i represents the i-th layer of the Gaussian pyramid, x is the input feature, Reduce(.) represents the Gaussian blur and downsampling operation; L i represents the i-th layer of the Laplacian pyramid, and Expand(.) represents upsampling, which converts the image from G i+1 Size expansion to G i The size of Channel(.) indicates channel concatenation, BI(.) indicates bilinear interpolation, HC and LC are the concatenation results of the input feature maps with higher and lower resolutions, respectively. Channel(HC, BI(LC)) and Channel(MaxPool 8*8 (HC), LC) combine the high-resolution and low-resolution feature maps, HX and LX represent the high-resolution feature map and low-resolution feature map finally output by the 1-pyramid module.

[0018] The multi-resolution cross-Mamba module includes two cross-selective scanning Mamba modules, two bilinear interpolation and two channel stitching; this module performs feature cross-enhancement on high-resolution RGB features and depth features through the cross-selective scanning Mamba module, and uses another independent cross-selective scanning Mamba module to perform feature cross-enhancement on low-resolution RGB features and depth features; then, the high-resolution and low-resolution RGB features are bilinearly interpolated and channel stitched, and the depth features are processed in the same way, finally generating RGB feature output and depth feature output: HY RGB ,HY DSM =CSMamba(HX RGB ,HX DSM);LY RGB ,LY DSM =CSMamba(LX RGB ,LX DSM ); Y RGB =Channel(HY RGB ,BI(LY RGB )); Y DSM =Channel(HY DSM ,BI(LY DSM )); Where HX RGB and HX DSM They are high-resolution RGB feature input and deep feature input, CSMamba(.) represents the cross selective scanning Mamba module, HX RGB and HX DSM They correspond to high-resolution RGB and depth feature inputs, HY RGB and HY DSM They correspond to high-resolution RGB and depth feature outputs, respectively. RGB and LX DSM They are low-resolution RGB feature input and deep feature input, LY RGB and LY DSM They are low-resolution RGB feature input and deep feature output, Channel(.) represents channel concatenation, BI(.) represents bilinear interpolation; Y RGB and Y DSM They are the fusion features of high and low resolution RGB features and depth features.

[0019] The cross-selective scanning Mamba module is different from the discontinuous four-way scanning method adopted by the classic two-dimensional selective scanning module. The present invention performs continuous scanning in the horizontal and vertical directions respectively. When the end of each row (or column) is scanned, the next mark to be scanned is the mark adjacent to it in the next row (or column); four sequences are obtained after continuous scanning of RGB features and depth features. Two sequences with the same scanning direction in the four sequences are input into the cross-Mamba enhancement module for independent processing. Subsequently, the results are reversely scanned and merged as the final output features, and the input features x1 and x2 are cross-enhanced; finally, the four sequences of the two inputs are reversely reconstructed and added to the corresponding dimensions: CH1, CV1 = HS (z1); CH2, CV2 = HS (z2); H1,H2=CSSM(CI1,CI2); V1,V2=CSSM(CO1,CO2); Y1=Sum(Re(H1,V1)); Y2=Sum(Re(H2,V2)); Wherein HS(z1,z2) represents continuous scanning in two directions, CH1 and CV1 are the output sequences of horizontal and vertical scanning of input feature z1 respectively, CH2 and CV2 are the output sequences of horizontal and vertical scanning of input feature z2 respectively; CSSM(.) represents the cross-Mamba enhancement module, H1 and H2 are the output results generated by the cross-Mamba enhancement module of the horizontal output sequences of z1 and z2 respectively, V1 and O2 are the output feature sequences generated by the cross-Mamba enhancement module of the vertical output sequences of z1 and z2 respectively, Re(.) represents the sequence after rearrangement processing, which is the inverse transform of continuous scanning HS(.), Sum(Re(.)) is to rearrange the processed sequence into a two-dimensional feature map and add the dimensions of the two paths; Y1 and Y2 are the output features of the two input features after the cross-selective scanning Mamba module.

[0020] The cross-Mamba enhancement module can realize the information interaction and cross-enhancement of two inputs; taking the two sequences CH1 and CH2 obtained by horizontal scanning of two input features as examples, CH1 and CH2 are respectively denoted as x1 and x2, B1, C1, D1, Δ1 are functions of sequence x1, B2, C2, D2, Δ2 are functions of sequence x2, parameters A1 and A2 are evolution parameters, which are constructed by HiPPO continuous time memory theory. It stores all previous historical information represented by the coefficient matrix, and parameters Δ1 and Δ2 are sampling time scale parameters. Δ1 is used to convert continuous parameters A1 and B1 into discrete parameters, and Δ2 is used to convert continuous parameters A2 and B2 into discrete parameters, and determines the influence of the previous hidden state on the hidden state used to update the hidden state space state at the next moment, and then performs state space equation operations: Where x 1,t 、x 2,t , are the two input features of the current state, h 1,t 、h 2,t is the current state, h 1,t-1 、h 2,t-1 It is the previous state; is a structured convolution kernel, are the discrete evolution parameters in the state space model of RGB features, are the discretized evolution parameters in the state space model of the deep features, y1 and y2 represent the output features of the two state space models respectively. The cross-Mamba enhancement module realizes the information interaction and cross-enhancement of the two input sequences by exchanging the output matrices C1 and C2 of the two state space models and the direct transfer matrices D1 and D2.

[0021] The feature weighted mask attention fusion module in the present invention weights the RGB features and the depth features respectively to highlight the important feature areas, and then performs cross-attention fusion features; the feature weighted mask attention fusion module accepts two features as input and finally generates an output feature: A mask ,B mask =Softmax(DSConv(A,B)); A out ,B out =Reshape(A,B)*Reshape(A mask ,B mask ); M=Reshape(Sum(A out ,B out )); Q=Wight(Reshape(A)); K, V=Wight(M); Y = Attention(Q,K,V); Where DSConv(.) represents the depthwise separable convolution, Softmax(.) is the Softmax activation function, which converts the input vector into a probability distribution. Each output value ranges between [0,1], and the sum of all output values ​​is 1; A mask , B mask is a weight mask, which is used to give higher weights to important features; Reshape(.) represents a reshaping operation. Assuming that the shape of input A and B is C*H*W, the predicted A mask and B mask Reshape to K*(H*W), reshape A and B to (H*W)*C, and then perform matrix multiplication to extract K global context features, where K is a custom parameter; A out , B out The output result of the weighted mask operation, the shape is K*C, Sum(.) means dimension addition, M is A out , B outThe reshaped output features after dimension addition have a shape of C*2K, where K, V=Wight(M) means that the key K and value V of the attention mechanism are obtained based on M, Q=Wight(Reshape(A) means that the query Q of the attention mechanism is obtained based on the input A, and Y=Attention(Q,K,V) means that the key K, value V and query Q are input into the cross attention mechanism.

[0022] The decoder module consists of a convolutional layer, a linear layer, upsampling, and a nonlinear activation function (sigmoid): Y=sigmoid(upsamping(linear(DSConv(x)))); Where SDConv(.) is the depthwise separable convolution, linear(.) is the linear layer, upsampling(.) is the upsampling, sigmoid(.) is the activation function, Y is the output feature of the decoder module, and the upsampling module uses the bilinear interpolation method.

[0023] After completing the construction of the remote sensing image semantic segmentation network model, training is performed. A remote sensing image semantic segmentation label is sampled from the training data, and then a time step t is randomly selected. A noise sample ε is randomly extracted from the Gaussian distribution. This noise sample represents the noise that may appear at a specific time step and is the interference that the model needs to learn to remove. The randomly extracted noise is added to the remote sensing image semantic segmentation label to obtain noisy input data, providing the model with training samples that are closer to the actual situation. The noise data ε is input into the remote sensing image semantic segmentation network model. The model predicts the noise estimate ∈ by learning the noise characteristics in the training data. t , this estimation reflects the model's understanding and prediction of the noise in the input data. The segmented image output by the diffusion model is compared with the actual segmented image, and the error between the two is calculated. If the error is greater than the set threshold, the parameters are updated by back propagation. It is repeated until the error is less than the set threshold. The parameter update stops and the network parameters are saved. The Adam algorithm is used to solve the parameters. The parameters are updated according to the gradient information during the error back propagation. The missing data in the test set are input into the trained and verified remote sensing image semantic segmentation network model to obtain the reconstructed seismic data.

[0024] Compared with the prior art, the present invention has the following beneficial effects: the present invention combines the methods of Transformer, Mamba, diffusion model and multi-source data, comprehensively extracts the directional features, local features and global features of RGB data and depth data in remote sensing images, and performs cross-enhancement to improve the semantic segmentation effect of remote sensing images. Specifically, the row-column attention in the row-column pooling Transformer module is used to extract the vertical and horizontal detail information and large-scale structural information of RGB data and depth data, and fuse the information in these directions, while reducing the computational complexity. The Laplacian pyramid in the cross-Mamba attention feature fusion module is used to generate feature maps of multiple resolutions through Gaussian blurring and downsampling, and then these feature maps are combined into high-resolution and low-resolution feature maps. The high-resolution feature map retains more detail information and local information, while the low-resolution feature map contains more global information. The multi-resolution cross-Mamba module cross-enhances the RGB features and depth features of the same resolution, which can effectively utilize the complementary information between different modalities, thereby improving the richness of the overall feature expression. Then, the feature maps of different resolutions of the same data source are fused through bilinear interpolation and channel splicing, thereby realizing the integration of local and global information. Finally, cross attention is used to fuse features between multi-source data, and feature weighted masks are introduced to enhance the effect of feature fusion by giving higher weights to important feature information, thereby improving the accuracy of the overall feature expression and achieving better semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Shown is the overall flow chart of semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model of the present invention; Figure 2 The figure shows the schematic diagram of the forward noise addition process and the reverse noise removal process of the diffusion model principle; Figure 3 The figure shows the overall structure diagram of the semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model of the present invention; Figure 4 Shown is an embedding flow chart of the time step of the present invention; Figure 5 The structure diagram of the row-column pooling Transformer module of the present invention is shown; Figure 6 Shown is a structural diagram of the row-column pooling attention module of the present invention; Figure 7 Shown is a structural diagram of the row pooling attention module of the present invention; Figure 8 Shown is a schematic diagram of column pooling and row pooling of the present invention; Fig. 9 The cross-Mamba attention feature fusion module of the present invention is shown; Fig.10 Shown is a Laplace pyramid module structure diagram of the present invention; Fig.11 Shown is a diagram of the structure of a multi-resolution cross-Mamba module of the present invention; Fig.12 Shown is a structural diagram of the cross selective scanning Mamba module of the present invention; Fig.13 Shown is a structural diagram of the cross-Mamba enhancement module of the present invention; Fig.14 Shown is a structural diagram of the feature weighted mask attention fusion module of the present invention; Fig.15 Shown is a structural diagram of a decoder module of the present invention; Fig.16 The flowchart of the present invention using a training set to train semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model; Fig.17 Flowchart of the remote sensing image semantic segmentation steps of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] A multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model, including preparing a seismic data set, building, training and verifying a seismic data reconstruction network model, and performing data reconstruction; building a remote sensing image semantic segmentation network model includes designing an encoder-decoder hybrid architecture based on Transformer and Mamba, wherein the encoder is composed of an RGB information branch, a depth information branch and a noisy semantic segmentation information branch in parallel, and the features of the three branches are fused and then the semantic segmentation map of the remote sensing image is obtained through the decoder; Improve the time step position coding, combine Gaussian position coding and trigonometric function position coding, and embed the time step into the model; Construct a row-column pooling Transformer module, use the row-column attention in the row-column pooling Transformer module to extract RGB information, depth information, and vertical and horizontal detail information and large-scale structure information containing noisy semantic segmentation information, and fuse the information in these directions; Construct a cross-Mamba attention feature fusion module. The RGB information branch and the depth information branch synchronously process the input feature map, and use the Laplacian pyramid to generate high-resolution and low-resolution feature maps. Use the multi-resolution cross-Mamba module to cross-enhance the RGB features and depth features of the same resolution, and merge feature maps of different resolutions. Use the feature weighted mask attention fusion module to perform feature fusion between the RGB information branch and the depth information branch. Construct a multi-resolution cross-Mamba module, and cross-enhance the RGB features and depth features of the same resolution by cross-selectively scanning the Mamba module; Construct a feature weighted mask attention fusion module and introduce feature weighted mask to enhance the effect of feature fusion by giving higher weight to important feature information; Construct a decoder module to achieve feature downsampling.

[0028] The embedding process of the time step includes Gaussian position encoding, trigonometric position encoding, square root Swish activation function and 2 linear layers: PE=linear(Swish(linear(PE g +λPE s ))); In the formula is the time step position encoding, linear(.) represents the linear layer, Swish represents the Swish activation function, PE g is Gaussian position encoding, PE s is the trigonometric function position encoding, and λ is a custom variable parameter; Gaussian position encoding part: μ T,i =XT+Yi+Z;σ T,i =AT+Bi+C; Where i is the i-th element of the sequence, X, Y, Z, A, B, and C are all adaptive parameters, T is the specific position of the time step t in the sequence, and the mean μ represents the center position of the Gaussian distribution. The standard deviation σ represents the width or extension range of the Gaussian distribution, where the mean μ and standard deviation σ are (1, t, d model ) matrix, where d model is the embedding dimension of the feature map, PE g is the Gaussian position encoding; Trigonometric function position encoding part: Where T is the specific position of the time step t in this sequence, 2i represents an even number, 2i+1 represents an odd number, and d model is the dimension of the data.

[0029] The row-column pooling Transformer module includes a linear layer, RMS layer normalization, a row-column pooling attention module, a multi-layer perceptron, and a skip connection; For x, the time step position encoding is embedded through nonlinear activation function and RMS layer normalization, and column attention operations are performed by pooling rows and columns of features of different scales respectively, and shortcut connections are added in the row-column pooling Transformer module: A=rcAttention(RMS(SqrtSwish(x),linear(PE))); Y=Sum(A,MLP(RMS(A,linear(PE)))); Where SqrtSwish(x) represents the Swish activation function of the present invention, RMS(.) represents the root mean square layer normalization, The time step position encoding is embedded in the input feature and normalized by the root mean square layer. rcAttention(.) represents the row-column pooling attention module, A is the output feature of the row-column pooling attention module, MLP(.) represents the multi-layer perceptron, and Sum(.) represents the dimension addition.

[0030] Root mean square layer normalization (RMS Norm) replaces the layer normalization layer of the traditional Transformer module. The learnable parameters include not only the scaling factor σ and offset β obtained by input features, but also the scaling factor ω and offset ξ obtained by regressing the embedding vector of time step t; these embedding vectors are processed by a nonlinear activation function (Swish activation function), which weights the input value and outputs the weighted result: Where x i represents a vector representation of the input feature, R(x) represents the root mean square of the input feature, represents the root mean square of a vector without scaling and offset, rms(x i ,t) i represents the root mean square after scaling and offset, ε is a small constant for numerical stability, δ and β are normalization parameters adaptively adjusted according to the statistical characteristics of the input features, ω and ξ are the scaling factor and offset obtained by linear regression from the time step position encoding.

[0031] In the row-column pooling attention module, the attention mechanism is divided into two parts: row pooling attention and column pooling attention. The input features are input to the row pooling attention module and the column pooling attention module respectively, and the time step position encoding is embedded. The row pooling attention module and the column pooling attention module generate scales of and The output feature matrix of is multiplied by the matrix, so as to fuse the output features in the vertical and horizontal directions, so that the model can obtain the direction information and generate the scale The output features of , while downsampling is achieved: RA=RAttention(linear(PE),Conv 1*1 (x));CA=CAttention(linear(PE),Conv 1*1 (x)); A = CA * RA; Where x represents the input feature of the row-column pooling attention module, Conv 1*1 (.) represents 1*1 convolution, It represents the linear change of the time step position encoding, and the time information is added to the row pooling attention module and the column pooling attention module respectively; RAttention(.) and CAttention represent the row pooling attention module and the column pooling attention module respectively, RA and CA are the output features of the two modules, and A is the overall output feature of the row and column pooling attention module; the row pooling attention module and the column pooling attention module are symmetrical modules.

[0032] The row pooling attention module performs row pooling operations of different scales on the input features of (N,N) to generate three inputs of different scales x1, x2, and x3, whose scales are Use x1, x2, x3 and the time step position encoding to generate the query Q, key K, and value V of the attention mechanism, respectively, with the scale sizes being Then the input x is passed through the linear layer to generate the mapping matrix D, where the number of rows and columns of D are both N. Then the query Q and the mapping matrix D are matrix multiplied to obtain a scale of The result is that the key K and the mapping matrix D are matrix multiplied to obtain a scale of The result is adjusted to the scale by linear interpolation. Finally, they are multiplied by the transpose of the value V and the dimensions are added to generate the scale The output matrix is: x1=MaxPool 2*1 (x); x2 = MaxPool 4*1(x); x3 = MaxPool 8*1 (x); Q=W Q *(x1+linear(PE(t))); K=W K *(x2+linear(PE(t))); V=W v *(x3+linear(PE(t))); D = linear(x); Where x represents the input feature of the row-column pooling attention module, MaxPool(.) 2*1 、MaxPool(.) 4*1 and MaxPool(.) 8*1 represents the row maximum pooling of 2*1, 4*1, and 8*1 respectively, x1, x2, and x3 are the corresponding pooling results, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and W Q , W K , W V Represent the weight matrices of Q, K, and V respectively, Indicates that the time step position encoding is scaled through the linear layer; the mapping matrix D is generated by the input x through the linear layer; RuLU(.) is the activation function, represents the transpose of the value matrix V and LI represents linear interpolation.

[0033] The Cross-Mamba Attention Feature Fusion Module includes RMS layer normalization, depth-wise separable convolution, Laplacian pyramid module, multi-resolution Cross-Mamba, and feature weighted mask attention fusion module; the input features are RGB features and depth features. This module accepts two features as input and finally generates an output feature: L RH ,L RL =Laplacian(Conv(RMS(x R , PE)); L DH ,L DL =Laplacian(Conv(RMS(x D , PE)); C R ,C D =MCMamba(L RH ,L RL ,L DH ,L DL ); Z R ,Z D =Sum(Conv(RMS(C R ,CD )),RMS(x R ,x D )); W = MaskAttention(Z R ,Z D ); In the formula, Respectively, they represent the RMS layer normalization of RGB features and depth features and embedding them into the time step position encoding, Conv(.) represents the depthwise separable convolution, Laplacian(.) represents the Laplacian pyramid module, and L RH , L RL , L DH , L DL They are high-resolution features of RGB features, low-resolution features of RGB features, high-resolution features of depth information, and low-resolution features of depth information. MCMamba(.) represents the multi-resolution cross-Mamba module. R , C D Represents the output features of the multi-resolution cross-Mamba module, and Sum(.) represents dimensional addition. The multi-resolution cross-Mamba module is a four-input dual-output system. Through cross-Mamba, the module realizes cross-enhancement of dual-channel multi-resolution data. MaskAttention(.) represents the feature weighted mask attention fusion module, W is the output of the module, and the feature weighted mask attention fusion module performs deep feature fusion on the cross-enhanced RGB features and depth features.

[0034] The Laplacian pyramid module generates 6 feature maps of different resolutions through Gaussian blur and downsampling, and then processes them into high-resolution and low-resolution feature maps through bilinear transformation and channel splicing: G0=x;L0=x; G i =Reduce(G i-1 ),(i>0);L i =G i -Expand(G i+1 ),(i>0); HC=Channel(L0,BI(L1),BI(L2)); LC=Channel(L3,BI(L4),BI(L5)); HX=Channel(HC,BI(LC)); LX=Channel(MaxPool 8*8 (HC), LC); Where G irepresents the i-th layer of the Gaussian pyramid, x is the input feature, Reduce(.) represents the Gaussian blur and downsampling operation; L i represents the i-th layer of the Laplacian pyramid, and Expand(.) represents upsampling, which converts the image from G i+1 Size expansion to G i The size of Channel(.) indicates channel concatenation, BI(.) indicates bilinear interpolation, HC and LC are the concatenation results of the input feature maps with higher and lower resolutions, respectively. Channel(HC, BI(LC)) and Channel(MaxPool 8*8 (HC), LC) combines the high-resolution and low-resolution feature maps, and HX and LX represent the high-resolution feature map and low-resolution feature map finally output by the 1-pyramid module.

[0035] The multi-resolution cross-Mamba module includes two cross-selective scanning Mamba modules, two bilinear interpolation and two channel stitching; this module performs feature cross-enhancement on high-resolution RGB features and depth features through the cross-selective scanning Mamba module, and uses another independent cross-selective scanning Mamba module to perform feature cross-enhancement on low-resolution RGB features and depth features; then, the high-resolution and low-resolution RGB features are bilinearly interpolated and channel stitched, and the depth features are processed in the same way, finally generating RGB feature output and depth feature output: HY RGB ,HY DSM =CSMamba(HX RGB ,HX DSM );LY RGB ,LY DSM =CSMamba(LX RGB ,LX DSM ); Y RGB =Channel(HY RGB ,BI(LY RGB )); Y DSM =Channel(HY DSM ,BI(LY DSM )); Where HX RGB and HX DSM They are high-resolution RGB feature input and deep feature input, CSMamba(.) represents the cross selective scanning Mamba module, HX RGB and HX DSM They correspond to high-resolution RGB and depth feature inputs, HY RGB and HY DSMThey correspond to high-resolution RGB and depth feature outputs, respectively. RGB and LX DSM They are low-resolution RGB feature input and deep feature input, LY RGB and LY DSM They are low-resolution RGB feature input and deep feature output, Channel(.) represents channel concatenation, BI(.) represents bilinear interpolation; Y RGB and Y DSM They are the fusion features of high and low resolution RGB features and depth features.

[0036] The cross-selective scanning Mamba module is different from the discontinuous four-way scanning method adopted by the classic two-dimensional selective scanning module. The present invention performs continuous scanning in the horizontal and vertical directions respectively. When the end of each row (or column) is scanned, the next mark to be scanned is the mark adjacent to it in the next row (or column); four sequences are obtained after continuous scanning of RGB features and depth features. Two sequences with the same scanning direction in the four sequences are input into the cross-Mamba enhancement module for independent processing. Subsequently, the results are reversely scanned and merged as the final output features, and the input features x1 and x2 are cross-enhanced; finally, the four sequences of the two inputs are reversely reconstructed and added to the corresponding dimensions: CH1, CV1 = HS (z1); CH2, CV2 = HS (z2); H1,H2=CSSM(CI1,CI2); V1,V2=CSSM(CO1,CO2); Y1=Sum(Re(H1,V1)); Y2=Sum(Re(H2,V2)); Where HS(z1,z2) represents continuous scanning in two directions, CH1 and CV1 are the output sequences of horizontal and vertical scanning of input feature z1, respectively, and CH2 and CV2 are the output sequences of horizontal and vertical scanning of input feature z2, respectively. CSSM(.) represents the cross-Mamba enhancement module, H1 and H2 are the output results of the horizontal output sequences of z1 and z2 generated by the cross-Mamba enhancement module, V1 and O2 are the output feature sequences of the vertical output sequences of z1 and z2 generated by the cross-Mamba enhancement module, Re(.) represents the sequence after rearrangement processing, which is the inverse transformation of continuous scanning HS(.), Sum(Re(.)) is to rearrange the processed sequence into a two-dimensional feature map and add the dimensions of the two paths; Y1 and Y2 are the output features of the two input features after the cross-selective scanning Mamba module.

[0037] The cross-Mamba enhancement module can realize the information interaction and cross-enhancement of two inputs; taking the two sequences CH1 and CH2 obtained by horizontal scanning of two input features as examples, CH1 and CH2 are respectively denoted as x1 and x2, B1, C1, D1, Δ1 are functions of sequence x1, B2, C2, D2, Δ2 are functions of sequence x2, parameters A1 and A2 are evolution parameters, which are constructed by HiPPO continuous time memory theory. It stores all previous historical information represented by the coefficient matrix, and parameters Δ1 and Δ2 are sampling time scale parameters. Δ1 is used to convert continuous parameters A1 and B1 into discrete parameters, and Δ2 is used to convert continuous parameters A2 and B2 into discrete parameters, and determines the influence of the previous hidden state on the hidden state used to update the hidden state space state at the next moment, and then performs state space equation operations: Where x 1,t 、x 2,t , are the two input features of the current state, h 1,t 、h 2,t is the current state, h 1,t-1 、h 2,t-1 It is the previous state; is a structured convolution kernel, are the discrete evolution parameters in the state space model of RGB features, are the discretized evolution parameters in the state space model of the deep features, y1 and y2 represent the output features of the two state space models respectively. The cross-Mamba enhancement module realizes the information interaction and cross-enhancement of the two input sequences by exchanging the output matrices C1 and C2 of the two state space models and the direct transfer matrices D1 and D2.

[0038] The feature weighted mask attention fusion module in the present invention weights the RGB features and the depth features respectively to highlight the important feature areas, and then performs cross-attention fusion features; the feature weighted mask attention fusion module accepts two features as input and finally generates an output feature: A mask ,B mask =Softmax(DSConv(A,B)); A out ,B out =Reshape(A,B)*Reshape(A mask ,B mask ); M=Reshape(Sum(Aout ,B out )); Q=Wight(Reshape(A)); K, V=Wight(M); Y = Attention(Q,K,V); Where DSConv(.) represents the depthwise separable convolution, Softmax(.) is the Softmax activation function, which converts the input vector into a probability distribution. Each output value ranges between [0,1], and the sum of all output values ​​is 1; A mask , B mask is a weight mask used to give higher weights to important features; Reshape(.) represents a reshaping operation. Assuming that the shape of input A and B is C*H*W, the predicted A mask and B mask Reshape to K*(H*W), reshape A and B to (H*W)*C, and then perform matrix multiplication to extract K global context features, where K is a custom parameter; A out , B out is the output result of the weighted mask operation, with a shape of K*C, Sum(.) represents dimension addition, and M is A out , B out The reshaped output features after dimension addition have a shape of C*2K, where K, V=Wight(M) means that the key K and value V of the attention mechanism are obtained based on M, Q=Wight(Reshape(A) means that the query Q of the attention mechanism is obtained based on the input A, and Y=Attention(Q,K,V) means that the key K, value V and query Q are input into the cross attention mechanism.

[0039] The decoder module consists of a convolutional layer, a linear layer, upsampling, and a nonlinear activation function (sigmoid): Y=sigmoid(upsamping(linear(DSConv(x)))); Where DSConv(.) is the depthwise separable convolution, linear(.) is the linear layer, upsamping(.) is the upsampling, sigmoid(.) is the activation function, Y is the output feature of the decoder module, and the upsampling module uses the bilinear interpolation method;

[0040] After completing the construction of the remote sensing image semantic segmentation network model, training is performed. A remote sensing image semantic segmentation label is sampled from the training data, and then a time step t is randomly selected. A noise sample ε is randomly extracted from the Gaussian distribution. This noise sample represents the noise that may appear at a specific time step and is the interference that the model needs to learn to remove. The randomly extracted noise is added to the remote sensing image semantic segmentation label to obtain noisy input data, providing the model with training samples that are closer to the actual situation. The noise data ε is input into the remote sensing image semantic segmentation network model. The model predicts the noise estimate ∈ by learning the noise characteristics in the training data. t , this estimation reflects the model's understanding and prediction of the noise in the input data. The segmented image output by the diffusion model is compared with the actual segmented image, and the error between the two is calculated. If the error is greater than the set threshold, the parameters are updated by back propagation. It is repeated until the error is less than the set threshold. The parameter update stops and the network parameters are saved. The Adam algorithm is used to solve the parameters. The parameters are updated according to the gradient information during the error back propagation. The RGB data and depth data in the training set are input into the trained remote sensing image semantic segmentation network model. The semantic segmentation data output by the model is compared with the accurate semantic segmentation label, and the error between the two is calculated. If the error is greater than the set threshold, the training of the remote sensing image semantic segmentation network model is returned. If the error is less than the set threshold, the verification of the seismic data reconstruction network model is completed: The RGB data and depth data in the test set are input into the trained and verified remote sensing image semantic segmentation network model to obtain the semantic segmentation results of the remote sensing image.

[0041] In the specific implementation of the present invention, a multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model is provided to solve the problems that the remote sensing semantic segmentation method in the prior art fails to fully utilize the directional information, global information and local information of multi-source information, and the multi-source information integration is difficult and the calculation amount is too large, thereby improving the semantic segmentation effect of remote sensing images.

[0042] The backbone network of the multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model of the present invention is a U-type network. The U-type network has a symmetrical encoder-decoder structure, can extract and utilize multi-scale features of data, and fuses the features of the corresponding layers of the encoder and the decoder through jump connections, thereby ensuring the full fusion of feature information and spatial information, enhancing the positioning accuracy of the network for small targets and boundaries, and improving the generalization ability of the model. In order to obtain features of multiple dimensions of semantic segmentation labels, the present invention replaces each encoder module of the U-type network with a row-column pooling Transformer module and a cross Mamba attention feature fusion module to provide more spatial information for the model; and adds an RGB and depth feature fusion part for supplementing conditional information.

[0043] The encoder part of the overall architecture of multi-source remote sensing image semantic segmentation based on Transformer, Mamba and diffusion model designed by the present invention is composed of row-column pooling Transformer module and cross-Mamba attention feature fusion module. The row-column pooling Transformer module extracts features from input features, and fuses RGB features with depth features through the cross-Mamba attention feature fusion module. In order to obtain the features of multi-source data, the row-column pooling Transformer module of the present invention extracts multi-scale features from RGB features, depth features and noisy remote sensing image semantic segmentation data respectively, aiming to improve the accuracy of semantic segmentation through multi-source information, and by performing column attention operations on row pooling features and column pooling features of different scales respectively, it not only provides accurate detail information and large-scale structural information, but also captures the directional information of the image. The cross-Mamba attention feature fusion module of the present invention performs feature cross-enhancement and feature fusion on multi-scale RGB features and depth features. The module performs Gaussian blurring and downsampling of RGB features and depth features to different degrees by introducing a Laplacian pyramid module, generates high-resolution and low-resolution feature maps, and retains the local information and global information of the feature maps respectively. The multi-resolution cross-Mamba module performs cross-Mamba operations on RGB features and depth features of the same resolution, realizes cross-enhancement of RGB features and depth features of the same resolution, and then merges high-resolution and low-resolution feature maps of the same data. Finally, the feature weighted mask attention feature fusion module is used to further enhance features through feature weighted masks and perform feature fusion through cross attention to achieve better semantic segmentation effects.

[0044] The time step encoding method designed by the embedding flowchart of the time step of the present invention is different from the trigonometric function position encoding commonly used in diffusion models. We combine Gaussian position encoding and trigonometric function position encoding. Trigonometric function position encoding is generated by sine and cosine functions, has periodicity and interpretability, and can capture global position information; while Gaussian position encoding provides high-precision and continuous local changes, and better captures the relationship between local features and adjacent positions. The combination of the two can capture global and local time features at the same time, and improve the robustness of time step information. Finally, the Swish activation function is used as a nonlinear correction unit. The Swish activation function is similar to the Sigmoid function, but has a wider output range. Over the entire real number domain, the Swish activation function is continuously differentiable, which helps in gradient calculation during back propagation, thereby making network training more stable. It is adaptive and can automatically adjust the shape of the response curve according to different input ranges to adapt to different data distributions and model requirements.

[0045] The row-column pooling Transformer in the present invention is a self-attention improvement on the conventional visual Transformer. The row-column pooling attention module in the row-column pooling Transformer module of the present invention extracts multi-scale features through column pooling and row pooling, thereby enhancing the ability to capture detail information and large-scale structural information. The row pooling attention module and the column pooling attention module process the features in the vertical and horizontal directions respectively to improve the feature representation capability. The attention mechanism combining column pooling and row pooling operations of different scales can better capture the directional information of the image. Time step position encoding information is added to the attention operation of the row pooling attention module and the column pooling attention module, which provides important information about the degree of noise in the diffusion model, so that the module can effectively handle different noise states. The scale of the input feature of the row-column pooling attention module of the present invention is (N*N), and the row pooling attention module and the column pooling attention module generate scales of respectively. and the scale is The output feature matrix matrix of is matrix multiplied to generate a scale of The output features of the image are downsampled. At the same time, since column pooling and row pooling reduce the dimension of the feature map, the amount of calculation is effectively reduced, thereby reducing the complexity of subsequent attention calculations. This design ensures that the image features are fully captured and understood while maintaining efficient calculations, which helps to improve the performance and accuracy of the model.

[0046] The row-column pooling attention module in the present invention includes a row pooling attention module and a column pooling attention module, which are symmetrically designed modules. Column pooling and row pooling reduce the feature map dimensions in the vertical and horizontal directions, respectively. Taking the row pooling attention module as an example, in a conventional Transformer, the softmax attention mechanism first maps the input matrix (scale is (N, N)) into three different matrices: a query matrix Q, a key matrix K, and a value matrix V (scale is (N, N)). Then, the transposed matrices of Q and K are multiplied, and then normalized by the softmax function, and finally the obtained attention weight matrix is ​​multiplied with V. In the present invention, the row pooling attention module first performs row pooling operations of different scales on the input features to generate three inputs x1, x2, and x3 of different scales, with scales of respectively The query Q and key K are the mappings of x1 and x2, respectively, with scales It is used to extract the detail texture information in the vertical direction, and x3 is the mapping of the value V, with a scale of It is used to extract large-scale structural information in the vertical direction, and introduces time step position encoding when calculating query Q, key K and value V, providing important information about the vertical direction of the diffusion model noise, so that the module can effectively handle different noise states. Then the input x passes through the linear layer to generate the mapping matrix D (scale (N*N)). Matrix multiplication is performed on the query Q and the mapping matrix D to obtain the scale The result is: the key K and the mapping matrix D are matrix multiplied to obtain a scale of The result is then linearly interpolated into Finally, multiply the results by the transpose of the value V and generate a scale of The row pooling attention module extracts vertical detail information and large-scale structural information through multi-scale row pooling, while effectively reducing the amount of computation.

[0047] The cross-Mamba attention feature fusion module of the present invention performs feature cross-enhancement and feature fusion on multi-scale RGB features and depth features. The module introduces a Laplacian pyramid module to perform different degrees of Gaussian blur and downsampling on RGB features and depth features, generating high-resolution and low-resolution feature maps, respectively retaining local information and global information of the feature maps. The multi-resolution cross-Mamba module performs cross-Mamba operations on RGB features and depth features of the same resolution, realizes cross-enhancement of RGB features and depth features of the same resolution, and then merges high-resolution and low-resolution feature maps of the same data. Finally, the feature weighted mask attention feature fusion module is used to further enhance the features through feature weighted masks and perform feature fusion through cross-attention, so as to achieve better semantic segmentation effect.

[0048] The Laplacian pyramid module in the cross-Mamba attention feature fusion module of the present invention uses the Laplacian pyramid method, which is a multi-resolution image processing and analysis technology. It captures the different scale details of the image layer by layer by decomposing the original image into different spatial frequency bands. This decomposition method enables us to apply different operators on different frequency bands, thereby achieving more accurate algorithm optimization. In the Laplacian pyramid, each time you move to the upper layer, the resolution of the image is halved, and the captured details gradually decrease. Too many layers will increase the amount of calculation, while too few layers may not fully reflect the global features. Therefore, selecting a 6-layer pyramid can effectively capture the global features and local details of the image at different scales. By splicing these feature maps, outputs of two spatial frequency bands of high resolution and low resolution can be generated. Specifically, by continuously Gaussian blurring and downsampling, 6 feature maps of different resolutions are generated. Then, these feature maps are processed into high-resolution and low-resolution feature maps through bilinear transformation and channel splicing. The high-resolution feature map retains more details and local information, while the low-resolution feature map contains more global information. In this way, the module can make full use of the information of feature maps of different resolutions, enhance the ability to understand image details and overall structure, and improve the effect and robustness of feature fusion.

[0049] The multi-resolution cross-Mamba module in the cross-Mamba attention feature fusion module of the present invention performs local feature cross-enhancement on high-resolution RGB features and depth features through a cross-selective scanning Mamba module, and uses another independent cross-selective scanning Mamba module to perform global feature cross-enhancement on low-resolution RGB features and depth features. The cross-selective scanning Mamba module realizes the interaction between RGB features and depth features. Finally, the high-resolution and low-resolution feature maps of the same data are merged, thereby making full use of the information of feature maps of different resolutions, fusing local and global information, enhancing the model's ability to understand image details and overall structure, and realizing the interaction between RGB features and depth features, thereby improving the effect and robustness of feature fusion.

[0050] The cross-selective scanning Mamba module in the multi-resolution cross-Mamba module of the present invention adopts vertical and horizontal continuous scanning methods to generate two sequence information from one input feature. During the scanning process, when the end of each row (or column) is reached, the next mark to be scanned will select the mark that is closest to the current mark in space, rather than jumping to the farthest mark in the next row (or column). This continuous scanning method maintains spatial and semantic continuity, reduces the potential risk of information loss, helps to maintain the consistency of image features, and avoids large-scale jumps, thereby improving computational efficiency and feature extraction effects. The cross-Mamba enhancement module cross-enhances the RGB features and depth features from the same scanning direction. This process preliminarily fuses the two input features so that the information between them can be complementary and enriched, thereby improving the feature expression capability. Through the two sets of cross-Mamba enhancement modules, the RGB features and the depth features are deeply interacted, and the model can better understand complex scenes and express complex data, thereby improving the accuracy and reliability of prediction.

[0051] The cross-Mamba enhancement module in the cross-selective scanning Mamba module of the present invention is similar to two parallel Mamba modules, which process the RGB feature sequence x1 and the depth feature sequence x2 respectively. The feature matrix generated by the module includes A1, B1, C1, D1, Δ1 (corresponding to the sequence x1) and A2, B2, C2, D2, Δ2 (corresponding to the sequence x2), wherein B1, C1, D1, Δ1 is a function of the input sequence x1, B2, C2, D2, Δ2 is a function of the sequence x2, and the parameters A1 and A2 are evolution parameters, which are constructed by the HiPPO continuous time memory theory. It stores all previous historical information represented by the coefficient matrix. The module realizes the information interaction of the two input sequences by exchanging the output matrices C1 and C2 of the two state space models and the direct transfer matrices D1 and D2. This design improves the expressiveness of the features, enhances the complementarity and fusion effect of the features, helps to capture more image details, and enables the model to better understand the semantic content of the remote sensing image. This cross-enhancement module enables the features of input sequences x1 and x2 to complement each other through parallel processing and information interaction, thus improving the richness and diversity of features. Specifically, by combining the features of different input sequences, the module enhances the model's ability to capture image details and overall structure, thereby improving the effect of feature fusion. This design not only improves the model's expressiveness, but also improves its ability to understand and process complex remote sensing images, ensuring full utilization and effective fusion of information.

[0052] The feature weighted mask attention fusion module in the cross-Mamba attention feature fusion module of the present invention combines the feature weighted mask operation and the cross-attention feature fusion operation. The feature weighted mask assigns different weights to each spatial position of the two input features through the Softmax function, thereby emphasizing important feature areas and suppressing unimportant areas. This process is similar to the self-attention mechanism, which enhances the model's attention to these areas by assigning higher weights to key areas. The two features are added by dimensions to generate the key K and value V in the cross-attention mechanism, and the query Q of the cross-attention is generated by convolution and reshaping the input features, and the attention calculation is performed with the key and value generated in the previous part. One of the two input information of this module is used as the information source of the query Q. Specifically, in the RGB and deep feature fusion part, the RGB feature is mapped to the query Q of the cross-attention, and in the U-shaped network backbone part, the remote sensing semantic information is mapped to the query Q of the cross-attention.

[0053] The present invention needs to prepare three data sets: a training set, a validation set and a test set, and the ratio of the three is 3:1:1. Among them, the training set is responsible for training the semantic segmentation model of the remote sensing image, the validation set is responsible for adjusting the semantic segmentation model, and the test set is responsible for testing the segmentation effect of the model. The method for constructing the training set is: select multiple image blocks containing RGB information and depth information from the remote sensing image data set. The number of rows and columns of each image block is h, and h and w are set to 256. Pair each original image (including RGB and depth information) with its corresponding annotated data to form a training set. The construction method of the validation set and the test set is the same as that of the training set, except that the image blocks in the validation set must be different from the image blocks in the training set.

[0054] The backbone network of the overall architecture of the semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model designed by the present invention is a U-net network (U-net). The U-net has a symmetrical encoder-decoder structure, which can extract and utilize the multi-scale features of the data, and fuse the features of the corresponding layers of the encoder and decoder through jump connections, thereby ensuring the full fusion between multi-source information and enhancing the network's positioning accuracy for small targets and boundaries. In order to improve the semantic segmentation capability of noisy remote sensing images, this paper replaces each traditional convolution module of the conventional U-net with a row-column pooling Transformer module and a cross Mamba attention feature fusion module, and additionally adds a RGB and depth feature fusion part. The present invention first uses a row-column pooling Transformer module to perform independent feature extraction on RGB features, depth features and semantic segmentation features, and then fuses RGB features with depth features through a cross Mamba attention feature fusion module, and finally further fuses the fused features with semantic segmentation features through another cross Mamba attention feature fusion module. This multi-scale feature cross-enhancement and fusion of multi-source information effectively improves the model's utilization of RGB features, depth features, and semantic segmentation features, thereby improving the semantic segmentation effect of remote sensing images.

[0055] In the present invention, the layer normalization layer of the traditional Transformer module is replaced by the root mean square layer normalization (RMS Norm). This improvement is intended to reduce the computational complexity. The principle is to dynamically adjust the normalization parameters according to the statistical characteristics of the input features to adapt to the distribution of different samples. The root mean square layer normalization reduces the amount of calculation by removing the mean calculation and subtraction steps in the traditional layer normalization, retaining only the scaling operation. In the root mean square layer normalization of the present invention, the learnable parameters include not only the scaling factor σ and the offset β obtained by the input features, but also the scaling factor ω and the offset ξ obtained by regressing the embedded vector of the time step t. These embedded vectors are processed by the nonlinear activation function (square root Swish activation function) proposed by the present invention. The vector weights the input value through the Swish activation function and outputs the weighted result. Compared with other common activation functions, the square root Swish activation function helps to capture complex patterns and relationships through smooth nonlinear characteristics, especially in deep neural networks. In addition, the Swish activation function is adaptive and can automatically adjust the response curve within different input ranges to better adapt to different data distributions and model requirements.

[0056] The row-column pooling Transformer can capture the vertical and horizontal detail information and large-scale structural information of the input data from a holistic perspective, while reducing the computational complexity in the original Transformer through row-column pooling.

[0057] The computational complexity of obtaining the global attention map from the self-attention layer of a conventional Transformer is 5*N 3 *C. The computational complexity of the row pooling attention module of the row-column pooling Transformer of the present invention is For a feature tensor with a length and width of 32 and a channel number of 1 extracted from a remote sensing image data record, the computational complexity of the conventional self-attention module is 163840, and the computational complexity of the row pooling attention module is 59392. It can be found that the row-column pooling attention module effectively reduces the computational complexity.

[0058] The cross-Mamba attention feature fusion module performs cross-enhancement and feature fusion on multi-scale RGB features and depth features. By introducing the Laplacian pyramid module, the RGB features and depth features are Gaussian blurred and downsampled to different degrees to generate high-resolution and low-resolution feature maps, which retain the local information and global information of the feature maps respectively. The multi-resolution cross-Mamba module performs cross-Mamba operations on RGB features and depth features of the same resolution, realizes cross-enhancement of RGB features and depth features of the same resolution, and then merges the high-resolution and low-resolution feature maps of the same data. Finally, the feature weighted mask attention feature fusion module is used to further enhance the features through feature weighted masks and perform feature fusion through cross attention to achieve better semantic segmentation effects.

[0059] The Laplacian pyramid module is a multi-resolution image processing and analysis technology. It decomposes the original image into different spatial frequency bands to capture the details of the image at different scales layer by layer. This decomposition method allows us to apply different operators on different frequency bands, thereby achieving more accurate algorithm optimization. In the Laplacian pyramid, each time you move up a layer, the resolution of the image is halved and the captured details gradually decrease. Too many layers will increase the amount of calculation, while too few layers may not fully reflect the global features. Therefore, choosing a 6-layer pyramid can effectively capture the global features and local details of the image at different scales. By splicing these feature maps, outputs of two spatial frequency bands, high resolution and low resolution, can be generated. Specifically, by continuously Gaussian blurring and downsampling, 6 feature maps of different resolutions are generated. Then, these feature maps are processed into high-resolution and low-resolution feature maps through bilinear transformation and channel splicing. The high-resolution feature map retains more details and local information, while the low-resolution feature map contains more global information. In this way, the module can make full use of the information of feature maps of different resolutions, enhance the ability to understand the details and overall structure of the image, and improve the effect and robustness of feature fusion.

[0060] The cross-selective scanning Mamba module performs local feature cross-enhancement on high-resolution RGB features and depth features, and uses another independent cross-selective scanning Mamba module to perform global feature cross-enhancement on low-resolution RGB features and depth features. The cross-selective scanning Mamba module realizes the interaction between RGB features and depth features. Finally, the high-resolution and low-resolution feature maps of the same data are merged to make full use of the information of feature maps of different resolutions, fuse local and global information, enhance the model's ability to understand image details and overall structure, and realize the interaction between RGB features and depth features, thereby improving the effect and robustness of feature fusion. .

[0061] The cross-selective scanning Mamba module uses vertical and horizontal continuous scanning to generate two sequence information from one input feature. Compared with the two-dimensional selective scanning, the four directions are reduced to two directions. The two-dimensional selective scanning is a single-input single-output system, while the cross-Mamba and attention feature fusion module of the present invention is a dual-input dual-output module and the module adds the cross-Mamba operation. This process preliminarily fuses the two input features so that the information between them can be complementary and enriched, thereby improving the feature expression ability. Through the deep interaction of RGB features and depth features by two sets of cross-Mamba enhancement modules, the model can better understand complex scenes and express complex data, thereby improving the accuracy and reliability of prediction.

[0062] The cross-Mamba enhancement module is similar to two parallel Mamba modules. This module realizes the information interaction of two input sequences by exchanging the output matrices C1 and C2 of the two state-space models and the direct transfer matrices D1 and D2. This design improves the expressiveness of features, enhances the complementarity and fusion effect of features, helps capture more image details, and enables the model to better understand the semantic content of remote sensing images. This cross-enhancement module enables the features of input sequences x1 and x2 to complement each other through parallel processing and information interaction, thereby improving the richness and diversity of features. Specifically, this module enhances the model's ability to capture image details and overall structure by combining the features of different input sequences, thereby improving the effect of feature fusion. This design not only improves the expressiveness of the model, but also improves the ability to understand and process complex remote sensing images, ensuring full utilization and effective fusion of information.

[0063] The feature weighted mask assigns different weights to each spatial position of the two input features through the Softmax function, thereby emphasizing important feature areas and suppressing unimportant areas. This process is similar to the self-attention mechanism, which increases the model's attention to these areas by assigning higher weights to key areas. The two features are added together after dimensioning to generate the key K and value V in the cross-attention mechanism. The cross-attention query Q is generated by convolution and reshaping the input features, and the attention calculation is performed with the key and value generated in the previous part.

[0064] Input the RGB data and depth data in the training set into the constructed network model. The output of the network is the semantic segmentation data of the network remote sensing image described in the present invention. The remote sensing image semantic segmentation data is compared with the semantic segmentation label, and the error between the two is calculated at the same time. If the error is greater than the set threshold (the threshold value is set according to the actual situation), the parameters are back-propagated and updated. Repeated iteration until the error is less than the set threshold, stop updating the parameters and save the network parameters. The present invention uses the Adam algorithm to solve the parameters, and when the error is back-propagated, the parameters are updated according to the gradient information. The specific form of the loss function is shown in the following formula: Where Loss(·) is the loss function; ξ is the learnable parameter of the model; is the Gaussian noise added at the tth moment; x0 is the input remote sensing image semantic segmentation data; x t is the semantic segmentation data of the remote sensing image after the forward process and noise at the tth moment; ∈ ξ (x t ,t) is the predicted noise; t~Uniform(1,…,1000) is the time step at a moment, It is the cumulative multiplication of heavy parameters, which is used to maintain a certain degree of data details. By minimizing this loss function, MSE is used as the loss function to calculate the difference between the real noise and the estimated noise; n is the number of samples; ∈ i is the true noise value of the i-th sample; is the predicted noise value of the i-th sample; Represents the sum of all samples; the smaller the value of the MSE loss function, the smaller the difference between the model's prediction result and the true value, that is, the better the performance of the model; the diffusion network model can learn the ability to predict remote sensing semantic segmentation noise until the model converges, thereby achieving denoising of semantic segmentation data.

[0065] The RGB data and depth data in the validation set are input into the constructed network model. The output of the network is the semantic segmentation data of the remote sensing image described in the present invention. The semantic segmentation data of the remote sensing image is compared with the semantic segmentation label, and the error between the two is calculated. If the error is less than the set threshold (the threshold value is set according to the actual situation), it proves that the trained network parameters have reached the optimal value, and the semantic segmentation of the remote sensing image can be tested. If the error is greater than the set threshold, it proves that the trained network parameters have not reached the optimal value, and the network is returned to be retrained until a good reconstruction effect can be achieved in the validation stage.

[0066] The RGB data and depth data in the test set are input into the trained and verified network of the present invention, and the output of the network is the semantic segmentation data of the remote sensing image according to the present invention.

[0067] In the present invention, the overall process of semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model is as follows: Figure 1 As shown in FIG, it includes 6 steps, namely, preparing a data set, forward denoising, building a network model, training a network model, verifying the network model, and outputting semantic segmentation results. The forward denoising process and the reverse denoising process of the diffusion model principle of the present invention are shown in FIG. Figure 2 As shown in the figure, in the forward denoising process, a small amount of noise is gradually added to the remote sensing semantic segmentation image until the data conforms to the Gaussian distribution. The reverse denoising process needs to be trained and learned through a deep learning network. Through prediction and denoising operations, the model gradually restores the noisy data to its original state. This process will gradually reduce the noise in the data and make the data gradually approach the real data distribution. The overall architecture of the multi-source remote sensing image semantic segmentation based on Transformer, Mamba and diffusion model of the present invention is shown in the figure. Figure 3 As shown, RGB data, depth data and noisy remote sensing semantic segmentation data are respectively subjected to feature extraction by row-column pooling Transformer modules, and then, RGB data and depth data are subjected to feature fusion by cross-Mamba attention feature fusion module. Finally, the fusion information of RGB data and depth data and the noisy remote sensing semantic segmentation map data are input into the cross-Mamba attention feature fusion module of the trunk part for deeper feature fusion. The noisy remote sensing semantic segmentation data will pass through N row-column pooling Transformer modules and N cross-Mamba attention feature fusion modules in sequence, where N is the number of network layers of the present invention. By increasing N, a greater feature extraction depth and better segmentation effect can be achieved, but this will lead to greater computational complexity. Therefore, in order to balance the depth of feature extraction and computational cost, N=6 is set in this example.

[0068] The embedding process of the time step of the present invention is as follows Figure 4As shown, it includes three-function position coding and Gaussian position coding and two layers of linear layers and nonlinear activation functions; the row-column pooling Transformer module structure of the present invention is as follows Figure 5 As shown, the row-column pooling Transformer module includes a linear layer, a root mean square layer normalization, a row-column pooling attention module, a multi-layer perceptron and a jump connection. Specifically, the input data is normalized by the root mean square layer and embedded in the time step position encoding, and then the row-column pooling attention module performs multi-scale column pooling and row pooling on the input data, and performs attention operations on the row pooling data and column pooling data of different scales respectively, which not only provides accurate detail information and large-scale structural information, but also captures the directional information of the image; the row-column pooling attention module structure of the present invention is as shown Figure 6 As shown, it includes a convolutional layer, a time step position encoding, a row pooling attention module and a column pooling attention module; the structure diagram of the row pooling attention module of the present invention is shown in Figure 7 As shown in FIG. 1 , the row pooling attention module operates in an unusual way. It first performs row pooling operations on the input features to generate query, key, value and mapping matrices D of different scales for extracting large-scale structural information in the vertical direction. The attention operation of the present invention is performed and the time step position encoding is added. The column pooling and row pooling processes of the present invention are shown in FIG. Figure 8 As shown, scaling is performed vertically and horizontally respectively.

[0069] The cross-Mamba attention feature fusion module of the present invention is as follows Fig. 9 As shown in the figure, the module introduces a Laplacian pyramid module to perform Gaussian blur and downsampling of RGB features and depth features to different degrees, generate high-resolution and low-resolution feature maps, and retain local information and global information of the feature maps respectively. The multi-resolution Mamba module performs cross-Mamba operations on RGB features and depth features of the same resolution, realizes cross-enhancement of RGB features and depth features of the same resolution, and then merges the high-resolution and low-resolution feature maps of the same data. Finally, the feature weighted mask attention feature fusion module is used to further enhance the features through feature weighted masks and perform feature fusion through cross-attention, so as to achieve better semantic segmentation effect. The Laplacian pyramid module structure of the present invention is as follows Fig.10 As shown, the Laplacian pyramid method is used, which is a multi-resolution image processing and analysis technology. By decomposing the original image into different spatial frequency bands, the different scale details of the image are captured layer by layer. The multi-resolution cross Mamba module structure of the present invention is as follows Fig.11As shown, the cross-selective scanning Mamba module is used to perform local feature cross-enhancement on the high-resolution RGB features and depth features, and another independent cross-selective scanning Mamba module is used to perform global feature cross-enhancement on the low-resolution RGB features and depth features; the cross-selective scanning Mamba module of the present invention is as shown Fig.12 As shown in FIG. 1 , a vertical and horizontal continuous scanning method is adopted to generate two sequence information from one input feature. During the scanning process, when the end of each row (or column) is reached, the next mark to be scanned will select the mark closest to the current mark in space, instead of jumping to the farthest mark in the next row (or column). This continuous scanning method maintains spatial and semantic continuity and reduces the potential risk of information loss. The cross Mamba enhancement module structure of the present invention is as follows: Fig.13 As shown in FIG. 1 , the module realizes the information interaction of the two input sequences by exchanging the output matrices C1 and C2 of the two state space models and the direct transfer matrices D1 and D2. This design improves the expressiveness of the features, enhances the complementarity and fusion effect of the features, helps to capture more image details, and enables the model to better understand the semantic content of the remote sensing image. The structure of the feature weighted mask attention fusion module of the present invention is as follows: Fig.14 As shown in the figure, the feature weighted mask operation and the cross-attention feature fusion operation are combined. The feature weighted mask assigns different weights to each spatial position of the two input features through the Softmax function, thereby emphasizing important feature areas and suppressing unimportant areas.

[0070] The decoder module structure of the present invention is as follows Fig.15 As shown, it consists of depth-wise separable convolution, linear layer, upsampling, and non-linear activation function (sigmoid).

[0071] The present invention uses a training set to train the process of semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model as follows: Fig.16 As shown, it includes randomly sampling remote sensing semantic segmentation data, step size t, noise, forward noise addition, initialization of model parameters, calculation of MSE loss function, reverse update, and saving of network model parameters.

[0072] The flowchart of the remote sensing image semantic segmentation step of the present invention is as follows Fig.17 As shown, it includes putting the RGB data and depth data of the test set into the trained model and outputting the semantic segmentation results of the remote sensing image.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model, characterized in that: It includes preparing earthquake data sets, building, training, and verifying earthquake data reconstruction network models, and performing data reconstruction; building remote sensing image semantic segmentation network models includes designing an encoder-decoder hybrid architecture based on Transformer and Mamba. The encoder consists of an RGB information branch, a depth information branch, and a noisy semantic segmentation information branch in parallel. The features of the three branches are cross-enhanced and fused in turn, and then the semantic segmentation map of the remote sensing image is obtained through the decoder; Construct a row-column pooling Transformer module, use the row-column attention in the row-column pooling Transformer module to extract RGB information, depth information, and vertical and horizontal detail information and large-scale structure information containing noisy semantic segmentation information, and fuse the information in these directions; Construct a cross-Mamba attention feature fusion module. The RGB information branch and the depth information branch synchronously process the input feature map, and use the Laplacian pyramid to generate high-resolution and low-resolution feature maps. The multi-resolution cross-Mamba module is used to cross-enhance the RGB features and depth features of the same resolution, and merge feature maps of different resolutions. Use the feature weighted mask attention fusion module to perform feature fusion between the RGB information branch and the depth information branch; Construct a multi-resolution cross-Mamba module, and cross-enhance the RGB features and depth features of the same resolution by cross-selectively scanning the Mamba module; Construct a feature weighted mask attention fusion module and introduce feature weighted masks to enhance the effect of feature fusion by giving higher weights to important feature information; Construct a decoder module to achieve feature downsampling.

2. According to claim 1, a multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model is characterized in that: The embedding process of the time step includes Gaussian position encoding, trigonometric position encoding, square root Swish activation function and 2 linear layers: PE=linear(Swish(linear(PE))); In the formula is the time step position encoding, linear(.) represents the linear layer, Swish represents the Swish activation function, PE g is Gaussian position encoding, PE s is the trigonometric function position encoding, and λ is a custom variable parameter; Gaussian position encoding part: μ T,i =XT+Yi+Z:σ T,i =AT+Bi+C): Where i is the i-th element of the sequence, X, Y, Z, A, B, and C are all adaptive parameters, T is the specific position of the time step t in the sequence, the mean μ represents the center position of the Gaussian distribution, and the standard deviation σ represents the width or extension range of the Gaussian distribution. The mean μ and standard deviation σ are (1, t, d model ) matrix, where d model is the embedding dimension of the feature map, PE g is the Gaussian position encoding; Trigonometric function position encoding part: Where T is the specific position of the time step t in this sequence, 2i represents an even number, 2i+1 represents an odd number, and d model is the dimension of the data.

3. The multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model according to claim 2, characterized in that: The row-column pooling Transformer module includes a linear layer, RMS layer normalization, a row-column pooling attention module, a multi-layer perceptron, and a skip connection; For x, the time step position encoding is embedded through nonlinear activation function and RMS layer normalization, and column attention operations are performed by pooling rows and columns of features of different scales respectively, and shortcut connections are added in the row-column pooling Transformer module: A=rcAttention(RMS(SqrtSwish(x),linear(PE))); Y=Sum(A,MLP(RMS(A,linear(PE)))); Wherein, SqrtSwish(x) represents the Swish activation function of the present invention, RMS(.) represents the root mean square layer normalization, The time step position encoding is embedded in the input feature and normalized by the root mean square layer. rcAttention(.) represents the row-column pooling attention module, A is the output feature of the row-column pooling attention module, MLP(.) represents the multi-layer perceptron, and Sum(.) represents the dimension addition.

4. The multi-source remote sensing image semantic segmentation method based on Transformer, Mamba and diffusion model according to claim 3 is characterized in that: The row-column pooling Transformer module includes a linear layer, RMS layer normalization, a row-column pooling attention module, a multi-layer perceptron, and a skip connection; For x, the time step position encoding is embedded through nonlinear activation function and RMS layer normalization, and column attention operations are performed by pooling rows and columns of features of different scales respectively, and shortcut connections are added in the row-column pooling Transformer module: A=rcAttention(RMS(SqrtSwish(x),linear(PE))); Y=Sum(A,MLP(RMS(A,linear(PE)))); Where SqrtSwish(x) represents the Swish activation function of the present invention, RMS(.) represents the root mean square layer normalization, The time step position encoding is embedded in the input feature and normalized by the root mean square layer. rcAttention(.) represents the row-column pooling attention module, A is the output feature of the row-column pooling attention module, MLP(.) represents the multi-layer perceptron, and Sum(.) represents the dimension addition.

5. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 4, characterized in that: Root mean square layer normalization (RMS Norm) replaces the layer normalization layer of the traditional Transformer module. The learnable parameters include not only the scaling factor σ and offset β obtained by input features, but also the scaling factor ω and offset ξ obtained by regressing the embedding vector of time step t; these embedding vectors are processed by a nonlinear activation function (Swish activation function), which weights the input value and outputs the weighted result: Where x i represents a vector representation of the input feature, R(x) represents the root mean square of the input feature, represents the root mean square of a vector without scaling and offset, rms(x i , t) i represents the root mean square after scaling and offset, ε is a small constant for numerical stability, δ and β are normalization parameters adaptively adjusted according to the statistical characteristics of the input features, ω and ξ are the scaling factor and offset obtained by linear regression from the time step position encoding.

6. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 5, characterized in that: In the row-column pooling attention module, the attention mechanism is divided into two parts: row pooling attention and column pooling attention. The input features are input into the row pooling attention module and the column pooling attention module respectively, and the time step position encoding is embedded. The row pooling attention module and the column pooling attention module generate scales of and The output feature matrix of is multiplied by the matrix, so as to fuse the output features in the vertical and horizontal directions, so that the model can obtain the direction information and generate the scale The output features of , while downsampling is achieved: RA=RAttention(linear(PE),Conv 1*1 (x));CA=CAttention(linear(PE),Conv 1*1 (x)); A = CA * RA; Where x represents the input feature of the row-column pooling attention module, Conv 1*1 (.) represents 1*1 convolution, It represents the linear change of the time step position encoding, and the time information is added to the row pooling attention module and the column pooling attention module respectively; RAttention(.) and CAttention represent the row pooling attention module and the column pooling attention module respectively, RA and CA are the output features of the two modules, and A is the overall output feature of the row and column pooling attention module; the row pooling attention module and the column pooling attention module are symmetrical modules.

7. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 6, characterized in that: The row pooling attention module performs row pooling operations of different scales on the input features of (N, N) to generate three inputs of different scales x1, x2, and x3, whose scales are Use x1, x2, x3 and the time step position encoding to generate the query Q, key K, and value V of the attention mechanism, respectively, with the scale sizes being Then the input x is passed through the linear layer to generate the mapping matrix D, where the number of rows and columns of D are both N. Then the query Q and the mapping matrix D are matrix multiplied to obtain a scale of The result is that the key K and the mapping matrix D are matrix multiplied to obtain a scale of The result is adjusted to the scale by linear interpolation. Finally, they are multiplied by the transpose of the value V and the dimensions are added to generate the scale The output matrix is: x1=MaxPool 2*1 (x);x2=MaxPool 4*1 (x);x3=MaxPool 8*1 (x); Q=W Q *(x1+linear(PE(t)));K=W K *(x2+linear(PE(t)));V=W v *(x3+linear(PE(t))); D=linear(x); Where x represents the input feature of the row-column pooling attention module, MaxPool(.) 2*1 、MaxPool(.) 4*1 and MaxPool(.) 8*1 represents the row maximum pooling of 2*1, 4*1, and 8*1 respectively, x1, x2, and x3 are the corresponding pooling results, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and W Q , W K , W V Represent the weight matrices of Q, K, and V respectively, Indicates that the time step position encoding is scaled through the linear layer; the mapping matrix D is generated by the input x through the linear layer; RuLU(.) is the activation function, represents the transpose of the value matrix V and LI represents linear interpolation.

8. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 7, characterized in that: The Cross-Mamba Attention Feature Fusion Module includes RMS layer normalization, depth-wise separable convolution, Laplacian pyramid module, multi-resolution Cross-Mamba, and feature weighted mask attention fusion module; the input features are RGB features and depth features. This module accepts two features as input and finally generates an output feature: L RH ,L RL =Laplacian(Conv(RMS(x R ,PE)); L DH ,L DL =Laplacian(Conv(RMS(x D ,PE)); C R ,C D =MCMamba(L RH ,L RL ,L DH ,L DL ); Z R ,Z D =Sum(Conv(RMS(C R ,C D )),RMS(x R ,x D )); In=MaskAttention(From R ,WITH D ); In the formula Respectively, they represent the RMS layer normalization of RGB features and depth features and embedding them into the time step position encoding, Conv(.) represents the depthwise separable convolution, Laplacian(.) represents the Laplacian pyramid module, and L RH , L RL , L DH , L DL They are high-resolution features of RGB features, low-resolution features of RGB features, high-resolution features of depth information, and low-resolution features of depth information. MCMamba(.) represents the multi-resolution cross-Mamba module. R , C D Represents the output features of the multi-resolution cross-Mamba module, and Sum(.) represents dimensional addition. The multi-resolution cross-Mamba module is a four-input dual-output system. Through cross-Mamba, the module realizes cross-enhancement of dual-channel multi-resolution data. MaskAttention(.) represents the feature weighted mask attention fusion module, W is the output of the module, and the feature weighted mask attention fusion module performs deep feature fusion on the cross-enhanced RGB features and depth features.

9. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 8, characterized in that: The Laplacian pyramid module generates 6 feature maps of different resolutions through Gaussian blur and downsampling, and then processes them into high-resolution and low-resolution feature maps through bilinear transformation and channel splicing: G0=x; L0=x; G i =Reduce(G i-1 ),(i>0);L i =G i -Expand(G i+1 ),(i>0); HC=Channel(L0,BI(L1),BI(L2)); LC=Channel(L3,BI(L4),BI(L5)); HX=Channel(HC,BI(LC));LX=Channel(MaxPool 8*8 (HC),LC); Where G i represents the i-th layer of the Gaussian pyramid, x is the input feature, Reduce(.) represents the Gaussian blur and downsampling operation; L i represents the i-th layer of the Laplacian pyramid, and Expand(.) represents upsampling, which converts the image from G i+1 Size expansion to G i The size of Channel(.) indicates channel concatenation, BI(.) indicates bilinear interpolation, HC and LC are the concatenation results of the input feature maps with higher and lower resolutions, respectively. Channel(HC, BI(LC)) and Channel(MaxPool 8*8 (HC), LC) combine the high-resolution and low-resolution feature maps, HX and LX represent the high-resolution feature map and low-resolution feature map finally output by the 1-pyramid module.

10. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 9, characterized in that: The multi-resolution cross-Mamba module includes two cross-selective scanning Mamba modules, two bilinear interpolation and two channel splicing; this module performs feature cross-enhancement on high-resolution RGB features and depth features through the cross-selective scanning Mamba module, and uses another independent cross-selective scanning Mamba module to perform feature cross-enhancement on low-resolution RGB features and depth features; Subsequently, the high-resolution and low-resolution RGB features are bilinearly interpolated and channel concatenated, and the depth features are processed in the same way to finally generate RGB feature output and depth feature output: HY RGB ,HY DSM =CSMamba(HX RGB ,HX DSM );LY RGB ,LY DSM =CSMamba(LX RGB ,LX DSM ); THE RGB =Channel(HY RGB , BI(LY RGB ));THE DSM =Channel(HY DSM , BI(LY DSM )): Where HX RGB and HX DSM They are high-resolution RGB feature input and deep feature input, CSMamba(.) represents the cross selective scanning Mamba module, HX RGB and HX DSM They correspond to high-resolution RGB and depth feature inputs, HY RGB and HY DSM They correspond to high-resolution RGB and depth feature outputs, respectively. RGB and LX DSM They are low-resolution RGB feature input and deep feature input, LY RGB and LY DSM They are low-resolution RGB feature input and deep feature output, Channel(.) represents channel concatenation, BI(.) represents bilinear interpolation; Y RGB and Y DSM They are the fusion features of high and low resolution RGB features and depth features.

11. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 10, characterized in that: The cross-selective scanning Mamba module is different from the discontinuous four-way scanning method adopted by the classic two-dimensional selective scanning module. The present invention performs continuous scanning in the horizontal and vertical directions respectively. When scanning to the end of each row (or column), the next mark to be scanned is the mark adjacent to it in the next row (or column); After continuous scanning of RGB features and depth features, four sequences are obtained. Two sequences with the same scanning direction are input into the cross-Mamba enhancement module for independent processing. Then, the results are reversely scanned and merged as the final output features, and the input features X1 and X2 are cross-enhanced. Finally, the four sequences of the two inputs are reversely reconstructed and the corresponding dimensions are added: CH1, CV1 = HS (z1); CH2, CV2 = HS (z2); H1,H2=CSSM(CI1,CI2); V1,V2=CSSM(CO1,CO2); Y1=Sum(Re(H1,V1)); Y2=Sum(Re(H2,V2)); Wherein HS(z1, z2) represents continuous scanning in two directions, CH1 and CV1 are the output sequences of horizontal and vertical scanning of input feature z1 respectively, CH2 and CV2 are the output sequences of horizontal and vertical scanning of input feature z2 respectively; CSSM(.) represents the cross-Mamba enhancement module, H1 and H2 are the output results generated by the cross-Mamba enhancement module of the horizontal output sequences of z1 and z2 respectively, V1 and O2 are the output feature sequences generated by the cross-Mamba enhancement module of the vertical output sequences of z1 and z2 respectively, Re(.) represents the sequence after rearrangement processing, which is the inverse transform of continuous scanning HS(.), Sum(Re(.)) is to rearrange the processed sequence into a two-dimensional feature map and add the dimensions of the two paths; Y1 and Y2 are the output features of the two input features after the cross-selective scanning Mamba module.

12. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 11, characterized in that: The cross-Mamba enhancement module can realize the information interaction and cross-enhancement of two inputs; taking the two sequences CH1 and CH2 obtained by horizontal scanning of two input features as examples, CH1 and CH2 are respectively denoted as x1 and x2, B1, C1, D1, Δ1 are functions of sequence x1, B2, C2, D2, Δ2 are functions of sequence x2, parameters A1 and A2 are evolution parameters, which are constructed by HiPPO continuous time memory theory. It stores all previous historical information represented by the coefficient matrix, and parameters Δ1 and Δ2 are sampling time scale parameters. Δ1 is used to convert continuous parameters A1 and B1 into discrete parameters, and Δ2 is used to convert continuous parameters A2 and B2 into discrete parameters, and determines the influence of the previous hidden state on the hidden state used to update the hidden state space state at the next moment, and then performs state space equation operations: Where x 1，t 、x 2，t , are the two input features of the current state, h 1，t 、h 2，t is the current state, h 1，t-1 、h 2，t-1 It is the previous state; is a structured convolution kernel, are the discrete evolution parameters in the state space model of RGB features, are the discretized evolution parameters in the state space model of the deep features, y1 and y2 represent the output features of the two state space models respectively. The cross-Mamba enhancement module realizes the information interaction and cross-enhancement of the two input sequences by exchanging the output matrices C1 and C2 of the two state space models and the direct transfer matrices D1 and D2.

13. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 12, characterized in that: The feature weighted mask attention fusion module in the present invention performs weighted processing on RGB features and depth features respectively to highlight important feature areas, and then performs cross-attention fusion features; The feature weighted mask attention fusion module takes two features as input and finally produces an output feature: A mask ,B mask =Softmax(DSConv(A,B));A out ,B out =Reshape(A,B)*Reshape(A mask ,B mask ); M=Reshape(Sum(A out ,B out ));Q=Wight(Reshape(A));K,V=Wight(M); Y = Attention(Q,K,V); Where DSConv(.) represents the depthwise separable convolution, Softmax(.) is the Softmax activation function, which converts the input vector into a probability distribution. Each output value ranges between [0,1], and the sum of all output values ​​is 1; A mask , B mask is a weight mask, which is used to give higher weights to important features; Reshape(.) represents a reshaping operation. Assuming that the shape of input A and B is C*H*W, the predicted A mask and B mask Reshape to K*(H*W), reshape A and B to (H*W)*C, and then perform matrix multiplication to extract K global context features, where K is a custom parameter; A out , B out The output result of the weighted mask operation, the shape is K*C, Sum(.) means dimension addition, M is A out , B out The reshaped output features after dimension addition have a shape of C*2K, where K, V=Wight(M) means that the key K and value V of the attention mechanism are obtained based on M, Q=Wight(Reshape(A) means that the query Q of the attention mechanism is obtained based on the input A, and Y=Attention(Q, K, V) means that the key K, value V and query Q are input into the cross attention mechanism.

14. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 13, characterized in that: The decoder module consists of a convolutional layer, a linear layer, upsampling, and a nonlinear activation function (sigmoid): Y=sigmoid(upsamping(linear(DSConv(x)))); Where DSConv(.) is the depthwise separable convolution, linear(.) is the linear layer, upsampling(.) is the upsampling, sigmoid(.) is the activation function, Y is the output feature of the decoder module, and the upsampling module uses the bilinear interpolation method.

15. The method for semantic segmentation of multi-source remote sensing images based on Transformer, Mamba and diffusion model according to claim 14, characterized in that: After completing the construction of the remote sensing image semantic segmentation network model, training is performed. A remote sensing image semantic segmentation label is sampled from the training data, and then a time step t is randomly selected. A noise sample ε is randomly extracted from the Gaussian distribution. This noise sample represents the noise that may appear at a specific time step and is the interference that the model needs to learn to remove. The randomly extracted noise is added to the remote sensing image semantic segmentation label to obtain noisy input data, providing the model with training samples that are closer to the actual situation. Input the noise data ε into the remote sensing image semantic segmentation network model; The model predicts the noise estimate ∈ by learning the noise characteristics in the training data t , this estimation reflects the model's understanding and prediction of the noise in the input data. The segmented image output by the diffusion model is compared with the actual segmented image, and the error between the two is calculated. If the error is greater than the set threshold, the parameters are updated by back propagation. It is repeated until the error is less than the set threshold. The parameter update stops and the network parameters are saved. The Adam algorithm is used to solve the parameters. The parameters are updated according to the gradient information during the error back propagation. The RGB data and depth data in the training set are input into the trained remote sensing image semantic segmentation network model, and the semantic segmentation data output by the model is compared with the accurate semantic segmentation label, and the error between the two is calculated. If the error is greater than the set threshold, the training of the remote sensing image semantic segmentation network model is returned. If the error is less than the set threshold, the verification of the seismic data reconstruction network model is completed. The RGB data and depth data in the test set are input into the trained and verified remote sensing image semantic segmentation network model to obtain the semantic segmentation results of the remote sensing image.

Images