A remote sensing image semantic segmentation method and device based on Kan-Mamba model

Abstract

The present invention discloses a multi-source remote sensing image semantic segmentation method and device based on KAN-Mamba, which is used to improve the accuracy of remote sensing image semantic segmentation, including a KAN-Mamba feature extraction network and an Uper_Head decoder; the KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module, and the KAN-Mamba feature extraction network performs staged feature extraction on the fused remote sensing image to obtain multi-scale features; the Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer, and the side feature construction unit and the pooling unit perform side feature construction and pooling operations on the multi-scale features, then perform step-by-step feature fusion by the feature fusion unit, then perform channel fusion by the BottleNeck layer, and finally obtain semantic segmentation labels by the classification layer. The present invention has higher semantic segmentation accuracy for multi-source remote sensing images, faster model convergence, and more accurate detail information.

Description

Technical Field

[0001] The present invention relates to computer vision and image processing technology, and in particular to a multi-source remote sensing image semantic segmentation method and device based on a Kan-Mamba model, belonging to remote sensing image segmentation technology. Background Art

[0002] Remote sensing technology, which acquires spectral information about the Earth's surface, can be used to monitor numerous environmental issues, such as changes in vegetation cover, water pollution, and land use change. For example, multispectral or hyperspectral remote sensing data can distinguish different vegetation types, assess vegetation health, and effectively monitor ecosystems. Remote sensing technology is also extremely useful in monitoring natural disasters such as floods, droughts, and forest fires, providing critical information for disaster warning and emergency response. Furthermore, with the rapid development of remote sensing and drone technology, the ability to acquire high-resolution imagery continues to increase. However, extracting useful information from massive amounts of image data, particularly pixel-level semantic segmentation, has become a pressing challenge.

[0003] Currently, the use of optical remote sensing images for detection and analysis is a common method. However, this is only a single remote sensing data source and cannot meet actual needs. When using only optical remote sensing images for analysis and segmentation, there are many obvious drawbacks: optical images are easily affected by weather conditions, such as clouds, rain, and haze, which can seriously reduce image quality and cause information loss in some areas; optical remote sensing mainly obtains information in the visible light and near-infrared bands, lacking observations in other bands (such as thermal infrared and microwaves), which limits the effective analysis of certain specific landforms (such as water bodies and wetlands); shadows cast by buildings and trees may cause ground features to be misclassified. In addition, the reflections of certain landforms (such as water surfaces and smooth ground) at different angles may interfere with observation results and cause misjudgment; in complex environments such as cities or forests, buildings or trees may obscure ground features, making some information invisible and affecting the accurate identification of landforms.

[0004] To overcome these challenges, using multiple remote sensing images to provide complementary features is a practical solution. In multi-source segmentation, data fusion techniques can be used to integrate data from different sources. For example, the color features of optical images can be combined with the texture features of SAR images, because SAR can acquire images in various weather conditions (including clouds, rain, and at night) and is not affected by illumination. At the same time, SAR can provide high-spatial-resolution images, which are conducive to detail analysis and feature extraction. This can more comprehensively display the characteristics of the ground objects, thereby improving classification accuracy. However, given the heterogeneity of multi-source data, directly integrating it into existing remote sensing detection methods is difficult or it is difficult to achieve the expected performance.

[0005] Mamba is an innovative approach in the field of computer vision. On the one hand, it addresses the challenges faced by convolutional neural networks (CNNs) and visual transformers (ViTs). CNNs excel at extracting local features, but struggle to capture long-range dependencies without modifying their complex architecture. ViTs effectively model global relationships, but suffer from high computational costs due to the quadratic complexity of their self-attention mechanism. On the other hand, Mamba leverages a selectively structured state-space model to effectively capture long-range dependencies with linear computational complexity. This enables it to excel in overcoming the limitations of existing methods and opens new possibilities for the development of remote sensing image segmentation.

[0006] The Kolmogorov-Arnold representation theorem states that if f is a continuous function of multiple variables over a bounded domain, then f can be written as a finite combination of continuous functions of a single variable and a binary addition operation. The MLP is inspired by the universal approximation theorem, while the Kolmogorov-Arnold representation theorem is inspired by the Kolmogorov-Arnold representation theorem. Like the MLP, the KAN has a fully connected structure. However, while the MLP uses fixed activation functions at the nodes ("neurons"), the KAN uses learnable activation functions at the edges ("weights"). Therefore, the KAN does not have a linear weight matrix at all: each weight parameter is replaced by a learnable one-dimensional function parameterized as a spline. KAN nodes simply sum the input signals, without applying any nonlinear factors. As a result, the computational graph of a KAN is typically much smaller than that of an MLP.

[0007] Existing remote sensing image segmentation methods still face numerous difficulties: 1. Multi-source data may contain redundant or irrelevant information, making it difficult to select appropriate features and effectively extract them, making it difficult to avoid information noise; 2. To effectively process multi-source data, the model may require a more complex architecture, which increases the difficulty of training and optimization and easily leads to overfitting; 3. Different types of remote sensing data may have semantic differences. For example, SAR images and optical images present the same features differently, which may cause the model to make errors in classification. To address these challenges, an effective solution is to find a method that is better able to process high-dimensional data from satellite and aerial imagery, making it suitable for the field of remote sensing image segmentation. Summary of the Invention

[0008] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention proposes a multi-source remote sensing image semantic segmentation method and device based on the Kan-Mamba model, which comprehensively utilizes multi-source remote sensing image information to improve segmentation accuracy and interpretability.

[0009] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0010] A multi-source remote sensing image semantic segmentation method based on KAN-Mamba includes the following steps:

[0011] Step 1: Obtain multi-source remote sensing images, including optical images, synthetic aperture radar (SAR) images, and semantic segmentation labels (MASK) for training. Perform channel fusion, cropping, and normalization on the remote sensing images. The resulting fused remote sensing image is 5 × H × W in size to ensure data accuracy and consistency.

[0012] Step 2: Divide the processed fused remote sensing image into a training set and a test set according to the ratio, where the training set includes the fused remote sensing image O train and semantic label M train , the test set includes fused remote sensing images O test ;

[0013] Step 3: Construct a KAN-Mamba feature extraction network. The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth-stage KAN-Mamba feature extraction modules also include a downsampling module.

[0014] O train The data is input to the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained. Feature10 is used as the input of the first-stage KAN-Mamba feature extraction module.

[0015] In the first stage KAN-Mamba feature extraction module, feature10 is processed by all the VSS-KAN feature extraction modules in series in this stage to obtain the feature sequence Out O_0 ;

[0016] In the second stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_1 ;

[0017] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_1Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ;

[0018] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ;

[0019] Step 4: Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder for feature fusion;

[0020] For Out O_0 、Out O_1 、Out O_2 , first construct the side features separately, and then perform 1×1 convolution to obtain For Out O_3 , first perform multi-scale pooling or average pooling processing, and then obtain the output feature Out by convolution dimension reduction U_3 ;To Out U_3 After upsampling, Perform addition operation to obtain output feature Out U_2 ;To Out U_2 After upsampling, Perform addition operation to obtain output feature Out U_1 ;right After upsampling, Perform addition operation to obtain output feature Out U_0 ;To Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then pass through the BottleNeck layer and the classification layer to obtain the fusion feature Out train ;

[0021] Step 5: Compare Out train and M train , using the cross entropy loss function loss train Calculate Out train and M train the differences between;

[0022] Step 6: Through loss train Perform backpropagation to optimize the KAN-Mamba feature extraction network and Uper_Head decoder;

[0023] Step 7: Repeat Step 4 to Step 6 until the loss train Convergence, completing the training of the KAN-Mamba feature extraction network;

[0024] Step 8: First, test Input the trained KAN-Mamba feature extraction network and Uper_Head decoder, and output multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder, output fusion feature Out train , Out train As the prediction result of semantic label.

[0025] Specifically, in the KAN-Mamba feature extraction network, the number of channels of the one-stage KAN-Mamba feature extraction module, the two-stage KAN-Mamba feature extraction module, the three-stage KAN-Mamba feature extraction module, and the four-stage KAN-Mamba feature extraction module are 96, 192, 384, and 768, respectively.

[0026] Specifically, in the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the normalized input image into four scan sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector and inputs it into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer.

[0027] Specifically, the Mamba discretized state space equation is expressed as:

[0028] h′(t)=Ah(t)+Bu(t)

[0029] y(t)=Ch(t)+Du(t)

[0030] in: is the hidden input at time t, h′(t) is the derivative of h(t), is the input stimulus at time t, is the output response at time t; is the state space equation weight matrix, is the weighting parameter; N is the dimension of the hidden input, and L is the length of the input stimulus.

[0031] Specifically, the Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers, and an output layer. The linear layer performs linear mapping on the input image, and the Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x):

[0032]

[0033] Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, ..., a m ,b0,b1,…,b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

[0034] Specifically, the Group-Rational KAN layer divides the number of channels of the input layer into more than one group. Within each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) are different for different groups.

[0035] Specifically, Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are Where H and W are the height and width of the fused remote sensing image obtained after Step 1 processing; Out U_0 、Out U_1、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the output is Out through the 3×3 convolution and classification layer train ,Out train The size is K represents the total number of categories for image segmentation.

[0036] Specifically, the classification layer includes a Dropout layer and a 1×1 convolution layer. The dropout ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories for image segmentation, that is, the total number of label categories.

[0037] Specifically, the cross entropy loss function loss train Expressed as:

[0038]

[0039] Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

[0040] A multi-source remote sensing image semantic segmentation device based on KAN-Mamba, used to implement the above method, including a multi-source remote sensing image acquisition unit, a KAN-Mamba feature extraction network, an Uper_Head decoder, a training unit and a testing unit;

[0041] The multi-source remote sensing image acquisition unit is used to acquire fused remote sensing images and semantic labels of the same size and geometric alignment;

[0042] The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The KAN-Mamba feature extraction network performs staged feature extraction on the fused remote sensing image to obtain multi-scale features.

[0043] The Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer. The side feature construction unit and the pooling unit perform side feature construction and pooling operations on the multi-scale features, and then perform step-by-step feature fusion through the feature fusion unit. Then, the BottleNeck layer performs channel fusion, and finally obtains the fusion feature through the classification layer. The BottleNeck layer performs a 3×3 convolution operation on the input feature, converting the number of channels of the input feature from 4C to C, where C is the number of output channels set by the Uper_Head decoder. The classification layer performs a Dropout operation and a 1×1 convolution operation on the input feature, and outputs the prediction result Out of the semantic label. train ;

[0044] The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and the Uper_Head decoder;

[0045] The test unit uses the trained KAN-Mamba feature extraction network and Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

[0046] Specifically, in the KAN-Mamba feature extraction network:

[0047] For the input fused remote sensing image O train , after four-fold downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained, and feature10 is used as the input of the first-stage KAN-Mamba feature extraction module;

[0048] In the first stage KAN-Mamba feature extraction module, feature10 is processed by all the VSS-KAN feature extraction modules in series in this stage to obtain the feature sequence Out O_0 ;

[0049] In the second stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_1 ;

[0050] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ;

[0051] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ;

[0052] Out O_0 、Out O_1 、Out O_2 、Out O_3 Together they are called multi-scale features.

[0053] Specifically, in the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the normalized input image into four scan sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector and inputs it into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer; the Group-Rational The KAN layer consists of an input layer, two linear layers, two Group-KAN layers, and an output layer.

[0054] Beneficial effect: The multi-source remote sensing image semantic segmentation method and device based on the Kan-Mamba model provided by the present invention establish a segmentation model by introducing a feature extraction framework based on Kan-Mamba and an Uper_Head decoder, and perform semantic segmentation on multi-source remote sensing images, so that the semantic segmentation accuracy of multi-source remote sensing images is higher, the model converges faster, and the detail information is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the framework structure of the device of the present invention;

[0056] Figure 2 Flow chart for the implementation of the method of the present invention;

[0057] Figure 3For pairs of original optical images and original synthetic aperture radar images, fused remote sensing images and semantic labels, the prediction results of the semantic labels of the trained KAN-Mamba feature extraction network and Uper_Head decoder. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 The figure shows a multi-source remote sensing image semantic segmentation device based on KAN-Mamba, which includes a multi-source remote sensing image acquisition unit, a KAN-Mamba feature extraction network, an Uper_Head decoder, a training unit and a testing unit. The multi-source remote sensing image acquisition unit is used to obtain fused remote sensing images and semantic labels of the same size and geometric alignment; the KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module, and the KAN-Mamba feature extraction network performs staged feature extraction on the fused remote sensing image to obtain multi-scale features; the Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer, and performs side feature construction and pooling operations on the multi-scale features through the side feature construction unit and the pooling unit, and then performs step-by-step feature fusion through the feature fusion unit, and then performs channel fusion through the BottleNeck layer, and finally obtains the fused features through the classification layer; the BottleNeck layer performs a 3×3 convolution operation on the input features, converts the number of channels of the input features from 4C to C, and C is the number of output channels set by the Uper_Head decoder; the classification layer performs a Dropout operation and a 1×1 convolution operation on the input features, and outputs the prediction result Out of the semantic label. train ; The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and Uper_Head decoder; the testing unit uses the trained KAN-Mamba feature extraction network and Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

[0060] like Figure 2 As shown, for use Figure 1 The process of performing multi-source remote sensing image semantic segmentation by the multi-source remote sensing image semantic segmentation device based on KAN-Mamba is described in detail below in conjunction with each step.

[0061] Step 1: Acquire multi-source remote sensing images

[0062] Acquire multi-source remote sensing images, including optical images (Optical), synthetic aperture radar images (SAR) and semantic segmentation labels (MASK) for training, and perform channel fusion, cropping and standardization on the remote sensing images. The size of the fused remote sensing image obtained after processing is 5×H×W to ensure the accuracy and consistency of the data for easy subsequent processing; Figure 3 As shown in Figure 2, the fused remote sensing image and semantic labels are obtained after processing.

[0063] Step 2: Divide the training set and test set

[0064] The processed fused remote sensing images are divided into training set and test set according to the ratio of 7:3, where the training set includes the fused remote sensing images O train and semantic label M train , the test set includes fused remote sensing images O test .

[0065] Step 3: Build the KAN-Mamba feature extraction network

[0066] The KAN-Mamba feature extraction network can also be called the KAN-Mamba backbone network, which includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module.

[0067] 3.1 Architecture of the KAN-Mamba Feature Extraction Network

[0068] The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth-stage KAN-Mamba feature extraction modules also include a downsampling module. The number of channels of the first-stage KAN-Mamba feature extraction module, the second-stage KAN-Mamba feature extraction module, the third-stage KAN-Mamba feature extraction module, and the fourth-stage KAN-Mamba feature extraction module are 96, 192, 384, and 768, respectively.

[0069] In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer.

[0070] The normalization layer performs a layer normalization operation on the input image.

[0071] The SS2D module includes a cross-scan module and an S6 module. The cross-scan module first expands the normalized input image into four scan sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left. Scan1, Scan2, Scan3, and Scan4 are then spliced ​​into a four-channel vector and input into the S6 module. The Mamba discretized state space equation in the S6 module is used for feature extraction to obtain output features. The Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer. The Mamba discretized state space equation is expressed as:

[0072] h′(t)=Ah(t)+Bu(t)

[0073] y(t)=Ch(t)+Du(t)

[0074] in: is the hidden input at time t, h′(t) is the derivative of h(t), is the input stimulus at time t, is the output response at time t; is the state space equation weight matrix, is the weighting parameter; N is the dimension of the hidden input, and L is the length of the input stimulus.

[0075] The Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers, and an output layer. The linear layer performs linear mapping on the input image. The Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x):

[0076]

[0077] Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, ..., a m ,b0,b1,…,b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

[0078] The Group-Rational KAN layer divides the number of channels of the input layer into more than one group. In each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) are different for different groups.

[0079] 3.1 Execution Process of KAN-Mamba Feature Extraction Network

[0080] O train The image is input to the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained. Feature10 is used as the input of the first-stage KAN-Mamba feature extraction module.

[0081] In the first stage KAN-Mamba feature extraction module, feature10 is processed by all the VSS-KAN feature extraction modules in series in this stage to obtain the feature sequence Out O_0 .

[0082] In the second stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_1 .

[0083] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 .

[0084] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 .

[0085] Multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are

[0086] Step 4: Build the Uper_Head decoder

[0087] Construct the Uper_Head decoder for multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 Perform feature fusion; the Uper_Head decoder includes a side feature construction unit, a PPM multi-scale pooling unit, a feature fusion unit and a classification layer; the classification layer includes a Dropout layer and a 1×1 convolution layer, the discard ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories of image segmentation, that is, the total number of categories of labels.

[0088] For Out O_0 、Out O_1 、Out O_2 First, use the side feature construction unit to construct the side features separately, and then perform 1×1 convolution to obtain For Out O_3 , first use the PPM multi-scale pooling unit to perform multi-scale pooling or average pooling processing, and then use convolution to reduce the dimension to obtain the output feature Out U_3 ;To Out U_3 After upsampling, Perform addition operation to obtain output feature Out U_2 ;To Out U_2 After upsampling, Perform addition operation to obtain output feature Out U_1 ;right After upsampling, Perform addition operation to obtain output feature Out U_0 ;To Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then perform a 3×3 convolution operation through the BottleNeck layer to convert Out concat The number of channels is converted from 4C to C; finally, the Dropout operation and 1×1 convolution operation are performed through the classification layer to output the fusion feature Out train , that is, the prediction result of the semantic label Out train .

[0089] Out U_0 、Out U_1 、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the output is Out through the 3×3 convolution and classification layer train ,Out train The size is K represents the total number of categories for image segmentation.

[0090] Step 5: Use loss function for supervised training

[0091] The training unit uses the loss function to supervise the training of the KAN-Mamba feature extraction network and the Uper_Head decoder. In the process of training the KAN-Mamba feature extraction network and the Uper_Head decoder, the Out train Use bilinear interpolation to resize to the same size as M train Consistent, then use the cross entropy loss function loss to calculate Out train and M train The differences between:

[0092]

[0093] Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

[0094] If the cross entropy loss converges, the training of the KAN-Mamba feature extraction network and Uper_Head decoder is completed; otherwise, backpropagation is performed, the parameters are updated, and training is performed again until the loss converges.

[0095] Step 6: Test using the trained KAN-Mamba feature extraction network and Uper_Head decoder

[0096] like Figure 2 As shown, O test Input to the trained KAN-Mamba feature extraction network and output multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder to get the predicted semantic label Out train.from Figure 3 Comparing the prediction results and labels in the figure shows that the proposed method excels in detail and boundaries, accurately distinguishing different types of land cover, and can handle a wide range of land cover types, such as forest, grassland, buildings, and water bodies. Each type of land can be accurately segmented and classified, without poor performance due to the rarity of certain categories.

[0097] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the scope of protection of the present invention.

Claims

1. A multi-source remote sensing image semantic segmentation method based on KAN-Mamba, characterized by: The steps include: Step 1: Obtain multi-source remote sensing images, including optical images, synthetic aperture radar images, and semantic segmentation labels for training. Perform channel fusion, cropping, and normalization on the remote sensing images. The resulting fused remote sensing image has a size of 5 × H × W. Step 2: Divide the processed fused remote sensing image into a training set and a test set according to the ratio, where the training set includes the fused remote sensing image O train and semantic label M train , the test set includes fused remote sensing images O test ; Step 3: Construct a KAN-Mamba feature extraction network. The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth-stage KAN-Mamba feature extraction modules also include a downsampling module. O train The data is input to the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained. Feature10 is used as the input of the first-stage KAN-Mamba feature extraction module. In the first stage KAN-Mamba feature extraction module, feature10 is processed by all the VSS-KAN feature extraction modules in series in this stage to obtain the feature sequence Out O_0 ; In the second stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_1 ; In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ; In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ; Step 4: Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder for feature fusion; For Out O_0 、Out O_1 、Out O_2 , first construct the side features separately, and then perform 1×1 convolution to obtain For Out O_3 , first perform multi-scale pooling or average pooling processing, and then obtain the output feature Out by convolution dimension reduction U_3 ;To Out U_3 After upsampling, Perform addition operation to obtain output feature Out U_2 ;To Out U_2 After upsampling, Perform addition operation to obtain output feature Out U_1 ;right After upsampling, Perform addition operation to obtain output feature Out U_0 ;To Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then pass through the BottleNeck layer and the classification layer to obtain the fusion feature Out train ; Step 5: Compare Out train and M train , using the cross entropy loss function loss train Calculate Out train and M train the differences between; Step 6: Through loss train Perform backpropagation to optimize the KAN-Mamba feature extraction network and Uper_Head decoder; Step 7: Repeat Step 4 to Step 6 until the loss train Convergence, completing the training of the KAN-Mamba feature extraction network; Step 8: First, test Input the trained KAN-Mamba feature extraction network and Uper_Head decoder, and output multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder, output fusion feature Out train , Out train As the prediction result of semantic label.

2. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the normalized input image into four scan sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector and inputs it into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer.

3. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 2, characterized in that: The Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers, and an output layer. The linear layer performs linear mapping on the input image. The Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x): Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, ..., a m ,b0,b1,…,b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

4. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 3, characterized in that: The Group-Rational KAN layer divides the number of channels of the input layer into more than one group. In each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) are different for different groups.

5. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are Where H and W are the height and width of the fused remote sensing image obtained after Step 1 processing; Out U_0 、Out U_1 、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the output is Out through the 3×3 convolution and classification layer train ,Out train The size is K represents the total number of categories for image segmentation.

6. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: The classification layer includes a Dropout layer and a 1×1 convolution layer. The dropout ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories for image segmentation, that is, the total number of categories for labels.

7. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: The cross entropy loss function loss train Expressed as: Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

8. A multi-source remote sensing image semantic segmentation device based on KAN-Mamba, characterized by: It includes multi-source remote sensing image acquisition unit, KAN-Mamba feature extraction network, Uper_Head decoder, training unit and testing unit; The multi-source remote sensing image acquisition unit is used to acquire fused remote sensing images and semantic labels of the same size and geometric alignment; The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The KAN-Mamba feature extraction network performs staged feature extraction on the fused remote sensing image to obtain multi-scale features. The Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer. The side feature construction unit and the pooling unit perform side feature construction and pooling operations on the multi-scale features, and then perform step-by-step feature fusion through the feature fusion unit. Then, the BottleNeck layer performs channel fusion, and finally obtains the fusion feature through the classification layer. The BottleNeck layer performs a 3×3 convolution operation on the input feature, converting the number of channels of the input feature from 4C to C, where C is the number of output channels set by the Uper_Head decoder. The classification layer performs a Dropout operation and a 1×1 convolution operation on the input feature, and outputs the prediction result Out of the semantic label. train ; The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and the Uper_Head decoder; The test unit uses the trained KAN-Mamba feature extraction network and Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

9. The multi-source remote sensing image semantic segmentation device based on KAN-Mamba according to claim 8, characterized in that: In the KAN-Mamba feature extraction network: For the input fused remote sensing image O train , after four-fold downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained, and feature10 is used as the input of the first-stage KAN-Mamba feature extraction module; In the first stage KAN-Mamba feature extraction module, feature10 is processed by all the VSS-KAN feature extraction modules in series in this stage to obtain the feature sequence Out O_0 ; In the second stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_1 ; In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ; In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract the Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ; Out O_0 、Out O_1 、Out O_2 、Out O_3 Together they are called multi-scale features.

10. The remote sensing image semantic segmentation device based on KAN-Mamba according to claim 8, characterized in that: In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the normalized input image into four scan sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector and inputs it into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer; the Group-Rational The KAN layer consists of an input layer, two linear layers, two Group-KAN layers, and an output layer.

Images