Remote sensing image semantic segmentation method and device based on Kan-Mamba model

Abstract

The invention discloses a KAN-Mama-based multi-source remote sensing image semantic segmentation method and device, which is used for improving the accuracy of remote sensing image semantic segmentation, and comprises a KAN-Mama feature extraction network and a UpHead decoder. The KAN-Mama feature extraction network comprises a Stem convolutional neural network and a four-stage KAN-Mama feature extraction module, and staged feature extraction is performed on the fused remote sensing image through the KAN-Mama feature extraction network to obtain multi-scale features; the UpHead decoder comprises a side edge feature construction unit, a pooling unit, a feature fusion unit and a classification layer, side edge feature construction and pooling operation are performed on multi-scale features through the side edge feature construction unit and the pooling unit, step-by-step feature fusion is performed through the feature fusion unit, channel fusion is performed through a BottleNeck layer, and the classification layer is used for classifying the multi-scale features. And finally, obtaining a semantic segmentation label through the classification layer. The semantic segmentation accuracy of the multi-source remote sensing image is higher, the model convergence is faster, and the detail information is more accurate.

Description

Technical Field

[0001] The invention relates to computer vision and image processing technology, and in particular to a multi-source remote sensing image semantic segmentation method and device based on a Kan-Mamba model, belonging to remote sensing image segmentation technology. Background Art

[0002] Acquiring surface spectral information through remote sensing technology can monitor many environmental issues, such as changes in vegetation cover, water pollution, and changes in land use. For example, with multispectral or hyperspectral remote sensing data, different vegetation types can be distinguished, vegetation health can be assessed, and ecosystems can be effectively monitored. In addition, remote sensing technology is also very useful in monitoring natural disasters, such as floods, droughts, forest fires, etc., and can provide key information for disaster warning and emergency response. At the same time, with the rapid development of remote sensing technology and drone technology, the ability to obtain high-resolution images continues to increase. However, how to extract useful information from massive image data, especially pixel-level semantic segmentation, has become a difficult problem that needs to be solved urgently.

[0003] At present, it is a common method to use optical remote sensing images for detection and analysis. However, this is only a single remote sensing data source and it is difficult to meet actual needs. When only optical remote sensing images are used for analysis and segmentation, there are many obvious defects: optical images are easily affected by weather conditions, such as clouds, rain, haze, etc., which will seriously reduce the image quality and cause information loss in some areas; optical remote sensing mainly obtains information in visible light and near-infrared bands, lacks observations of other bands (such as thermal infrared, microwaves, etc.), and limits the effective analysis of certain specific objects (such as water bodies, wetlands, etc.); shadows cast by buildings and trees may cause ground features to be misclassified. In addition, the reflection of certain objects (such as water surface and smooth ground) at different angles may interfere with the observation results and cause misjudgment; in complex environments such as cities or forests, buildings or trees may block ground features, resulting in some information being invisible, affecting the accurate identification of objects.

[0004] To overcome the above difficulties, using multiple remote sensing images to provide complementary features is a feasible solution. In multi-source segmentation, data fusion technology can be used to integrate data from different sources. For example, the color features of optical images are combined with the texture features of SAR images, because SAR can acquire images under various weather conditions (including clouds, rain, and night) and is not affected by light; at the same time, SAR can provide high-spatial-resolution images, which is conducive to detail analysis and feature extraction. This can more comprehensively display the characteristics of the ground objects, thereby improving the classification accuracy. However, given the heterogeneity of multi-source data, it is difficult to directly integrate it into existing remote sensing detection methods, or it is difficult to achieve the expected performance.

[0005] In the field of computer vision, Mamba is an innovative approach. On the one hand, it can resolve the dilemma faced by convolutional neural networks (CNNs) and visual transformers (ViTs). CNNs are good at extracting local features, but it is difficult to capture long-distance dependencies without modifying the complex architecture; ViTs can effectively simulate global relationships, but the quadratic complexity of the self-attention mechanism leads to high computational costs. On the other hand, Mamba uses a selectively structured state space model to effectively capture long-range dependencies with linear computational complexity. This makes it outstanding in overcoming the limitations of existing methods and brings new possibilities for the development of remote sensing image segmentation.

[0006] The Kolmogorov-Arnold representation theorem states that if f is a multivariate continuous function over a bounded domain, then f can be written as a finite combination of univariate continuous functions and binary addition operations. The MLP is inspired by the universal approximation theorem, while the KAN is inspired by the Kolmogorov-Arnold representation theorem. Like the MLP, the KAN has a fully connected structure. However, while the MLP sets fixed activation functions on the nodes ("neurons"), the KAN sets learnable activation functions on the edges ("weights"). Therefore, the KAN does not have a linear weight matrix at all: each weight parameter is replaced by a learnable one-dimensional function parameterized as a spline. The nodes of the KAN only need to sum the input signals, without applying any nonlinear factors. Therefore, the computational graph of the KAN is usually much smaller than that of the MLP.

[0007] Existing remote sensing image segmentation methods still face many difficulties: 1. Multi-source data may contain redundant or irrelevant information, making it difficult to select appropriate features and effectively extract them, making it difficult to avoid information noise; 2. To effectively process multi-source data, the model may require a more complex architecture, which increases the difficulty of training and optimization and easily leads to overfitting; 3. Different types of remote sensing data may have semantic differences. For example, SAR images and optical images present the same object differently, which may cause the model to make mistakes in classification. To deal with these problems, an effective solution is to find a method that has better processing capabilities for high-dimensional data from satellite and aerial images, so that it can adapt to the field of remote sensing image segmentation. Summary of the invention

[0008] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention proposes a multi-source remote sensing image semantic segmentation method and device based on the Kan-Mamba model, which comprehensively utilizes multi-source remote sensing image information to improve segmentation accuracy and interpretability.

[0009] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0010] A multi-source remote sensing image semantic segmentation method based on KAN-Mamba includes the following steps:

[0011] Step 1: Obtain multi-source remote sensing images, including optical images (Optical), synthetic aperture radar images (SAR) and semantic segmentation labels (MASK) for training, and perform channel fusion, cropping and standardization on the remote sensing images. The size of the fused remote sensing image obtained after processing is 5×H×W to ensure the accuracy and consistency of the data;

[0012] Step 2: Divide the processed fused remote sensing images into training set and test set according to the proportion, where the training set includes fused remote sensing image O train and semantic label M train , the test set includes fused remote sensing images O test ;

[0013] Step 3: Construct a KAN-Mamba feature extraction network. The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth stage KAN-Mamba feature extraction modules also include a downsampling module.

[0014] O train The data is input to the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature 10 is obtained. Feature 10 is used as the input of the first-stage KAN-Mamba feature extraction module.

[0015] In the first-stage KAN-Mamba feature extraction module, feature 10 is processed by all the VSS-KAN feature extraction modules connected in series in this stage to obtain the feature sequence Out. O_0 ;

[0016] In the second-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules connected in series in this stage are used to obtain the feature sequence Out. O_1 ;

[0017] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_1Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ;

[0018] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ;

[0019] Step 4: Out O_0 、Out O_1 、Out O_2 、Out O_3 Input to Uper_Head decoder for feature fusion;

[0020] For Out O_0 、Out O_1 、Out O_2 First, construct the side features separately, and then perform 1×1 convolution to obtain For Out O_3 , first perform multi-scale pooling or average pooling, and then use convolution to reduce the dimension to obtain the output feature Out U_3 ; for Out U_3 After upsampling, Perform addition operation to get output feature Out U_2 ; for Out U_2 After upsampling, Perform addition operation to get output feature Out U_1 ;right After upsampling, Perform addition operation to get output feature Out U_0 ; for Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then pass through the BottleNeck layer and the classification layer to obtain the fusion feature Out train ;

[0021] Step 5: Compare Out train and M train , using the cross entropy loss function loss train Calculate Out train and M train The difference between

[0022] Step 6: Through loss train Perform back propagation to optimize the KAN-Mamba feature extraction network and Uper_Head decoder;

[0023] Step7: Repeat Step 4 to Step 6 until loss train Convergence, completing the training of the KAN-Mamba feature extraction network;

[0024] Step8: First, test Input the trained KAN-Mamba feature extraction network and Uper_Head decoder, and output the multi-scale feature Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder, output fusion feature Out train , Out train As the prediction result of semantic label.

[0025] Specifically, in the KAN-Mamba feature extraction network, the number of channels of the one-stage KAN-Mamba feature extraction module, the two-stage KAN-Mamba feature extraction module, the three-stage KAN-Mamba feature extraction module and the four-stage KAN-Mamba feature extraction module are 96, 192, 384 and 768 respectively.

[0026] Specifically, in the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization operation on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the input image after the normalization operation into four scanning sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector input into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer.

[0027] Specifically, the Mamba discretized state space equation is expressed as:

[0028] h′(t)=Ah(t)+Bu(t)

[0029] y(t)=Ch(t)+Du(t)

[0030] in: is the hidden input at time t, h′(t) is the derivative of h(t), is the input stimulus at time t, is the output response at time t; is the state space equation weight matrix, is the weighting parameter; N is the dimension of the hidden input, and L is the length of the input stimulus.

[0031] Specifically, the Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers and an output layer. The linear layer performs linear mapping on the input image. The Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x):

[0032]

[0033] Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, …, a m , b0, b1, …, b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

[0034] Specifically, the Group-Rational KAN layer divides the number of channels of the input layer into more than one group. In each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) of different groups are different.

[0035] Specifically, Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are Where H and W are the height and width of the fused remote sensing image obtained after Step 1 processing; Out U_0 、Out U_1、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the 3×3 convolution and classification layer outputs Out train ,Out train The size is K represents the total number of categories for image segmentation.

[0036] Specifically, the classification layer includes a Dropout layer and a 1×1 convolution layer, the dropout ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories of image segmentation, that is, the total number of categories of labels.

[0037] Specifically, the cross entropy loss function loss train It is expressed as:

[0038]

[0039] Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

[0040] A multi-source remote sensing image semantic segmentation device based on KAN-Mamba, used to implement the above method, comprising a multi-source remote sensing image acquisition unit, a KAN-Mamba feature extraction network, an Uper_Head decoder, a training unit and a testing unit;

[0041] The multi-source remote sensing image acquisition unit is used to acquire fused remote sensing images and semantic labels of the same size and geometric alignment;

[0042] The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The KAN-Mamba feature extraction network is used to extract features of the fused remote sensing image in stages to obtain multi-scale features.

[0043] The Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer. The side feature construction unit and the pooling unit perform side feature construction and pooling operations on the multi-scale features, and then the feature fusion unit performs step-by-step feature fusion, and then the BottleNeck layer performs channel fusion, and finally the classification layer obtains the fusion feature; the BottleNeck layer performs a 3×3 convolution operation on the input feature, converts the number of channels of the input feature from 4C to C, and C is the number of output channels set by the Uper_Head decoder; the classification layer performs a Dropout operation and a 1×1 convolution operation on the input feature, and outputs the prediction result Out of the semantic label. train ;

[0044] The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and the Uper_Head decoder;

[0045] The test unit uses the trained KAN-Mamba feature extraction network and Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

[0046] Specifically, in the KAN-Mamba feature extraction network:

[0047] For the input fused remote sensing image O train , after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained, and feature10 is used as the input of the first-stage KAN-Mamba feature extraction module;

[0048] In the first-stage KAN-Mamba feature extraction module, feature 10 is processed by all the VSS-KAN feature extraction modules connected in series in this stage to obtain the feature sequence Out. O_0 ;

[0049] In the second-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules connected in series in this stage are used to obtain the feature sequence Out. O_1 ;

[0050] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ;

[0051] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ;

[0052] Out O_0 、Out O_1 、Out O_2 、Out O_3 Together they are called multi-scale features.

[0053] Specifically, in the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization operation on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the input image after the normalization operation into four scanning sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector input to the S6 module, and uses the Mamba discretized state space equation in the S6 module to extract features to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer; the Group-Rational The KAN layer consists of an input layer, two linear layers, two Group-KAN layers, and an output layer.

[0054] Beneficial effect: The multi-source remote sensing image semantic segmentation method and device based on the Kan-Mamba model provided by the present invention establish a segmentation model by introducing a feature extraction framework based on Kan-Mamba and an Uper_Head decoder, and perform semantic segmentation on the multi-source remote sensing images, so that the semantic segmentation accuracy of the multi-source remote sensing images is higher, the model converges faster, and the detail information is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the framework structure of the device of the present invention;

[0056] Figure 2 1 is a flow chart for implementing the method of the present invention;

[0057] Figure 3For pairs of original optical images and original synthetic aperture radar images, fused remote sensing images and semantic labels, the prediction results of semantic labels of the trained KAN-Mamba feature extraction network and Uper_Head decoder. DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 The figure shows a multi-source remote sensing image semantic segmentation device based on KAN-Mamba, including a multi-source remote sensing image acquisition unit, a KAN-Mamba feature extraction network, an Uper_Head decoder, a training unit and a testing unit. The multi-source remote sensing image acquisition unit is used to acquire fused remote sensing images and semantic labels of the same size and geometric alignment; the KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module, and the KAN-Mamba feature extraction network performs staged feature extraction on the fused remote sensing image to obtain multi-scale features; the Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer, and performs side feature construction and pooling operations on multi-scale features through the side feature construction unit and the pooling unit, and then performs step-by-step feature fusion through the feature fusion unit, and then performs channel fusion through the BottleNeck layer, and finally obtains the fusion feature through the classification layer; the BottleNeck layer performs a 3×3 convolution operation on the input feature, and converts the number of channels of the input feature from 4C to C, where C is the number of output channels set by the Uper_Head decoder; the classification layer performs a Dropout operation and a 1×1 convolution operation on the input feature, and outputs the prediction result Out of the semantic label train ; The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and the Uper_Head decoder; the testing unit uses the trained KAN-Mamba feature extraction network and the Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

[0060] like Figure 2 As shown, for use Figure 1 The process of performing multi-source remote sensing image semantic segmentation by the multi-source remote sensing image semantic segmentation device based on KAN-Mamba is described in detail below in conjunction with each step.

[0061] Step 1: Obtain multi-source remote sensing images

[0062] Acquire multi-source remote sensing images, including optical images (Optical), synthetic aperture radar images (SAR) and semantic segmentation labels (MASK) for training, perform channel fusion, cropping and standardization on the remote sensing images. The size of the fused remote sensing image obtained after processing is 5×H×W to ensure the accuracy and consistency of the data for subsequent processing; Figure 3 As shown, the fused remote sensing image and semantic labels obtained after processing.

[0063] Step 2: Divide the training set and test set

[0064] The processed fused remote sensing images are divided into training set and test set in a ratio of 7:3, where the training set includes fused remote sensing images O train and semantic label M train , the test set includes fused remote sensing images O test .

[0065] Step 3: Build the KAN-Mamba feature extraction network

[0066] The KAN-Mamba feature extraction network can also be called the KAN-Mamba backbone network, which includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module.

[0067] 3.1 Architecture of KAN-Mamba Feature Extraction Network

[0068] The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth-stage KAN-Mamba feature extraction modules also include a downsampling module. The number of channels of the first-stage KAN-Mamba feature extraction module, the second-stage KAN-Mamba feature extraction module, the third-stage KAN-Mamba feature extraction module, and the fourth-stage KAN-Mamba feature extraction module are 96, 192, 384, and 768, respectively.

[0069] In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer.

[0070] The normalization layer performs a layer normalization operation on the input image.

[0071] The SS2D module includes a cross-scan module and an S6 module. The cross-scan module first expands the input image after normalization into four scanning sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector and inputs it into the S6 module. The Mamba discretized state space equation in the S6 module is used to extract features to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer. The Mamba discretized state space equation is expressed as:

[0072] h′(t)=Ah(t)+Bu(t)

[0073] y(t)=Ch(t)+Du(t)

[0074] in: is the hidden input at time t, h′(t) is the derivative of h(t), is the input stimulus at time t, is the output response at time t; is the state space equation weight matrix, is the weighting parameter; N is the dimension of the hidden input, and L is the length of the input stimulus.

[0075] The Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers and an output layer. The linear layer performs linear mapping on the input image. The Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x):

[0076]

[0077] Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, …, a m , b0, b1, …, b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

[0078] The Group-Rational KAN layer divides the number of channels of the input layer into more than one group. In each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) of different groups are different.

[0079] 3.1 Execution process of KAN-Mamba feature extraction network

[0080] O train The image is input into the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained. Feature10 is used as the input of the first-stage KAN-Mamba feature extraction module.

[0081] In the first-stage KAN-Mamba feature extraction module, feature 10 is processed by all the VSS-KAN feature extraction modules connected in series in this stage to obtain the feature sequence Out. O_0 .

[0082] In the second-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules connected in series in this stage are used to obtain the feature sequence Out. O_1 .

[0083] In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 .

[0084] In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 .

[0085] Multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are

[0086] Step 4: Build the Uper_Head decoder

[0087] Construct the Uper_Head decoder for multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 Feature fusion is performed; the Uper_Head decoder includes a side feature construction unit, a PPM multi-scale pooling unit, a feature fusion unit and a classification layer; the classification layer includes a Dropout layer and a 1×1 convolution layer, the discard ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories of image segmentation, that is, the total number of categories of labels.

[0088] For Out O_0 、Out O_1 、Out O_2 First, use the side feature construction unit to construct the side features respectively, and then perform 1×1 convolution to obtain For Out O_3 First, use the PPM multi-scale pooling unit to perform multi-scale pooling or average pooling, and then use convolution to reduce the dimension to obtain the output feature Out U_3 ; for Out U_3 After upsampling, Perform addition operation to get output feature Out U_2 ; for Out U_2 After upsampling, Perform addition operation to get output feature Out U_1 ;right After upsampling, Perform addition operation to get output feature Out U_0 ; for Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then perform a 3×3 convolution operation through the BottleNeck layer to convert Out concat The number of channels is converted from 4C to C; finally, the classification layer performs Dropout operation and 1×1 convolution operation to output the fusion feature Out train , that is, the prediction result of the semantic label Out train .

[0089] Out U_0 、Out U_1 、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the 3×3 convolution and classification layer outputs Out train ,Out train The size is K represents the total number of categories for image segmentation.

[0090] Step 5: Use loss function for supervised training

[0091] The training unit uses the loss function to supervise the training of the KAN-Mamba feature extraction network and the Uper_Head decoder. In the process of training the KAN-Mamba feature extraction network and the Uper_Head decoder, the Out signal output by the Uper_Head decoder is firstly train Use bilinear interpolation to resize to the same size as M train Consistent, then use the cross entropy loss function loss to calculate Out train and M train The difference between:

[0092]

[0093] Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

[0094] If the cross entropy loss converges, the training of the KAN-Mamba feature extraction network and Uper_Head decoder is completed; otherwise, backpropagation operation is performed, parameters are updated, and training is performed again until the loss value converges.

[0095] Step 6: Test using the trained KAN-Mamba feature extraction network and Uper_Head decoder

[0096] like Figure 2 As shown, O test Input to the trained KAN-Mamba feature extraction network and output multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then multi-scale features Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder to get the predicted semantic label Out train.from Figure 3 It can be seen from the comparison of the prediction results and labels in that the present invention performs well in details and boundaries, can accurately distinguish different types of land cover, and can handle multiple different types of land cover, such as forests, grasslands, buildings, water bodies, etc. Each type of land can be accurately segmented and classified, and will not perform poorly because some categories are less common.

[0097] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multi-source remote sensing image semantic segmentation method based on KAN-Mamba, characterized by: The steps include: Step 1: Obtain multi-source remote sensing images, including optical images, synthetic aperture radar images, and semantic segmentation labels for training, and perform channel fusion, cropping, and standardization on the remote sensing images. The size of the fused remote sensing image after processing is 5×H×W; Step 2: Divide the processed fused remote sensing images into training set and test set according to the proportion, where the training set includes fused remote sensing image O train and semantic label M train , the test set includes fused remote sensing images O test ; Step 3: Construct a KAN-Mamba feature extraction network. The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The four-stage KAN-Mamba feature extraction modules include 2, 2, 15, and 2 VSS-KAN feature extraction modules in series, respectively. The second, third, and fourth stage KAN-Mamba feature extraction modules also include a downsampling module. O train The data is input to the KAN-Mamba feature extraction network, and after four times downsampling by the Stem convolutional neural network, the convolution feature feature 10 is obtained. Feature 10 is used as the input of the first-stage KAN-Mamba feature extraction module. In the first-stage KAN-Mamba feature extraction module, feature 10 is processed by all the VSS-KAN feature extraction modules connected in series in this stage to obtain the feature sequence Out. O_0 ; In the second-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules connected in series in this stage are used to obtain the feature sequence Out. O_1 ; In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ; In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ; Step 4: Out O_0 、Out O_1 、Out O_2 、Out O_3 Input to Uper_Head decoder for feature fusion; For Out O_0 、Out O_1 、Out O_2 First, construct the side features separately, and then perform 1×1 convolution to obtain For Out O_3 , first perform multi-scale pooling or average pooling, and then use convolution to reduce the dimension to obtain the output feature Out U_3 ; for Out U_3 After upsampling, Perform addition operation to get output feature Out U_2 ; for Out U_2 After upsampling, Perform addition operation to get output feature Out U_1 ;right After upsampling, Perform addition operation to get output feature Out U_0 ; for Out U_0 、Out U_1 、Out U_2 、Out U_3 Splice to get Out concat , and then pass through the BottleNeck layer and the classification layer to obtain the fusion feature Out train ; Step 5: Compare Out train and M train , using the cross entropy loss function loss train Calculate Out train and M train The difference between Step 6: Through loss train Perform back propagation to optimize the KAN-Mamba feature extraction network and Uper_Head decoder; Step7: Repeat Step 4 to Step 6 until loss train Convergence, completing the training of the KAN-Mamba feature extraction network; Step8: First, test Input the trained KAN-Mamba feature extraction network and Uper_Head decoder, and output the multi-scale feature Out O_0 、Out O_1 、Out O_2 、Out O_3 ; Then Out O_0 、Out O_1 、Out O_2 、Out O_3 Input Uper_Head decoder, output fusion feature Out train , Out train As the prediction result of semantic label.

2. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization operation on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the input image after normalization operation into four scanning sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector input into the S6 module, and uses the Mamba discretized state space equation in the S6 module to perform feature extraction to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer.

3. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 2, characterized in that: The Group-Rational KAN layer includes an input layer, two linear layers, two Group-KAN layers and an output layer. The linear layer performs linear mapping on the input image. The Group-KAN layer parameterizes the activation function φ(x) as a rational number related to the m-order polynomial P(x) and the n-order polynomial Q(x): Where: x is the input of the activation function, a0, a1, ..., a m and b0, b1, …, b n are the coefficients of polynomial P(x) and polynomial Q(x), w is the scaling factor; a0, a1, …, a m , b0, b1, …, b n and w are the parameters of the activation function φ(x). The KAN-Mamba feature extraction network optimizes the parameters of the activation function φ(x) through back propagation.

4. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 3 is characterized in that: The Group-Rational KAN layer divides the number of channels of the input layer into more than one group. In each group, all channels share the parameters of the activation function φ(x), and the parameters of the activation function φ(x) of different groups are different.

5. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: Out O_0 、Out O_1 、Out O_2 、Out O_3 The sizes are Where H and W are the height and width of the fused remote sensing image obtained after Step 1 processing; Out U_0 、Out U_1 、Out U_2 、Out U_3 The sizes are Where C is the number of output channels set by the Uper_Head decoder; Out concat The size is Finally, the 3×3 convolution and classification layer outputs Out train ,Out train The size is K represents the total number of categories for image segmentation.

6. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: The classification layer includes a Dropout layer and a 1×1 convolution layer. The dropout ratio of the Dropout layer is designed to be 0.1, and the output channel of the 1×1 convolution layer is set to K, where K represents the total number of categories for image segmentation, that is, the total number of categories for labels.

7. The multi-source remote sensing image semantic segmentation method based on KAN-Mamba according to claim 1, characterized in that: The cross entropy loss function loss train It is expressed as: Where: M represents the total number of samples, K represents the total number of label categories, y m represents the true label corresponding to sample m, z m，k Indicates the probability that sample m is predicted to be category k.

8. A multi-source remote sensing image semantic segmentation device based on KAN-Mamba, characterized by: It includes multi-source remote sensing image acquisition unit, KAN-Mamba feature extraction network, Uper_Head decoder, training unit and testing unit; The multi-source remote sensing image acquisition unit is used to acquire fused remote sensing images and semantic labels of the same size and geometric alignment; The KAN-Mamba feature extraction network includes a Stem convolutional neural network and a four-stage KAN-Mamba feature extraction module. The KAN-Mamba feature extraction network is used to extract features of the fused remote sensing image in stages to obtain multi-scale features. The Uper_Head decoder includes a side feature construction unit, a pooling unit, a feature fusion unit and a classification layer. The side feature construction unit and the pooling unit perform side feature construction and pooling operations on the multi-scale features, and then the feature fusion unit performs step-by-step feature fusion, and then the BottleNeck layer performs channel fusion, and finally the classification layer obtains the fusion feature; the BottleNeck layer performs a 3×3 convolution operation on the input feature, converts the number of channels of the input feature from 4C to C, and C is the number of output channels set by the Uper_Head decoder; the classification layer performs a Dropout operation and a 1×1 convolution operation on the input feature, and outputs the prediction result Out of the semantic label. train ; The training unit uses a loss function to perform supervised training on the KAN-Mamba feature extraction network and the Uper_Head decoder; The test unit uses the trained KAN-Mamba feature extraction network and Uper_Head decoder to predict semantic labels for multi-source remote sensing images.

9. The multi-source remote sensing image semantic segmentation device based on KAN-Mamba according to claim 8, characterized in that: In the KAN-Mamba feature extraction network: For the input fused remote sensing image O train , after four times downsampling by the Stem convolutional neural network, the convolution feature feature10 is obtained, and feature10 is used as the input of the first-stage KAN-Mamba feature extraction module; In the first-stage KAN-Mamba feature extraction module, feature 10 is processed by all the VSS-KAN feature extraction modules connected in series in this stage to obtain the feature sequence Out. O_0 ; In the second-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_0 Downsampling is performed to obtain the convolution feature feature11, and then all the VSS-KAN feature extraction modules connected in series in this stage are used to obtain the feature sequence Out. O_1 ; In the three-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_1 Downsampling is performed to obtain the convolution feature feature12, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_2 ; In the four-stage KAN-Mamba feature extraction module, the downsampling module of this stage is first used to extract Out O_2 Downsampling is performed to obtain the convolution feature feature13, and then all the VSS-KAN feature extraction modules in series at this stage are used to obtain the feature sequence Out. O_3 ; Out O_0 、Out O_1 、Out O_2 、Out O_3 Together they are called multi-scale features.

10. The remote sensing image semantic segmentation device based on KAN-Mamba according to claim 8, characterized in that: In the KAN-Mamba feature extraction module, the VSS-KAN feature extraction module includes a normalization layer, an SS2D module and a Group-Rational KAN layer; the normalization layer performs layer normalization operation on the input image; the SS2D module includes a cross-scan module and an S6 module, the cross-scan module first expands the input image after normalization operation into four scanning sequences Scan1, Scan2, Scan3, and Scan4 along four different traversal paths from upper left to lower right, lower left to upper right, upper right to lower left, and lower right to upper left, and then splices Scan1, Scan2, Scan3, and Scan4 into a four-channel vector input to the S6 module, and uses the Mamba discretized state space equation in the S6 module to extract features to obtain output features; the Group-Rational KAN layer first flattens the output features of the SS2D module into a 1D sequence, then performs patch embedding and position encoding, and finally outputs the feature sequence through the KAN layer; the Group-Rational The KAN layer consists of an input layer, two linear layers, two Group-KAN layers, and an output layer.

Images