Semantic segmentation method and device of remote sensing image, electronic equipment and storage medium

Abstract

The invention relates to a semantic segmentation method and device for a remote sensing image, electronic equipment and a storage medium. Comprising the steps that an obtained remote sensing image data set is preprocessed, feature extraction and fusion are conducted on the preprocessed remote sensing image data set through a preset model framework, the fused features of each pixel are obtained, and the preset model framework comprises a Mamba2 encoder and a CNN decoder; and classifying the fused features of each pixel by using a preset classification network, and determining the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, and the problems that in the prior art, when large-size images are processed, the calculation cost is too high, and when complex dependency relationships are processed, the calculation cost is limited by the deficiency of training data and the generalization ability are solved. Therefore, more accurate ground feature classification and semantic information are provided.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and remote sensing image processing, and in particular to a remote sensing image semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] Remote sensing image semantic segmentation refers to the pixel-level classification and labeling of remote sensing images through computer vision technology, so that each pixel corresponds to a specific semantic category. Remote sensing images are widely used in environmental monitoring, urban planning, agricultural surveys, disaster assessment and other fields. Accurate semantic segmentation of these images is of great significance for extracting useful information, conducting scientific analysis and decision-making.

[0003] In related technologies, semantic segmentation of remote sensing images can be performed based on CNN (Convolutional Neural Networks) and Transformer (a sequence model based on an attention mechanism).

[0004] However, this method has high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when dealing with complex dependencies, which needs to be solved urgently. Summary of the invention

[0005] The present application provides a semantic segmentation method, device, electronic device and storage medium for remote sensing images to solve the problems of high computational cost in the prior art when processing large-size images and limited generalization ability due to insufficient training data when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0006] To achieve the above objectives, the first embodiment of the present application proposes a semantic segmentation method for remote sensing images, comprising the following steps:

[0007] Acquire remote sensing image datasets;

[0008] Preprocessing the remote sensing image data set, and performing feature extraction and fusion on the preprocessed remote sensing image data set using a preset model framework to obtain fused features of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder;

[0009] The fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

[0010] According to an embodiment of the present application, the method of extracting and fusing the preprocessed remote sensing image data set using a preset model framework to obtain the fused features of each pixel includes:

[0011] Inputting the preprocessed remote sensing image data set into the Mamba2 encoder, performing multi-level downsampling operations on the preprocessed remote sensing image data set using filters of different sizes and step sizes, and obtaining downsampling features of different scales;

[0012] Inputting the down-sampled features that meet the preset spatial resolution condition among the down-sampled features of different scales into the CNN decoder, and performing an up-sampling operation on the down-sampled features that meet the preset spatial resolution condition by using a bilinear interpolation algorithm to obtain up-sampled features;

[0013] The down-sampled features of different scales are processed to obtain a graph structure. Based on the graph structure, the down-sampled features of different scales are updated by using a graph convolutional neural network (GCN for short). The updated down-sampled features of different scales are fused with the up-sampled features of corresponding scales through a feature fusion layer to obtain the fused features of each pixel.

[0014] According to an embodiment of the present application, the down-sampled features of different scales are processed using a graph convolutional neural network GCN to obtain a graph structure, including:

[0015] Each pixel is taken as a node, the relationship between edges is determined by calculating the similarity between nodes, and the graph structure is obtained based on the relationship between the edges.

[0016] According to one embodiment of the present application, the update formula of the GCN is:

[0017]

[0018] Among them, H(1) is the feature matrix of the first layer of GCN, is the degree matrix of the graph, A is the adjacency matrix of the graph, H(1-1) is the feature matrix of the l-1 layer, W(1) is the weight matrix of the lth layer, and σ(·) is the nonlinear activation function.

[0019] According to an embodiment of the present application, the method of using the preset classification network to classify the fused features of each pixel to determine the semantic category of the remote sensing image corresponding to each pixel includes:

[0020] Using a classification head to map the fused features of each pixel into a category score vector of a fixed size through multiple fully connected layers;

[0021] The category score vector is processed using a preset softmax function to generate a probability distribution of each pixel belonging to each semantic category.

[0022] According to the semantic segmentation method of remote sensing images proposed in the embodiment of the present application, by preprocessing the acquired remote sensing image data set, and using a preset model framework to extract and fuse the preprocessed remote sensing image data set, the fused features of each pixel can be obtained, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; and then the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced model Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, solving the problem that the existing technology has too high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0023] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a semantic segmentation device for remote sensing images, comprising:

[0024] An acquisition module is used to acquire remote sensing image data sets;

[0025] A processing module, used for preprocessing the remote sensing image data set, extracting and fusing features of the preprocessed remote sensing image data set using a preset model framework to obtain fused features of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder;

[0026] The classification module is used to classify the fused features of each pixel using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

[0027] According to one embodiment of the present application, the processing module includes:

[0028] A first sampling unit is used to input the preprocessed remote sensing image data set into the Mamba2 encoder, and perform a multi-level downsampling operation on the preprocessed remote sensing image data set using filters of different sizes and step sizes to obtain downsampling features of different scales;

[0029] A second sampling unit is used to input the down-sampled features that meet the preset spatial resolution condition in the different scales into the CNN decoder, and perform an up-sampling operation on the down-sampled features that meet the preset spatial resolution condition by using a bilinear interpolation algorithm to obtain up-sampled features;

[0030] A fusion unit is used to process the down-sampled features of different scales to obtain a graph structure, and based on the graph structure, use a graph convolutional neural network (GCN) to update the down-sampled features of different scales, and fuse the updated down-sampled features of different scales with the up-sampled features of corresponding scales through a feature fusion layer to obtain the fused features of each pixel.

[0031] According to one embodiment of the present application, the fusion unit is specifically used for:

[0032] Each pixel is taken as a node, the relationship between edges is determined by calculating the similarity between nodes, and the graph structure is obtained based on the relationship between the edges.

[0033] According to one embodiment of the present application, the update formula of the GCN is:

[0034]

[0035] Among them, H(1) is the feature matrix of the first layer of GCN, is the degree matrix of the graph, A is the adjacency matrix of the graph, H(1-1) is the feature matrix of the l-1 layer, W(1) is the weight matrix of the lth layer, and σ(·) is the nonlinear activation function.

[0036] According to one embodiment of the present application, the classification module is specifically used for:

[0037] Using a classification head to map the fused features of each pixel into a category score vector of a fixed size through multiple fully connected layers;

[0038] The category score vector is processed using a preset softmax function to generate a probability distribution of each pixel belonging to each semantic category.

[0039] According to the semantic segmentation device of remote sensing images proposed in the embodiment of the present application, by preprocessing the acquired remote sensing image data set, and using the preset model framework to extract and fuse the features of the preprocessed remote sensing image data set, the fused features of each pixel can be obtained, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; and then the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced model Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, solving the problem that the existing technology has too high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0040] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the semantic segmentation method of remote sensing images as described in the above-mentioned embodiment.

[0041] To achieve the above objectives, the fourth aspect of the present application proposes a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the semantic segmentation method of remote sensing images as described in the above embodiments.

[0042] To achieve the above objectives, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed, is used to implement the semantic segmentation method of remote sensing images as described in the above embodiments.

[0043] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A flowchart of a method for semantic segmentation of remote sensing images provided according to an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a model architecture according to an embodiment of the present application;

[0047] Figure 3 A flowchart of another remote sensing image semantic segmentation method provided according to an embodiment of the present application;

[0048] Figure 4 A schematic block diagram of a semantic segmentation device for remote sensing images provided according to an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0051] The following describes the semantic segmentation method, device, electronic device and storage medium of remote sensing images proposed in accordance with the embodiments of the present application with reference to the accompanying drawings.

[0052] Figure 1 It is a flowchart of a method for semantic segmentation of remote sensing images according to an embodiment of the present application.

[0053] Before introducing the semantic segmentation method of remote sensing images proposed in the embodiment of the present application, the relevant technical background is first introduced.

[0054] Current semantic segmentation techniques mainly focus on exploring features at different scales, but often ignore the spatial topological structure between these features, resulting in incomplete extracted image information, limiting the complementary and guiding role between features at different scales, and thus affecting the integrity and accuracy of feature expression; in addition, the spatial topological relationship between cross-scale features is inherently complex. For example, in remote sensing image analysis, objects of the same scale are often adjacent or bordering, such as two buildings side by side, while objects of different scales exhibit nested or overlapping relationships, such as multiple cars in a parking lot. Therefore, the spatial relationships between objects of the same scale and different scales are both diverse and complex. If these spatial relationships are not distinguished during analysis, they may affect each other and interfere with the model's effective extraction of image content.

[0055] Current CNN and Transformer-based remote sensing image semantic segmentation methods face a series of challenges when processing remote sensing data, especially the large variations in image size and object size and the efficiency of capturing long-range dependencies. First, CNNs are usually limited in their effectiveness when processing image sizes and object sizes with high variability. Although CNNs are good at capturing local features, when objects in remote sensing images vary greatly in size in space, standard convolution operations may have difficulty effectively adjusting their receptive fields to capture all relevant contextual information. For example, for small-sized objects, such as small vehicles or road signs, traditional CNNs may not be able to accurately identify them against a wide background. Second, although the Transformer architecture improves the model's ability to capture long-range dependencies through the self-attention mechanism, which is particularly useful for understanding and segmenting remote sensing images with complex spatial relationships, this method is extremely computationally expensive when processing large-size images. The self-attention mechanism requires the calculation of the relationship between all input elements, which leads to a significant computational burden when the input size is large. In addition, when the input image resolution increases, the memory requirements and training difficulty of the Transformer also increase significantly. Although Transformer can handle complex dependencies, its practical application may still be limited by insufficient training data and limited generalization capabilities. In the field of remote sensing, images taken in different regions and under different conditions may have great differences, which requires the model to have high adaptability and generalization capabilities, and the current Transformer model may not be able to fully meet these requirements.

[0056] Therefore, although CNN and Transformer based methods have made progress in the field of semantic segmentation of remote sensing images, they still face many problems in dealing with objects with large size variations, capturing complex long-range dependencies, and managing high computational costs.

[0057] It is based on the above problems that the embodiment of the present application proposes a semantic segmentation method for remote sensing images. By preprocessing the acquired remote sensing image data set, and using a preset model framework to extract and fuse the preprocessed remote sensing image data set, the fused features of each pixel can be obtained, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; and then the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced model Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, solving the problem that the existing technology has too high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0058] For example, Figure 1 As shown, the semantic segmentation method of the remote sensing image includes the following steps:

[0059] In step S101, a remote sensing image dataset is acquired.

[0060] That is to say, the embodiment of the present application first involves the preparation of remote sensing image datasets, that is, obtaining remote sensing image datasets and annotating the types to ensure that the datasets include a variety of ground object types to support a wide range of semantic classification. The acquisition of remote sensing image datasets can be queried through various data platforms (such as the Digital Earth Open Platform, etc.) or service providers.

[0061] In step S102, the remote sensing image data set is preprocessed, and feature extraction and fusion are performed on the preprocessed remote sensing image data set using a preset model framework to obtain a fused feature of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder.

[0062] Among them, the Mamba2 encoder is a new model based on the state space model, which is designed to efficiently process sequence data. Compared with the traditional Transformer architecture, Mamba2 overcomes the quadratic complexity problem of Transformer when processing long sequences. It uses linearly expanded computing and memory methods, which is particularly suitable for processing long sequence tasks. The CNN decoder is used to decode the output of the Mamba2 encoder through the CNN network and generate feature maps of the same or similar size as the input image.

[0063] After obtaining the remote sensing image data set, the data set is first preprocessed, including image resizing, normalization and data enhancement operations (such as rotation, flipping, cropping, etc.). Through these operations, the remote sensing image is resized to meet the model requirements while maintaining the original aspect ratio. Random cropping and flipping can increase data diversity and prevent overfitting. Photometric distortion simulates different lighting conditions, and finally the preprocessed remote sensing image data set and labels are encapsulated into a format suitable for model training. Secondly, the preprocessed remote sensing image data set is input into a preset model framework. The preset model framework used in the embodiment of the present application is a variant of the U-Net (U-shaped Network, a deep learning network model for image segmentation, named because its network structure is U-shaped) architecture, which can be called UPerNet. The structure has symmetrical encoder (i.e., Mamba2 encoder) and decoder (i.e., CNN decoder) paths, as well as jump connections. By combining the UPerNet model, the Mamba2 model and the convolutional model CNN, a new remote sensing image semantic segmentation framework can be constructed, which helps to retain the information lost during the downsampling process, thereby improving the accuracy of the semantic segmentation task. Finally, by using the preset model framework to extract and fuse the preprocessed remote sensing image dataset, the fused features of each pixel can be obtained.

[0064] Next, we will explain in detail how to use the preset model framework to extract and fuse the features of the preprocessed remote sensing image dataset to obtain the fused features of each pixel.

[0065] As a possible implementation method, in some embodiments, a preset model framework is used to perform feature extraction and fusion on a preprocessed remote sensing image dataset to obtain fused features of each pixel, including: inputting the preprocessed remote sensing image dataset into a Mamba2 encoder, performing multi-level downsampling operations on the preprocessed remote sensing image dataset using filters of different sizes and step sizes to obtain downsampled features of different scales; inputting downsampled features of different scales that meet preset resolution conditions into a CNN decoder, and performing upsampling operations on the downsampled features that meet the preset resolution conditions using a bilinear interpolation algorithm to obtain upsampled features; processing the downsampled features of different scales to obtain a graph structure, and based on the graph structure, using a graph convolutional neural network GCN to update the downsampled features of different scales, and fusing the updated downsampled features of different scales with the upsampled features of the corresponding scales through a feature fusion layer to obtain the fused features of each pixel.

[0066] Specifically, if Figure 2As shown, first, a Mamba2 encoder is constructed. The Mamba2 encoder is mainly composed of multiple Mamba2 modules, each of which is responsible for performing a certain degree of downsampling operation and extracting features of the input image (i.e., the preprocessed remote sensing image data set). The encoder in the embodiment of the present application may include four Mamba2 modules, which extract the features of the input image from basic to complex step by step.

[0067] Among them, the Mamba2 module uses high-dimensional latent Map the input function x(t)∈R to the output function y(t)∈R, where h represents the number of feature channels. These hidden states evolve over time according to the parameters A, B, and C and are controlled by linear ordinary differential equations as shown below:

[0068]

[0069] y t =Ch t ;

[0070]

[0071]

[0072] in, represents the evolution parameter, and C∈R H×1 Represents the projection parameters. Approximation is done via a zero-order hold discretization formula. A learnable parameter Δ is used to balance the degree of attention or neglect given to the current state and input.

[0073] Secondly, a CNN decoder is constructed. The CNN decoder consists of multiple CNN modules, each of which is responsible for performing a certain degree of upsampling operations.

[0074] After constructing the Mamba2 encoder and CNN decoder, the preprocessed remote sensing image dataset is input into the Mamba2 encoder, and the remote sensing image dataset is subjected to multi-level downsampling operations through filters of different sizes and step sizes to reduce the spatial resolution of the image while retaining important visual information as much as possible. At the same time, the Mamba2 encoder can utilize its hybrid structure, through convolutional layers, pooling layers (such as PPM (Pyramid Pooling Module, a technology used to improve the performance of image semantic segmentation in deep learning) modules, and the performance of deep convolutional neural networks in image semantic segmentation tasks is improved by introducing pyramid pooling modules) and nonlinear activation functions, to capture features at multiple scales from local details to global contexts, thereby obtaining downsampled features at different scales, realizing effective encoding of the input remote sensing image dataset, and laying the foundation for subsequent image processing tasks. Then, the down-sampled features of different scales that meet the preset spatial resolution conditions (i.e., the down-sampled features of the last stage obtained by multi-level down-sampling) are input into the CNN decoder, and the down-sampled features that meet the preset spatial resolution conditions are up-sampled by the bilinear interpolation algorithm to obtain the up-sampled features, so as to restore the original resolution of the image, increase the pixel density of the image, and improve the visual quality. The image after up-sampling by the CNN encoder can be restored to a size close to its original size. By converting the down-sampled features of different scales output by the Mamba2 encoder into a graph structure, the graph convolutional neural network GCN can be used to update the down-sampled features of different scales based on the graph structure, and then the updated down-sampled features of different scales obtained by GCN processing are fused with the up-sampled features of the corresponding scales output by the CNN decoder through the feature fusion layer, so as to gradually restore the detailed spatial information and semantic content of the remote sensing image, and ensure that the information at different levels is utilized and integrated to the maximum extent, so as to obtain the fused features of each pixel, wherein the fusion method can be implemented by adding in the channel dimension.

[0075] The following describes in detail how to process downsampled features of different scales to obtain graph structures of different scales.

[0076] As a possible implementation method, in some embodiments, downsampled features of different scales are processed to obtain a graph structure, including: taking each pixel as a node, determining the relationship between edges by calculating the similarity between nodes, and obtaining the graph structure based on the relationship between edges.

[0077] It can be understood that a graph structure refers to a data structure consisting of nodes and edges, which is used to represent complex relationships between entities (such as pixels).

[0078] Specifically, in the graph structure, each pixel of the downsampled features of different scales output by the Mamba2 encoder is taken as a node. The node contains the feature information of the image at that position or region. The feature information of each node is composed of its corresponding pixel value (and possible context information). The edge connects the nodes in the graph. The edge between nodes can be constructed by their similarity, that is, the relationship between the edges can be obtained by calculating the similarity between node features to obtain the graph structure. The similarity can be based on the distance metric of the feature vector (such as Euclidean distance, cosine similarity, etc.), or it can be based on the spatial proximity of the pixel position. For example, if two pixels are similar or close in color, texture or spatial position, there can be an edge between them. This setting allows GCN to effectively aggregate information. By aggregating the information of adjacent nodes, the feature representation of each node (i.e., each pixel) can be enhanced. This can usually be done in an iterative manner, in which each node updates its own features based on the features of its neighboring nodes. This aggregation helps capture local and global structural information in the image and improves the performance of the model.

[0079] In some embodiments, the update formula of GCN is:

[0080]

[0081] Among them, H (1) is the feature matrix of the first layer of GCN, is the degree matrix of the graph, whose diagonal elements are the degrees of the nodes. The power is used for normalization. A is the adjacency matrix of the graph, which represents the connection relationship between nodes. H (l-1) is the l-1 layer feature matrix, W (1) is the weight matrix of the lth layer, which is usually learned during the training process, and σ(·) is a nonlinear activation function used to introduce nonlinearity.

[0082] In step S103, the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

[0083] That is to say, after layer-by-layer decoding and feature recovery, the fused features of each pixel can be input into a preset classification network, which can classify the pixels based on the fused features and determine which semantic category in the remote sensing image the pixel belongs to.

[0084] As a possible implementation method, in some embodiments, a preset classification network is used to classify the fused features of each pixel to determine the semantic category of the remote sensing image corresponding to each pixel, including: using a classification head to map the fused features of each pixel into a category score vector of a fixed size through multiple fully connected layers; using a preset softmax function to process the category score vector to generate a probability distribution of each pixel belonging to each semantic category.

[0085] It can be understood that the classification head is a part of the network that converts the extracted features into the final category prediction. The fully connected layer is one of the basic building blocks in the neural network, where each neuron is connected to all neurons in the previous layer. The length of the category score vector is the preset number of semantic categories, and each element corresponds to the preliminary score of a category. The softmax function is a multi-classification function that can convert the category score vector into a probability distribution.

[0086] Specifically, in the classification head, multiple fully connected layers are stacked to gradually convert the fused features of each pixel into a low-dimensional category score vector equal to the number of semantic categories. The preset softmax function can map each element in the category score vector to the (0,1) interval and ensure that the sum of all elements is 1, thereby forming a valid probability distribution. Therefore, after being processed by the softmax function, each pixel can obtain a probability distribution belonging to each semantic category, and the category with the largest probability is selected as the semantic classification result of the corresponding pixel.

[0087] Among them, the probability distribution of each pixel belonging to each semantic category can be expressed as:

[0088]

[0089] Among them, x i represents the score of a pixel in the i-th category, n represents the total number of categories, and softmax transforms these scores exponentially and then divides them by the sum of the exponentials of all category scores. Normalization is performed to ensure that the sum of the class probabilities of each pixel is 1.

[0090] Finally, the network can be trained using the cross entropy loss function, which quantifies the difference between the actual class labels and the model predictions. The cross entropy loss function can be expressed as:

[0091]

[0092] Where N represents the total number of samples (i.e. the total number of image pixels), y ic is the true category label of the i-th sample, and p icis the probability that the model predicts that the sample belongs to the cth class, M represents the total number of possible categories, and by minimizing the cross entropy loss, the network can be effectively trained to more accurately classify each pixel.

[0093] Through the above process, the embodiment of the present application can effectively process and segment remote sensing images of various scales and complexities, provide accurate object classification and semantic information, and greatly enhance the application value and practicality of remote sensing images. This method of combining the Mamba2 model, the graph convolutional neural network GCN and the convolutional decoder (CNN decoder) not only takes advantage of the high efficiency of CNN in processing image details, but also overcomes its limitations in capturing long-range dependencies, significantly improving processing efficiency and the ability to understand the features of remote sensing images of different scales and complexity.

[0094] To facilitate those skilled in the art to further understand the semantic segmentation method of remote sensing images proposed in the embodiment of the present application, Figure 3 For further elaboration.

[0095] like Figure 3 As shown, the semantic segmentation method of the remote sensing image may further include the following steps:

[0096] Step S301, obtaining a remote sensing image dataset.

[0097] Step S302: preprocessing the remote sensing image data set.

[0098] Step S303, constructing a Mamba2 encoder downsampling module, using the Mamba2 encoder to downsample the preprocessed remote sensing image dataset to obtain downsampled features of different scales.

[0099] Step S304a, constructing a GCN multi-scale information modeling module, based on the graph structure, processing the down-sampled features of different scales through GCN to obtain updated down-sampled features of different scales.

[0100] Step S304b, constructing a CNN decoder upsampling module, and using the CNN decoder to upsample the downsampled features of the last stage obtained by multi-stage downsampling to obtain upsampled features.

[0101] Step S305, construct a multi-scale feature fusion module, and use the multi-scale feature fusion module to fuse the updated down-sampled features of different scales with the up-sampled features of the corresponding scales to obtain the fused features of each pixel.

[0102] Step S306, constructing a classification module, and inputting the fused features of each pixel into the classification module for classification.

[0103] Step S307: output the semantic segmentation result.

[0104] Therefore, the embodiment of the present application adopts the UPerNet structure as a whole, using symmetrical encoder and decoder paths and jump connections to retain the information lost during the downsampling process. Among them, in the first stage, the Mamba2 model is used as an encoder to downsample the image. The Mamba2 model combines the local feature extraction capability of CNN with the global dependency capture advantage of the Transformer model. Through this combination, the processing efficiency and the accuracy of the segmentation effect are improved. In the second stage, GCN is used to perform in-depth relationship modeling on the different scale features output by the encoder to effectively capture the complex spatial topological relationship between features. In the third stage, the CNN model is used to upsample the image, and the original resolution of the image is restored by the interpolation algorithm. Finally, each pixel is accurately classified through the classification network, thereby achieving efficient semantic segmentation of remote sensing images. By combining advanced models and technologies, this application greatly improves the ability to process complex contextual information and spatial topological structures between features of remote sensing images, making the semantic segmentation of remote sensing images more accurate, smooth and comprehensive.

[0105] According to the semantic segmentation method of remote sensing images proposed in the embodiment of the present application, by preprocessing the acquired remote sensing image data set, and using a preset model framework to extract and fuse the preprocessed remote sensing image data set, the fused features of each pixel can be obtained, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; and then the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced model Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, solving the problem that the existing technology has too high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0106] Next, the semantic segmentation device for remote sensing images proposed in accordance with the embodiments of the present application will be described with reference to the accompanying drawings.

[0107] Figure 4 It is a block diagram of a semantic segmentation device for remote sensing images according to an embodiment of the present application.

[0108] like Figure 4 As shown, the remote sensing image semantic segmentation device 10 includes: an acquisition module 100, a processing module 200 and a classification module 300.

[0109] Wherein, the acquisition module 100 is used to acquire a remote sensing image data set;

[0110] The processing module 200 is used to preprocess the remote sensing image data set, and use a preset model framework to extract and fuse features of the preprocessed remote sensing image data set to obtain fused features of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder;

[0111] The classification module 300 is used to classify the fused features of each pixel using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

[0112] According to one embodiment of the present application, the processing module 200 includes:

[0113] The first sampling unit is used to input the preprocessed remote sensing image data set into the Mamba2 encoder, and perform multi-level downsampling operations on the preprocessed remote sensing image data set using filters of different sizes and step sizes to obtain downsampling features of different scales;

[0114] The second sampling unit is used to input the down-sampled features that meet the preset spatial resolution condition among the down-sampled features of different scales into the CNN decoder, and perform an up-sampling operation on the down-sampled features that meet the preset spatial resolution condition by using a bilinear interpolation algorithm to obtain up-sampled features;

[0115] The fusion unit is used to process the downsampled features of different scales to obtain a graph structure. Based on the graph structure, the graph convolutional neural network GCN is used to update the downsampled features of different scales. The updated downsampled features of different scales are fused with the upsampled features of the corresponding scales through the feature fusion layer to obtain the fused features of each pixel.

[0116] According to one embodiment of the present application, the fusion unit is specifically used for:

[0117] Each pixel is regarded as a node, the relationship between edges is determined by calculating the similarity between nodes, and the graph structure is obtained based on the relationship between edges.

[0118] According to one embodiment of the present application, the update formula of GCN is:

[0119]

[0120] Among them, H (1) is the feature matrix of the first layer of GCN, is the degree matrix of the graph, A is the adjacency matrix of the graph, H (l-1) is the l-1 layer feature matrix, W (1) is the weight matrix of the lth layer, and σ(·) is the nonlinear activation function.

[0121] According to one embodiment of the present application, the classification module 300 is specifically used for:

[0122] The classification head is used to map the fused features of each pixel into a fixed-size category score vector through multiple fully connected layers;

[0123] The category score vector is processed using the preset softmax function to generate the probability distribution of each pixel belonging to each semantic category.

[0124] It should be noted that the above explanation of the embodiment of the remote sensing image semantic segmentation method is also applicable to the remote sensing image semantic segmentation device of this embodiment, and will not be repeated here.

[0125] According to the semantic segmentation device of remote sensing images proposed in the embodiment of the present application, by preprocessing the acquired remote sensing image data set, and using the preset model framework to extract and fuse the features of the preprocessed remote sensing image data set, the fused features of each pixel can be obtained, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; and then the fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel. Therefore, by combining the advanced model Mamba2 model and the convolutional neural network CNN, remote sensing images of various scales and complexities can be effectively processed and segmented, solving the problem that the existing technology has too high computational cost when processing large-size images and is limited by insufficient training data and generalization ability when processing complex dependencies, thereby providing more accurate classification of objects and semantic information.

[0126] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0127] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0128] When the processor 502 executes the program, the semantic segmentation method of the remote sensing image provided in the above embodiment is implemented.

[0129] Furthermore, the electronic device further comprises:

[0130] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0131] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0132] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0133] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0134] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0135] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0136] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned semantic segmentation method of remote sensing images is implemented.

[0137] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed, it implements the above-mentioned semantic segmentation method of remote sensing images.

[0138] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0140] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A semantic segmentation method for remote sensing images, characterized in that: The following steps are involved: Acquire remote sensing image datasets; Preprocessing the remote sensing image data set, and performing feature extraction and fusion on the preprocessed remote sensing image data set using a preset model framework to obtain fused features of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; The fused features of each pixel are classified using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

2. The method according to claim 1, characterized in that The method uses a preset model framework to extract and fuse features of the preprocessed remote sensing image data set to obtain the fused features of each pixel, including: Inputting the preprocessed remote sensing image data set into the Mamba2 encoder, performing multi-level downsampling operations on the preprocessed remote sensing image data set using filters of different sizes and step sizes, and obtaining downsampling features of different scales; Inputting the down-sampled features that meet the preset spatial resolution condition among the down-sampled features of different scales into the CNN decoder, and performing an up-sampling operation on the down-sampled features that meet the preset spatial resolution condition by using a bilinear interpolation algorithm to obtain up-sampled features; The down-sampled features of different scales are processed to obtain a graph structure. Based on the graph structure, the down-sampled features of different scales are updated by using a graph convolutional neural network (GCN). The updated down-sampled features of different scales are fused with the up-sampled features of corresponding scales through a feature fusion layer to obtain the fused features of each pixel.

3. The method according to claim 2, characterized in that The processing of the down-sampled features of different scales to obtain a graph structure includes: Each pixel is taken as a node, the relationship between edges is determined by calculating the similarity between nodes, and the graph structure is obtained based on the relationship between the edges.

4. The method according to claim 3, characterized in that The update formula of the GCN is: Among them, H (1) is the feature matrix of the first layer of GCN, is the degree matrix of the graph, A is the adjacency matrix of the graph, H (l-1) is the l-1 layer feature matrix, W (1) is the weight matrix of the lth layer, and σ(·) is the nonlinear activation function.

5. The method according to claim 1, characterized in that The method of using the preset classification network to classify the fused features of each pixel to determine the semantic category of the remote sensing image corresponding to each pixel includes: Using a classification head to map the fused features of each pixel into a fixed-size category score vector through multiple fully connected layers; The category score vector is processed using a preset softmax function to generate a probability distribution of each pixel belonging to each semantic category.

6. A semantic segmentation device for remote sensing images, characterized in that: include: An acquisition module is used to acquire remote sensing image data sets; A processing module, used for preprocessing the remote sensing image data set, extracting and fusing features of the preprocessed remote sensing image data set using a preset model framework to obtain fused features of each pixel, wherein the preset model framework includes a Mamba2 encoder and a CNN decoder; The classification module is used to classify the fused features of each pixel using a preset classification network to determine the semantic category of the remote sensing image corresponding to each pixel.

7. The device according to claim 6, characterized in that The processing module comprises: A first sampling unit is used to input the preprocessed remote sensing image data set into the Mamba2 encoder, and perform a multi-level downsampling operation on the preprocessed remote sensing image data set using filters of different sizes and step sizes to obtain downsampling features of different scales; A second sampling unit is used to input the down-sampled features that meet the preset spatial resolution condition among the down-sampled features of different scales into the CNN decoder, and perform an up-sampling operation on the down-sampled features using a bilinear interpolation algorithm to obtain up-sampled features; A fusion unit is used to process the down-sampled features of different scales using a graph convolutional neural network (GCN) to obtain a graph structure, update the down-sampled features of different scales based on the graph structure, and fuse the updated down-sampled features of different scales with the up-sampled features of the corresponding scales through a feature fusion layer to obtain the fused features of each pixel.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the semantic segmentation method for remote sensing images as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the semantic segmentation method of remote sensing images as described in any one of claims 1 to 5.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, is used to implement the semantic segmentation method of remote sensing images as described in any one of claims 1 to 5.

Images