Remote sensing image semantic segmentation method and system based on multi-scale cross fusion

Abstract

The invention provides a remote sensing image semantic segmentation method and system based on multi-scale cross fusion, and relates to the technical field of remote sensing image processing. According to the method, after a high-resolution remote sensing image is obtained and preprocessed, a detail coding path is input for multi-level semantic feature extraction to obtain detail coding features, then the detail coding features are input into a multi-scale decoding path, and decoding features of each layer are obtained; in the multi-scale decoding path, performing cross fusion on encoding features output by an encoder and output features of a corresponding multi-branch decoding module through double-branch cross attention to obtain enhanced fusion features of each layer, and inputting the enhanced fusion features into a decoder; and inputting the decoding features of the last layer into a classifier to output a pixel-level semantic segmentation result. According to the method, the problems of large ground feature scale difference, complex background and feature redundancy are effectively solved, the remote sensing image segmentation precision and robustness are remarkably improved, and the method has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically, to a method and system for semantic segmentation of remote sensing images based on multi-scale cross-fusion. Background Technology

[0002] With the rapid development of remote sensing technology, high-resolution remote sensing images are playing an increasingly important role in urban planning, environmental monitoring, and agricultural inspection. These images provide rich and detailed information on ground features, laying a solid technical foundation for intelligent applications. However, semantic segmentation of high-resolution remote sensing images faces many challenges, mainly due to large differences in ground feature scale, complex backgrounds, and the similarity of features between categories. Traditional semantic segmentation methods for remote sensing images are mostly based on convolutional neural networks, such as fully convolutional networks and U-Net. Although these methods have achieved success to some extent, they still have significant shortcomings.

[0003] First, remote sensing images contain a wide range of ground features, including buildings, roads, vegetation, and vehicles, with varying sizes. Traditional convolutional neural networks typically use a fixed receptive field, making it difficult to effectively handle targets of different sizes, resulting in low accuracy in recognizing small targets and boundary regions. Second, in encoder-decoder structures, the decoding process often involves the loss of low-level details, especially boundary information and small target features in high-resolution images, which are not adequately preserved, thus affecting segmentation accuracy. Furthermore, the backgrounds of remote sensing images are usually complex and varied, containing multiple land cover categories. Traditional methods are easily affected by background interference during semantic segmentation, leading to blurred or erroneous segmentation results.

[0004] In recent years, some new methods have attempted to address these issues by introducing multi-scale modeling techniques and attention mechanisms. For example, the DeepLab series of methods extracts contextual information at different scales using a hollow spatial pyramid pooling module, improving the model's adaptability to changes in target size. However, these methods typically rely on single-pass multi-scale feature extraction at the encoder end, lacking continuous feature enhancement during the decoding stage, which can easily lead to information loss. Meanwhile, attention mechanisms are widely used in semantic segmentation tasks, enhancing salient regions and suppressing redundant information through positional and channel attention, thereby improving global context modeling capabilities. However, most existing attention mechanism methods are limited to modeling single-path features, failing to effectively combine the complementary information of the encoder and decoder, resulting in redundant information not being effectively suppressed when processing features at different scales.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention aims to provide a method and system for semantic segmentation of remote sensing images based on multi-scale cross-fusion, in order to solve the problems of large differences in the scale of ground objects, complex backgrounds, and insufficient feature fusion between encoders and decoders in existing methods for semantic segmentation of high-resolution remote sensing images.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: A remote sensing image semantic segmentation method based on multi-scale cross-fusion includes: S1, acquire high-resolution remote sensing images and preprocess them; S2, input the preprocessed high-resolution remote sensing image into the detail coding path constructed by N encoders and perform multi-level semantic feature extraction to obtain the detail coding features output by the Nth encoder. S3, input the detail coding features into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom layer decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detail coding path; S4, the bottom-level decoding features and the encoding features output by the N-1 layer encoder are cross-fused through dual-branch cross attention and then input into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module and then cross-fused with the encoding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder. S5, the decoded features output from the last layer decoder are input into the classifier for classification prediction, and the output is pixel-level semantic segmentation results.

[0008] Preferably, the preprocessing includes normalization and data augmentation; Normalization maps the pixel values ​​of the high-resolution remote sensing image to a set range. The data enhancement includes randomly adjusting the brightness, contrast, and saturation of the image, rotating it, flipping it, or adding Gaussian noise.

[0009] Preferably, the detail encoding path is based on the EfficientNetV2 backbone network, and extracts feature maps of different scales of the input image step by step through several encoders, namely detail encoding features, to capture semantic information from low level to high level; each encoder consists of convolutional blocks, activation layers and pooling layers. The convolutional blocks extract local spatial features and channel semantic information, and then the nonlinear transformation of the activation layers enhances the expressive power of the features. Finally, the pooling layers compress the dimensionality and output the encoded features of each layer as the input of the next layer encoder.

[0010] Preferably, each of the multi-branch decoding modules includes parallel branches of multiple convolutional blocks of different sizes and a 1×1 convolutional layer, used to perform multi-scale feature extraction, splicing and fusion and upsampling on the detail coding features to obtain multi-scale fused features; Each decoder contains upsampling and convolution to map input features into low-dimensional decoded features; the input features include multi-scale fusion features output by the multi-branch decoding module corresponding to the Nth layer encoder, or enhanced fusion features resulting from the cross-fusion of multi-scale fusion features of other layers and the corresponding layer's encoded features through bi-branch cross-attention.

[0011] Preferably, each convolutional block includes a convolutional layer, a batch normalization layer, and an activation layer; The convolutional layer is used to extract local features from the input data; the batch normalization layer is used to standardize the input features; and the activation layer enhances the network's ability to express complex features through nonlinear transformations.

[0012] Preferably, the dual-branch cross attention includes a channel attention branch and a spatial attention branch; The channel attention branch includes a pooling layer, a convolutional layer, and an activation layer, which are used to calculate channel attention weights for the encoded features output by the encoder and the multi-scale fusion features output by the corresponding multi-branch decoding module, and then perform weighted fusion with the corresponding encoded features and multi-scale fusion features to obtain channel enhancement features. The spatial attention branch includes a pooling layer, a convolutional layer, an activation layer, and a normalization layer, which are used to calculate the spatial attention weights of the channel enhancement features, and after normalization, they are weighted and fused with the features to obtain the enhanced fusion features. The enhanced fusion features of each layer are input into the decoder for upsampling and feature fusion to obtain the decoded features of the current layer, and then input into the next layer for attention cross-fusion and decoding, until the last layer.

[0013] Preferably, the calculation process of the channel attention branch is as follows: The encoded features output by each encoder are processed by global max pooling and global average pooling to generate two channel descriptions. The two channels are expanded by the first convolutional layer, then activated by the ReLU function and then restored by the second convolutional layer, to obtain the attention weights of the first channel and the attention weights of the second channel respectively. The attention weights of the first and second channels of each encoder are added together to obtain the attention output of the encoded channel. Similarly, the features output by each multi-branch decoding module are input into the channel attention branch to obtain the decoding channel attention output; The channel attention matrix is ​​obtained by multiplying and fusing the attention output of the encoding channel and the attention output of the decoding channel. The channel attention matrix is ​​weighted and fused with the corresponding coding features and the features output by the multi-branch decoding module to obtain channel-enhanced coding features and channel-enhanced decoding features.

[0014] Preferably, the calculation process of the spatial attention branch is as follows: The channel-enhanced coding features are combined after being subjected to max pooling and average pooling operations respectively to obtain a spatial description. The spatial description is passed through the ReLU activation function of the first convolutional layer and the activation layer, and then input into the second convolutional layer to obtain the spatial attention output. Softmax normalization is performed on the spatial attention output to obtain the encoding space enhancement weights; Similarly, the channel-enhanced decoding features are input into the spatial attention branch to obtain the decoding spatial enhancement weights; The cross matrix is ​​obtained by multiplying the encoding space enhancement weights and the decoding space enhancement weights. By performing element-wise weighted fusion of the cross matrix and the corresponding encoded features, the output space is enhanced with encoded features. The output space is enhanced by element-wise weighted fusion of the features output by the cross matrix and the corresponding multi-branch decoding module. The spatially enhanced coding features and the spatially enhanced decoding features are added element-wise and fused to obtain the enhanced fused features.

[0015] Preferably, the decoded features output from the last decoder are input into the classifier for classification prediction, outputting pixel-level semantic segmentation results, specifically: The enhanced fusion features of the last layer are upsampled using bilinear interpolation to restore them to the target spatial size, resulting in the reconstructed image. The reconstructed image is used to perform semantic classification prediction through a Softmax classifier, and the pixel-level semantic segmentation results of the land cover categories in the image are output.

[0016] This invention also provides a remote sensing image semantic segmentation system based on multi-scale cross-fusion, comprising: The data acquisition unit is used to acquire and preprocess high-resolution remote sensing images. The detail coding unit is used to input the preprocessed high-resolution remote sensing image into the detail coding path constructed by N encoders, and perform multi-level semantic feature extraction to obtain the detail coding features output by the Nth encoder. The bottom-level decoding unit is used to input the detail coding features into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom-level decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detail coding path; The multi-scale decoding unit is used to cross-fuse the bottom-level decoding features and the coding features output by the N-1 layer encoder through dual-branch cross-attention, and then input them into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module, and then cross-fuse them with the coding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder. The prediction and classification unit is used to input the decoded features output from the last layer decoder into the classifier for classification and prediction, and outputs pixel-level semantic segmentation results.

[0017] The present invention also provides a remote sensing image semantic segmentation device based on multi-scale cross-fusion, including a processor and a memory. The memory stores a computer program that can be executed by the processor to implement the remote sensing image semantic segmentation method based on multi-scale cross-fusion as described above.

[0018] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement the remote sensing image semantic segmentation method based on multi-scale cross-fusion as described above.

[0019] In summary, compared with the prior art, the present invention has the following beneficial effects: This invention addresses the common information loss problem in the decoding stage of traditional methods by combining detail encoding and multi-scale decoding paths, particularly demonstrating significant improvement in detail recovery for boundary regions. Simultaneously, the dual-branch cross-attention fusion module, through a combination of channel and spatial attention mechanisms, adaptively filters redundant information, effectively enhancing the expression of effective features and avoiding interference from excessive redundant features on segmentation accuracy. Furthermore, data augmentation operations improve the model's robustness, further enhancing the accuracy and detail restoration capabilities of remote sensing image segmentation.

[0020] This invention introduces a multi-scale feature extraction and a dual-branch cross-attention fusion module, which dynamically integrates the multi-scale features of the encoder and decoder to achieve deep fusion of semantic information and spatial details. This effectively alleviates the segmentation challenges caused by differences in ground object scale and background complexity, solves key problems in semantic segmentation of remote sensing images, significantly improves segmentation accuracy and robustness, and has broad application prospects. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a remote sensing image semantic segmentation method based on multi-scale cross-fusion, as provided in Example 1.

[0023] Figure 2 This is a schematic diagram of the structure of a remote sensing image semantic segmentation method based on multi-scale cross-fusion provided in Example 1.

[0024] Figure 3 This is a schematic diagram of the structure of the multi-branch decoding module (MDM) provided in Embodiment 1.

[0025] Figure 4 This is a schematic diagram of the dual-branch cross-attention fusion module DAFM provided in Example 1.

[0026] Figure 5 This is a schematic diagram of a remote sensing image semantic segmentation device based on multi-scale cross-fusion, as provided in Embodiment 2.

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] Example 1 Embodiment 1 of the present invention provides a remote sensing image semantic segmentation method based on multi-scale cross-fusion, which can be implemented by a remote sensing image semantic segmentation device based on multi-scale cross-fusion (hereinafter referred to as semantic segmentation device), specifically, executed by one or more processors within the semantic segmentation device.

[0030] In this embodiment, the semantic segmentation device may be an electronic device equipped with a processor, which carries a computer program for the remote sensing image semantic segmentation method based on multi-scale cross-fusion and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0031] In the semantic segmentation of high-resolution remote sensing images, ground objects range in size, including buildings, roads, vegetation, and vehicles. Traditional convolutional neural networks, which use fixed receptive fields for feature extraction, struggle to adapt to targets of different scales, leading to reduced recognition accuracy for small targets and boundary regions. Furthermore, in encoder-decoder structures, the decoding process involves the loss of low-level detail information, particularly boundary information and fine target features in high-resolution images. In addition, the complex and varied backgrounds of remote sensing images, containing multiple ground object categories, make traditional methods susceptible to background interference during segmentation, resulting in blurred or erroneous segmentation results and impacting the critical performance indicator of segmentation accuracy.

[0032] For example, in remote sensing image processing scenarios in urban areas, images contain dense clusters of buildings, narrow road networks, and scattered vehicle targets. Due to the significant scale differences between buildings and roads, and the fact that vehicles, as small targets, occupy relatively few pixels in the image, traditional methods struggle to accurately identify vehicle outlines. Simultaneously, the urban background contains numerous vegetation and shadow areas with similar textures, leading to discontinuous road boundary segmentation and missegmentation. Consequently, vehicle targets are often missed or confused with the background in the segmentation results, and road boundaries become blurred, directly impacting the accuracy of urban traffic planning and infrastructure management.

[0033] If the aforementioned technical problems are not resolved, the accuracy defects in semantic segmentation of remote sensing images will lead to a decrease in the reliability of decision-making systems based on the segmentation results, making applications such as urban planning and environmental monitoring face the risk of misjudgment, and seriously hindering the effective deployment of remote sensing technology in real-world scenarios.

[0034] Therefore, as Figure 1 As shown, this embodiment provides a remote sensing image semantic segmentation method based on multi-scale cross-fusion, which includes steps S1 to S5.

[0035] S1: Acquire high-resolution remote sensing images and preprocess them.

[0036] like Figure 2As shown, the overall structure of the method of this invention includes a detail encoding path, a multi-scale decoding path, and a dual-branch cross-attention fusion module. The entire process sequentially performs image data preprocessing, feature extraction, feature fusion, and classification prediction to achieve accurate segmentation of various land cover categories in remote sensing images.

[0037] In an embodiment of the present invention, the input is a high-resolution remote sensing image, and the output is a pixel-level semantic segmentation result.

[0038] First, the image data is preprocessed. This preprocessing includes normalization and data augmentation. Normalization maps the pixel values ​​of the high-resolution remote sensing image to a set range. Data augmentation includes random brightness adjustment, contrast adjustment, saturation adjustment, rotation, flipping, or adding Gaussian noise to generate diverse training samples and reduce the risk of overfitting, thereby improving the model's robustness and generalization ability.

[0039] S2, the preprocessed high-resolution remote sensing image is input into the detail coding path constructed by N encoders to extract multi-level semantic features, and the detail coding features output by the Nth encoder are obtained.

[0040] Specifically, the detail encoding path is based on the EfficientNetV2 backbone network. It extracts feature maps of different scales from the input image, i.e. detail encoding features, through several encoders to capture semantic information from low to high levels. Each encoder consists of a convolutional block, an activation layer, and a pooling layer. The convolutional block extracts local spatial features and channel semantic information. Then, the nonlinear transformation of the activation layer enhances the expressive power of the features. Finally, the pooling layer compresses the dimension and outputs the encoded features at the corresponding scale, which are used as the input to the next encoder layer. This process continues until the last encoder layer, which yields the detail encoding features.

[0041] For example, such as Figure 2 As shown, the input feature map is 3*512*512. Each layer's feature map is processed through 5 layers of convolutional layers, activation layers, and pooling layers to generate a series of encoded feature sets {F1, F2, F3, F4, F5} at different levels. These features provide rich information for subsequent decoding and fusion processes. By setting appropriate convolutional block sizes, encoded feature sets of different scales are generated. For example, F1-F5 have sizes of 48*128*128, 64*64*64, 128*32*32, 160*32*32, and 256*16*16, corresponding to the outputs of encoders 1-5, respectively. Experiments show that these five different sized feature maps contain information about ground features at various scales. Too large a size leads to the extraction of more redundant information, while too small a size results in overly cluttered extracted information, thus reducing segmentation accuracy.

[0042] In this embodiment, EfficientNetV2 is a lightweight and efficient convolutional neural network (CNN) backbone proposed by Google in 2021. Its core advantage is that it significantly improves training speed and inference efficiency while maintaining high accuracy. Through joint optimization of network depth, width, and resolution and structural innovation, it solves the pain points of traditional CNNs (such as EfficientNetV1 and ResNet) in terms of training cost and memory usage. It is widely used in computer vision tasks such as image classification, semantic segmentation, and object detection.

[0043] In this embodiment, each convolutional block includes a convolutional layer, a batch normalization layer, and an activation layer. The convolutional layer extracts local features from the input data; the batch normalization layer standardizes the input features; and the activation layer enhances the network's ability to express complex features through nonlinear transformations.

[0044] The input image is used to extract multi-level semantic and detail features through detail coding paths, providing rich representation information for subsequent fusion.

[0045] S3, input the detail coding features into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom layer decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detail coding path.

[0046] The multi-scale decoding path consists of several parallel multi-branch decoding modules (MDMs) and decoders, corresponding to the encoder of the detail coding path. Each MDM contains parallel branches of multiple convolutional blocks of different sizes and receptive fields, as well as 1×1 convolutional layers, used for multi-scale feature extraction and concatenation fusion of encoded features to obtain multi-scale fused features, thereby enhancing the model's ability to perceive ground features at different scales. The 1×1 convolutional layers perform linear combination of feature channels, aiming to effectively integrate the features extracted from different scale branches and generate multi-scale fused features with hierarchical semantic expression.

[0047] For example, each multi-branch decoding module (MDM) can contain three parallel convolutional blocks, such as... Figure 3 As shown, each convolutional block contains a convolutional layer, a batch normalization layer, and an activation layer (such as the SiLU activation function).

[0048] Assuming the three parallel convolutional blocks have kernels of 3×3, 5×5, and 7×7 respectively, and the input feature is X, the calculation of parallel convolution can be expressed as: ; ; ; in, For the output of each convolutional block branch, such as The output of the 3×3 convolution block branch. Indicates the kernel size as The convolution operation, BN is the batch normalization operation. is the activation function. The feature map extracted by a 3×3 convolutional kernel contains small-scale ground feature information, such as cars and single trees. A 5×5 convolutional kernel is better at identifying slightly larger ground features, such as houses. A 7×7 convolutional kernel can efficiently extract large-area ground features, such as large grasslands and large lakes.

[0049] After obtaining information at different scales, it is necessary to stitch and fuse them. These features at different scales are stitched and fused through the channel dimension to obtain multi-scale fused features.

[0050] Specifically, when fusing these spliced ​​features at different scales, a 1×1 convolutional layer can be used for fusion, as expressed in the following expression: ; ; in, Features concatenated from each multi-branch decoding module; The features are spliced ​​and fused together for each multi-branch decoding module.

[0051] The fused feature map Upsampling is performed using bilinear interpolation to adjust the sample size to the target size for subsequent decoder needs. The formula is as follows: ; in, The final output features of each multi-branch decoding module are the multi-scale fusion features of the current level. This is an upsampling operation.

[0052] Each decoder includes upsampling and convolution. Except for the decoder corresponding to the Nth layer encoder, which directly upsamples and decodes the multi-scale fusion features output by the current layer multi-branch decoding module and inputs them into the N-1 layer multi-branch decoding module DAFM, the decoders of the other layers are used to map the features (i.e., the enhanced fusion features of the current layer) after the multi-scale fusion features of the current layer and the encoded features of the corresponding layer are cross-fused by two-branch cross-attention to the low-dimensional decoded features, gradually restoring the spatial resolution and realizing the integration of ground feature information at different scales.

[0053] This application utilizes a parallel structure for multi-scale decoding paths, enabling the encoded features at each scale to be independently input into the corresponding multi-branch decoding module. Within the multi-branch decoding module, convolutional blocks of different sizes are processed synchronously in parallel branches. Small-sized convolutional kernel branches focus on extracting local spatial details of boundaries and small features, while large-sized convolutional kernel branches capture contextual semantic information. Subsequently, a 1×1 convolutional layer is used to concatenate and fuse the features output from each branch along the channel dimension, generating multi-scale fused features that retain the complete information flow from low-level edges to high-level semantics. The decoder then uses upsampling operations to gradually restore the spatial resolution of the feature map, mapping the high-dimensional multi-scale fused features into low-dimensional decoding features, thereby ensuring that the key information in the detailed encoded features is fully preserved and utilized during the decoding stage.

[0054] S4, the bottom-level decoding features and the encoding features output by the N-1 layer encoder are cross-fused through dual-branch cross attention and then input into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module and then cross-fused with the encoding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder.

[0055] In remote sensing semantic segmentation, noise can occur during the decoding process due to factors such as complex backgrounds and changes in illumination. To address this, this paper designs a dual-branch cross-attention fusion module, which filters out redundant information by retaining features that are of interest to both the encoder and decoder.

[0056] Specifically, the (N-1)th layer's dual-branch cross-attention fusion module takes the encoded features output by the encoder of the (N-1)th layer and the decoded features of the Nth layer as input. The dual-branch cross-attention fusion modules at other layers take the encoded features output by the encoder of the corresponding layer and the features output by the multi-branch decoding module of the corresponding layer as input. The input features are adaptively weighted using channel attention and spatial attention mechanisms, respectively. This module utilizes the dual-branch cross-attention mechanism to establish an efficient feature interaction relationship between details and semantic information, automatically identifying and strengthening key features while suppressing redundant or noisy information. This achieves adaptive optimization of feature selection, thereby improving the accuracy and boundary integrity of the segmentation results.

[0057] Specifically, the dual-branch cross attention includes a channel attention branch and a spatial attention branch.

[0058] The channel attention branch includes a pooling layer, a convolutional layer, and an activation layer, which calculate channel attention weights for the input features and then perform weighted fusion with the corresponding input features (encoded features and multi-scale fusion features output by the multi-branch decoding module) to obtain channel-enhanced features.

[0059] The spatial attention branch includes a pooling layer, a convolutional layer, an activation layer, and a normalization layer. It calculates spatial attention weights for the channel enhancement features, and after normalization, it fuses them with the features in a weighted manner to obtain the enhanced fusion features.

[0060] like Figure 4 As shown, during the calculation of channel / spatial attention weights, each encoded feature (or the multi-scale fusion feature output by the multi-branch decoding module) passes through two paths, sequentially passing through an average pooling layer (the other path is a max pooling layer, and so on), a first convolutional layer (e.g., a 1×1 convolution), a ReLU activation layer, and a second convolutional layer (e.g., a 1×1 convolution). The results of the two paths (attention weights) are then fused. The fused result is then weighted and fused with the features.

[0061] Specifically, the calculation process for the channel attention branch is as follows: The encoded features output by each encoder Channel descriptions are generated using global max pooling, such as channel descriptions with a shape of C×1×1; The channel descriptions are expanded by a first convolutional layer (e.g., a 1×1 convolution), then activated by a ReLU function before entering a second convolutional layer (e.g., a 1×1 convolution) for channel restoration, resulting in the first channel attention weights. The specific calculation method is as follows: ; in, Indicates the attention weight of the first channel; This represents a convolution operation with a kernel size of 1×1. For activation functions; This is for global max pooling.

[0062] Then encode features After performing global average pooling again, the second channel attention weights are obtained after passing through the first convolutional layer, ReLU activation, and the second convolutional layer. Then, with the first channel attention weight Add them together to obtain the attention output of the encoded channel. The formula is: .

[0063] Similarly, the features output by each multi-branch decoding module The input channel attention branch yields the decoded channel attention output. .

[0064] The channel attention matrix is ​​obtained by multiplying the attention output of the encoding channel by the attention output of the decoding channel. , It contains global information that is of common interest to both encoding and decoding.

[0065] The channel attention matrix is ​​respectively associated with the corresponding encoded features. Features output by the multi-branch decoding module (or decoding features output by the Nth layer decoder) Weighted fusion is performed to filter out interference and redundant information in the detail coding path and multi-scale decoding path, resulting in channel-enhanced coding features. Channel Enhancement Decoding Features The formula is: ; .

[0066] To further refine and A spatial attention branch is introduced. The calculation process of the spatial attention branch is as follows: Enhance the channel coding features Perform max pooling separately and average pooling operation After merging, a spatial description is obtained; The spatial description is input into the first convolutional layer (e.g., a 3×3 convolution), then activated by the ReLU function of the activation layer, and then input into the second convolutional layer (e.g., a 3×3 convolution) to obtain the spatial attention output. Softmax normalization is performed on the spatial attention output to obtain the encoding space augmentation weights. The expression is: ; in, This represents the merging of channel dimensions.

[0067] Similarly, the channel-enhanced decoding features are input into the spatial attention branch to obtain the decoding spatial enhancement weights. ; The cross matrix Weights are enhanced through the coding space. Enhanced weights with decoding space The expression obtained through matrix multiplication is: = ; By performing element-wise weighted fusion of the cross matrix and the corresponding encoded features, the output space is enhanced with encoded features. The expression is: ; Where T is the transpose symbol.

[0068] Through the cross matrix Features output by the corresponding multi-branch decoding module Perform element-wise weighted fusion to enhance the output space and decode features. The expression is: ; The spatial enhancement coding features and the spatial enhancement decoding features are added element-wise and fused to obtain the enhancement fusion features of each layer. The expression is: .

[0069] Enhanced fusion features of each layer The input decoder is upsampled to obtain the decoding features of the current layer.

[0070] S5, the decoded features output from the last layer decoder are input into the classifier for classification prediction, and the output is pixel-level semantic segmentation results.

[0071] The enhanced fusion features of each layer are upsampled and fused by the decoder to become the decoded features of the current layer, and then input into the next layer for attention cross-fusion and decoding, until the last layer. The decoded features of the last layer are input into the prediction module, which outputs pixel-level semantic results through the Softmax classifier, thus completing the accurate segmentation of different land features in the remote sensing image.

[0072] The enhanced fusion features of the last layer are upsampled and spatially restored before being input into the classifier for classification prediction. Specifically: The enhanced fusion features of the last layer are upsampled using bilinear interpolation to restore them to the target spatial size, resulting in the reconstructed image. The reconstructed image is used to perform semantic classification prediction through a Softmax classifier, and the pixel-level semantic segmentation result is output.

[0073] In a preferred embodiment, the model corresponding to the method of the present invention is trained and evaluated. During training, the AdamW optimizer is used with an initial learning rate of 6×10⁻⁶. -4 The weight decay was 0.01. During training, a cosine annealing learning rate scheduling strategy was used to dynamically adjust the learning rate to avoid oscillations during training. Random data augmentation was included in each training epoch to improve the model's robustness. For evaluation, multiple datasets such as ISPRS Vaihingen, Potsdam, and LoveDA were used, and mIoU (Mean Intersection over Union) and F1 score were used as evaluation metrics to verify the model's segmentation accuracy and robustness.

[0074] In summary, compared with the prior art, the present invention has the following beneficial effects: This invention addresses the common information loss problem in the decoding stage of traditional methods by combining detail encoding paths and multi-scale decoding paths, particularly demonstrating significant improvement in detail recovery for boundary regions. Furthermore, this invention employs a dual-branch cross-attention fusion module that combines channel and spatial attention mechanisms to adaptively filter redundant information, significantly enhancing the expressive power of effective features and avoiding interference from excessive redundant features on segmentation accuracy.

[0075] Furthermore, data augmentation during training improved the model's robustness and further enhanced the accuracy and detail restoration capabilities of remote sensing image segmentation.

[0076] In summary, this invention solves key problems in semantic segmentation of remote sensing images by introducing multi-scale feature extraction and bi-branch cross-attention fusion, significantly improving segmentation accuracy and robustness, and has broad application prospects.

[0077] Example 2 like Figure 5 As shown, the second embodiment of the present invention also provides a remote sensing image semantic segmentation system based on multi-scale cross-fusion, comprising: The data acquisition unit is used to acquire and preprocess high-resolution remote sensing images. The detail coding unit is used to input the preprocessed high-resolution remote sensing image into the detail coding path constructed by N encoders, and perform multi-level semantic feature extraction to obtain the detail coding features output by the Nth encoder. The bottom-level decoding unit is used to input the detail coding features into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom-level decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detail coding path; The multi-scale decoding unit is used to cross-fuse the bottom-level decoding features and the coding features output by the N-1 layer encoder through dual-branch cross-attention, and then input them into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module, and then cross-fuse them with the coding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder. The prediction and classification unit is used to input the decoded features output from the last layer decoder into the classifier for classification and prediction, and outputs pixel-level semantic segmentation results.

[0078] Example 3 The third embodiment of the present invention also provides a remote sensing image semantic segmentation device based on multi-scale cross-fusion, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the remote sensing image semantic segmentation method based on multi-scale cross-fusion as described above.

[0079] Example 4 The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, they implement the remote sensing image semantic segmentation method based on multi-scale cross-fusion as described above.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on multi-scale cross-fusion, characterized in that, include: Acquire high-resolution remote sensing images and preprocess them; The preprocessed high-resolution remote sensing image is input into the detail coding path constructed by N encoders to perform multi-level semantic feature extraction, and the detail coding features output by the Nth encoder are obtained. The detailed coding features are input into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom layer decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detailed coding path; The bottom-level decoding features and the encoding features output by the N-1 layer encoder are cross-fused through dual-branch cross attention and then input into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module and then cross-fused with the encoding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder. The decoded features output from the final decoder are input into the classifier for classification and prediction, and the output is a pixel-level semantic segmentation result.

2. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 1, characterized in that... The preprocessing includes normalization and data augmentation; Normalization maps the pixel values ​​of the high-resolution remote sensing image to a set range. The data enhancement includes randomly adjusting the brightness, contrast, and saturation of the image, rotating it, flipping it, or adding Gaussian noise.

3. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 1, characterized in that... The detailed encoding path is based on the EfficientNetV2 backbone network. It extracts feature maps of different scales from the input image step by step through several encoders, namely detailed encoding features, to capture semantic information from low level to high level. Each encoder consists of convolutional blocks, activation layers and pooling layers. The convolutional blocks extract local spatial features and channel semantic information. Then, the nonlinear transformation of the activation layers enhances the expressive power of the features. Finally, the pooling layers compress the dimensionality and output the encoded features of each layer as the input of the next encoder.

4. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 3, characterized in that... Each of the multi-branch decoding modules includes parallel branches of multiple convolutional blocks of different sizes and a 1×1 convolutional layer, used to perform multi-scale feature extraction, splicing and fusion and upsampling on the detail coding features to obtain multi-scale fused features; Each decoder contains upsampling and convolution to map input features into low-dimensional decoded features; the input features include multi-scale fusion features output by the multi-branch decoding module corresponding to the Nth layer encoder, or enhanced fusion features resulting from the cross-fusion of multi-scale fusion features of other layers and the corresponding layer's encoded features through bi-branch cross-attention.

5. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 4, characterized in that... Each convolutional block contains a convolutional layer, a batch normalization layer, and an activation layer; The convolutional layer is used to extract local features from the input data; the batch normalization layer is used to standardize the input features; and the activation layer enhances the network's ability to express complex features through nonlinear transformations.

6. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 4, characterized in that... The bi-branch cross attention includes a channel attention branch and a spatial attention branch; The channel attention branch includes a pooling layer, a convolutional layer, and an activation layer, which are used to calculate channel attention weights for the encoded features output by the encoder and the multi-scale fusion features output by the corresponding multi-branch decoding module, and then perform weighted fusion with the corresponding encoded features and multi-scale fusion features to obtain channel enhancement features. The spatial attention branch includes a pooling layer, a convolutional layer, an activation layer, and a normalization layer, which are used to calculate the spatial attention weights of the channel enhancement features, and after normalization, they are weighted and fused with the features to obtain the enhanced fusion features. The enhanced fusion features of each layer are input into the decoder for upsampling and feature fusion to obtain the decoded features of the current layer, and then input into the next layer for attention cross-fusion and decoding, until the last layer.

7. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 6, characterized in that... The calculation process for the channel attention branch is as follows: The encoded features output by each encoder are processed by global max pooling and global average pooling to generate two channel descriptions. The two channels are expanded by the first convolutional layer, then activated by the ReLU function and then restored by the second convolutional layer, to obtain the attention weights of the first channel and the attention weights of the second channel respectively. The attention weights of the first and second channels of each encoder are added together to obtain the attention output of the encoded channel. Similarly, the features output by each multi-branch decoding module are input into the channel attention branch to obtain the decoding channel attention output; The channel attention matrix is ​​obtained by multiplying and fusing the attention output of the encoding channel and the attention output of the decoding channel. The channel attention matrix is ​​weighted and fused with the corresponding coding features and the features output by the multi-branch decoding module to obtain channel-enhanced coding features and channel-enhanced decoding features.

8. The remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 7, characterized in that... The calculation process of the spatial attention branch is as follows: The channel-enhanced coding features are combined after being subjected to max pooling and average pooling operations respectively to obtain a spatial description. The spatial description is passed through the ReLU activation function of the first convolutional layer and the activation layer, and then input into the second convolutional layer to obtain the spatial attention output. Softmax normalization is performed on the spatial attention output to obtain the encoding space enhancement weights; Similarly, the channel-enhanced decoding features are input into the spatial attention branch to obtain the decoding spatial enhancement weights; The cross matrix is ​​obtained by multiplying the encoding space enhancement weights and the decoding space enhancement weights. By performing element-wise weighted fusion of the cross matrix and the corresponding encoded features, the output space is enhanced with encoded features. The output space is enhanced by element-wise weighted fusion of the features output by the cross matrix and the corresponding multi-branch decoding module. The spatially enhanced coding features and the spatially enhanced decoding features are added element-wise and fused to obtain the enhanced fused features.

9. A remote sensing image semantic segmentation method based on multi-scale cross-fusion according to claim 1, characterized in that... The decoded features output from the last layer decoder are input into the classifier for classification prediction, and the output is a pixel-level semantic segmentation result, specifically: The enhanced fusion features of the last layer are upsampled using bilinear interpolation to restore them to the target spatial size, resulting in the reconstructed image. The reconstructed image is used to perform semantic classification prediction through a Softmax classifier, and the pixel-level semantic segmentation results of the land cover categories in the image are output.

10. A remote sensing image semantic segmentation system based on multi-scale cross-fusion, characterized in that, include: The data acquisition unit is used to acquire and preprocess high-resolution remote sensing images. The detail coding unit is used to input the preprocessed high-resolution remote sensing image into the detail coding path constructed by N encoders, and perform multi-level semantic feature extraction to obtain the detail coding features output by the Nth encoder. The bottom-level decoding unit is used to input the detail coding features into the Nth layer multi-branch decoding module and decoder of the multi-scale decoding path to obtain the bottom-level decoding features; wherein, the multi-scale decoding path is composed of N parallel multi-branch decoding modules and decoders corresponding to the detail coding path; The multi-scale decoding unit is used to cross-fuse the bottom-level decoding features and the coding features output by the N-1 layer encoder through dual-branch cross-attention, and then input them into the corresponding decoder for decoding. The resulting decoding features are input into the N-1 layer multi-branch decoding module, and then cross-fuse them with the coding features output by the N-2 layer encoder and decoded by the decoder, until the last layer decoder. The prediction and classification unit is used to input the decoded features output from the last layer decoder into the classifier for classification and prediction, and outputs pixel-level semantic segmentation results.

Images