Remote sensing image semantic segmentation method based on double-branch multi-scale fusion network

Abstract

The invention discloses a remote sensing image semantic segmentation method based on a double-branch multi-scale fusion network, and relates to the technical field of remote sensing image semantic segmentation. According to the method, a double-encoder and single-decoder structure, a multi-scale feature fusion channel and spatial path information are adopted; an adaptive efficient parameter fine tuning SAM2 large model knowledge is utilized to accelerate new task learning, global feature and local feature information of the SAM2 large model knowledge is effectively captured through a global local window multi-head self-attention module, a mixed Logit adjustment loss function based on mutual information is adopted to adjust category weight aiming at the problem of sample category imbalance of a traditional remote sensing data set, and the new task learning is accelerated. The remote sensing image semantic segmentation effect is improved; according to the method, pixel classification and ground feature correspondence are realized, the segmentation problems of complex background, unbalanced sample category and large scale change are effectively solved, the accuracy and stability of image segmentation are improved, and rapid and accurate segmentation of the remote sensing image is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image semantic segmentation, and in particular to a remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network. Background Art

[0002] Semantic segmentation of remote sensing images has become an important part of modern geographic information systems and is widely used in urban planning, environmental monitoring, disaster management and other fields. With the development of remote sensing data acquisition technology, image resolution and acquisition frequency continue to improve, which provides us with rich geographic information. However, how to extract useful information from these massive remote sensing images, especially to effectively classify and analyze the objects in the images, is still a challenging task. As an important image processing technology, semantic segmentation can mark each pixel in a remote sensing image as a specific object category, thereby providing detailed object information.

[0003] Compared with natural images, remote sensing images have the characteristics of complex scenes and large scale variations of objects. In the current remote sensing segmentation model, CNN can extract image features through stacked convolution operations, but the locality of convolution operations makes it impossible to directly obtain global context information. Transformer can model long-distance dependencies based on the self-attention mechanism, but the model has low universality due to problems such as low efficiency and large memory usage. The segmented arbitrary model is known as an excellent pre-training model for image segmentation due to its ultra-large training set and powerful zero-sample segmentation capability. Since the segmented arbitrary model is trained on a large-scale general image dataset, the difference between the two will reduce the model segmentation performance when processing remote sensing images. In addition, the model has huge memory usage and computational cost. The emergence of SAM2 reduces memory usage and can segment images in real time on videos, but it is not an end-to-end segmentation model and requires additional prompts, which increases certain annotation costs. This method limits the usage of the model in different scenarios and tasks to a certain extent. Summary of the invention

[0004] The purpose of the present invention is to provide a remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network, comprising the following steps:

[0006] S1, using dual encoders and a single decoder to build a network model for remote sensing image semantic segmentation;

[0007] S2. Building a data set: preprocessing the acquired remote sensing images to be semantically segmented to build an image data set;

[0008] S3, training and testing network models;

[0009] S4. Select a loss function to quantify the difference between the model prediction and the actual value, and select an evaluation criterion to evaluate the performance of the network model.

[0010] Furthermore, in step S1: the encoder part uses the ResNet18 residual module as the main branch network, and the SAM2 image encoder is used as the auxiliary branch network, where the SAM2 module adds an efficient parameter fine-tuning Adapter module, and the two encoders are connected layer by layer using a multi-scale information aggregation module; the decoder part includes a global local window-multi-head self-attention module, a multi-scale information aggregation module and a feature refinement module.

[0011] Furthermore, the trunk branch encoder network uses four stages of Res Block in the ResNet18 residual module. Each layer downsamples the feature map by a scaling factor of 2 to capture features at different scales while increasing the receptive field. Its residual idea can effectively solve the problems of gradient vanishing and gradient exploding in deep neural network training, and can extract multi-scale semantic features and solve the scale change problem at a significantly low computational cost.

[0012] The modules included in SAM2 include image encoder module, mask decoder module and prompt encoder module; the image encoder module is divided into four layers, whose function is to map the image to the feature space and obtain multi-scale features; the mask decoder module generates high-quality object masks according to the input of the two encoders; the prompt encoder module is used to receive the encoded information of points, surfaces and texts; the auxiliary branch encoder network adopts the four-stage image encoder of SAM2, deletes its mask decoder module and prompt encoder module, and improves the universality of the network model; for the difference based on the migration of traditional image pre-training models to the field of remote sensing image semantic segmentation, all parameters of the image encoder are frozen, its original features are retained, and the focus is on training other parts of the network to avoid overfitting or maintain stability; secondly, the design adds an Adapter module before the Blocks of the image encoder for efficient parameter fine-tuning, which specifically includes a downsampling linear layer and a ReL u The SAM2Block module adds parameters and only trains the added parameters in the downstream fine-tuning task, keeping the original parameters of the pre-trained model unchanged. The calculations included are as follows:

[0013] Adapter(f input )=ReLu(Up(ReLu(Down(f input ))));

[0014] x′i =Attention(Norm(Adapter(x i-1 )))+x i-1 ;

[0015] x i =MLP(Norm(x′ i )+x′ i ;

[0016] Among them, x i and x i-1 are the output features of the i-th and i-1-th Transformer modules, respectively, x′ i It is the intermediate feature of the first sub-block in the i-th Transformer module. SAM2Block adds parameters and only trains the added parameters in the downstream fine-tuning task, keeping the original parameters of the pre-trained model unchanged. In this way, the pre-trained model occupies less memory and trains faster, effectively preventing the risk of overfitting.

[0017] Each stage of the two encoders performs feature fusion through a multi-scale information aggregation module; the multi-scale information aggregation module includes a detail preservation module and a feature fusion module. The detail preservation module uses the receptive field module to expand the receptive field and enhance the multi-scale feature information. At the same time, the number of channels of the four stages of the auxiliary branch encoder network is changed to make it the same as the number of channels of each stage of the trunk branch encoder network; the feature fusion module uses the temporal feature fusion attention module to fuse the bi-temporal features of the two encoders at the same stage through two main branches: the channel branch and the spatial branch. The channel branch focuses on enhancing the channel information in the feature map, while the spatial branch focuses on strengthening the spatial information. The bi-temporal weights are multiplied and summarized with the bi-temporal features to effectively fuse the bi-temporal features.

[0018] Furthermore, the decoder part consists of three global-local windows-multi-head self-attention modules, three multi-scale information aggregation modules and a feature refinement module;

[0019] The traditional Swin Transformer alternately executes the conventional multi-head self-attention module and the shifted window multi-head self-attention module in the continuous Swin Transformer Block; the present invention improves the shifted window self-attention mechanism of the Swin Transformer to obtain a global local window-multi-head self-attention module, and the specific improvements are as follows:

[0020] a. Replace the alternating execution of the regular multi-head self-attention module of Swin Transformer with a local window multi-head self-attention module;

[0021] b. Add a depthwise separable convolution before each of the two consecutive Swin Transformer Blocks to process the spatial and channel information separately, reduce the amount of computation, and improve the efficiency of feature extraction, thereby increasing the convergence speed of the Transformer module;

[0022] c. Replace the four Layer Norms in Swin Transformer Blocks with four Batch Norm normalization operations, which pays more attention to the size relationship between different samples and smoothes the size relationship between different features.

[0023] Furthermore, the alternating execution of the conventional multi-head self-attention module of Swin Transformer is replaced with a local window multi-head self-attention module; specifically, the input sequence is converted into a query vector, a key vector and a value vector through a linear transformation, and the key vector and the value vector are passed through a deep separable convolution layer. The query vector and the value vector are dot-producted and then Softmax normalized. The obtained attention weight is dot-producted with the value vector. The local window multi-head self-attention function can be described as:

[0024]

[0025] Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively. head is the number of attention heads;

[0026] The shifted window multi-head self-attention module of Swin Transformer is retained to enhance the cross-window information connection for extracting global feature information. Its global window multi-head self-attention function can be described as:

[0027]

[0028] Among them, RPE represents relative position encoding, and this local window multi-head self-attention module limits the receptive field of each query to its neighboring pixels, combining the advantages of local inductive bias of convolution and dynamic feature selection of self-attention; by combining local window multi-head self-attention with global window multi-head self-attention, it can capture local details and global context, and enhance the recognition ability of complex structures.

[0029] Furthermore, the four Layer Norms in the Swin Transformer Blocks are replaced with four BatchNorm normalization operations, which pay more attention to the size relationship between different samples and smooth out the size relationship between different features; the function of its global local window-multi-head self-attention module can be described as:

[0030] s″ i-1 =DSConv(s" i-1 )+s″′ i-1 ;

[0031] s′ i-1 =LocalAttention(BN(s″) i-1 ))+s″ i-1 );

[0032] s i-1 =MLP(BN(s′ i-1 ))+s′ i-1 );

[0033] s″ i =DSConv(s i-1 )+s i-1 ;

[0034] s′ i =GlobalAttention(BN(s″) i ))+s″ i )

[0035] s i =MLP(BN(s′ i ))+s′ i );

[0036] Among them, the right side of the equal sign is the input feature of the module, and the left side is the output feature, s″′ i-1 represents the input features of the depthwise separable convolutional layer module, s″ i-1 represents the output feature of the depthwise separable convolutional layer module, s′ i-1 Represents the output features of the LocalAttention module, s i-1 Represents the output features of the multi-layer perceptron module, s″ i represents the output feature of the depthwise separable convolutional layer module, s′ i Represents the output features of the GlobalAttention module, s i Represents the output features of the multi-layer perceptron module, DSConv represents the depthwise separable convolutional layer, BN represents batch normalization, and MLP represents the multi-layer perceptron;

[0037] The multi-scale information aggregation module in the decoder is basically the same as that in the encoder, but an upsampling operation based on the bilinear interpolation method is added to the feature fusion module, so that the image after each module of the decoder is expanded to twice the original size and fused with the feature information of the encoder, combining feature information from different scales, thereby improving the segmentation performance of the model;

[0038] The feature refinement module adopts the weighted summation operation of channel path attention and spatial path attention features. The channel path attention includes adaptive average pooling layer, adaptive maximum pooling layer, down-sampling 1×1 convolution layer, ReLu activation function, up-sampling 1×1 convolution layer, Sigmoid activation function, while the spatial path attention includes average pooling, maximum pooling, and Sigmoid activation function. After the weighted summation of the two attention path features, the residual connection is performed with the original features after the 3×3 convolution layer and batch normalization layer to prevent network degradation; then the final segmented image is obtained by the ReLu activation function, 1×1 convolution layer, and four-fold upsampling bilinear interpolation method.

[0039] Furthermore, in step S2: the acquired remote sensing images to be semantically segmented are cropped using a sliding window cropping method, each image is A×A pixels in size, overlapped by A / 2 pixels, and three image enhancement techniques, namely, random scaling, random vertical flipping, and random horizontal flipping, are used to expand the data set, and the data set is divided into a training set, a validation set, and a test set.

[0040] Furthermore, in step S3: the network model is trained and tested based on the PyTorch deep learning framework programming; the training set and validation set constructed in step S1 are used to perform parameter training on the semantic segmentation network with dual-branch multi-scale feature fusion, and the learning rate adjustment strategy is the cosine annealing algorithm to obtain the training weights, which are used to test the segmentation performance of the model on the test set, and finally the remote sensing segmentation image is obtained.

[0041] Further, in step S4: a loss function and an evaluation index are selected; since remote sensing images are highly complex and class-imbalanced, traditional classification methods often cannot effectively and accurately capture the rich information in the images, resulting in classification accuracy that cannot meet the needs of practical applications; the loss function is used to quantify the difference between the model prediction and the actual value, and guide parameter optimization to improve model performance; the evaluation index is used to evaluate the segmentation effect of the model and the complexity of the model;

[0042] The joint loss function of the logit adjustment loss function and the dice loss function can be expressed as:

[0043]

[0044] L=r×L logit +(1-r)×L dice ;

[0045] Among them, N and K represent the number of samples and the number of categories respectively, y (n) and represents the encoding of the semantic label and its corresponding network softmax output, n∈[1,…,N], is the confidence of k-type sample n, π k is the prior knowledge of category k, with weight r∈[0, 1]. When the categories are unbalanced, the model tends to predict categories with higher frequencies. To solve this problem, the mutual information-based Logit adjustment loss function adjusts the logit of each category and introduces the frequency of occurrence of each category as prior knowledge into the training process to solve the category imbalance problem.

[0046] Furthermore, the average F1 score and average intersection-over-union ratio are selected as the evaluation criteria for the overall accuracy of the algorithm, and the average intersection-over-union ratio is used as the evaluation criteria for the segmentation accuracy of each category; these evaluation indicators can comprehensively evaluate the segmentation effect of the model, among which the average F1 score is suitable for the evaluation of multi-category segmentation models because it considers both precision and recall; the average intersection-over-union ratio is a commonly used indicator in semantic segmentation tasks, which can reflect the contribution of the model to the overall segmentation results; the number of floating-point operations per second and the number of parameters are used to evaluate the complexity of the model. By comprehensively considering all indicators, the performance of the image semantic segmentation model can be more accurately evaluated; the evaluation criteria are described as:

[0047]

[0048] Among them, TP is the positive sample predicted to be correct; FP is the positive sample predicted to be wrong; TN is the negative sample predicted to be correct; FP is the negative sample predicted to be wrong; Precision is the accuracy rate, It indicates the proportion of positive samples among all samples predicted to be positive. Recall is the recall rate. It refers to the proportion of samples predicted to be positive among all samples that are actually positive.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. Through the transfer learning method, the knowledge learned from the pre-trained segmented arbitrary model 2 is used, and the model is efficiently fine-tuned through the designed Adapter module to quickly adapt to the new remote sensing image semantic segmentation task, and the model can be trained even on devices with limited memory;

[0051] 2. Based on the idea of ​​global-local attention mechanism, the global-local window-multi-head self-attention module is designed. It has powerful global and local feature capture capabilities, can effectively extract the spatial and spectral information of remote sensing images, and realize accurate semantic segmentation of remote sensing images;

[0052] 3. Design a feature refinement module that combines channel path attention and spatial path attention features to improve the feature differentiation ability and enhance the model's adaptability to complex scenes, further improving the performance and accuracy of segmentation tasks;

[0053] 4. Design a joint Logit adjustment loss function strategy to deal with the category imbalance problem of remote sensing image data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall model framework diagram of the remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network of the present invention;

[0055] Figure 2 A schematic diagram of the auxiliary branch encoder network structure of the remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network of the present invention;

[0056] Figure 3 It is a schematic diagram of the structure of the global local window-multi-head self-attention module of the remote sensing image semantic segmentation method based on the dual-branch multi-scale fusion network of the present invention;

[0057] Figure 4 This is a schematic diagram of the feature refinement module structure of the remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Example: Figure 1-Figure 4 As shown, a technical solution provided by the present invention is a remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network, comprising the following steps:

[0060] S1, using dual encoders and a single decoder to build a network model for remote sensing image semantic segmentation;

[0061] S2. Building a data set: preprocessing the acquired remote sensing images to be semantically segmented to build an image data set;

[0062] S3, training and testing network models;

[0063] S4. Select a loss function to quantify the difference between the model prediction and the actual value, and select an evaluation criterion to evaluate the performance of the network model.

[0064] In step S1: the encoder part uses the ResNet18 residual module as the main branch network, and the SAM2 image encoder is used as the auxiliary branch network, where the SAM2 image encoder adds an efficient parameter fine-tuning Adapter module, and the two encoders are connected layer by layer using a multi-scale information aggregation module; the decoder part includes a global local window-multi-head self-attention module, a multi-scale information aggregation module, and a feature refinement module.

[0065] The trunk-branch encoder network uses four stages of Res Block in the ResNet18 residual module. Each layer downsamples the feature map by a scaling factor of 2 to capture features at different scales while increasing the receptive field. Its residual idea can effectively solve the problems of gradient vanishing and gradient exploding in deep neural network training, and can extract multi-scale semantic features and solve the scale change problem at a significantly low computational cost.

[0066] The modules included in SAM2 are image encoder module, mask decoder module and prompt encoder module; the image encoder module is divided into four layers, which is used to map the image to the feature space and obtain multi-scale features; the mask decoder module generates high-quality object masks according to the input of the two encoders; the prompt encoder module is used to receive the encoding information of points, surfaces and texts; the auxiliary branch encoder network adopts the four-stage image encoder of SAM2, deletes its mask decoder module and prompt encoder module, and improves the universality of the network model; for the difference based on the migration of traditional image preprocessing models to the field of remote sensing image semantic segmentation, freeze all parameters of the image encoder, retain its original features, and focus on training other parts of the network to avoid overfitting or maintain stability; secondly, the design adds an Adapter module before the Blocks of the image encoder for efficient parameter fine-tuning, which specifically includes a down-sampling linear layer and a ReLu activation function, as well as an up-sampling linear layer and a ReLu activation function, and performs residual connection; the calculations included in the Adapter module and the SAM2Block module are:

[0067] Adapter(f input )=ReLu(Up(ReLu(Down(f input ))));

[0068] x′ i =Attention(Norm(Adapter(x i-1 )))+x i-1 ;

[0069] x i =MLP(Norm(x′ i ))+x′ i ;

[0070] Among them, x i and x i-1 are the output features of the i-th and i-1-th Transformer modules, respectively, x′ i It is the intermediate feature of the first sub-block in the i-th Transformer module. SAM2Block adds parameters and only trains the added parameters in the downstream fine-tuning task, keeping the original parameters of the pre-trained model unchanged. In this way, the pre-trained model occupies less memory and trains faster, effectively preventing the risk of overfitting.

[0071] Each stage of the two encoders performs feature fusion through a multi-scale information aggregation module; the multi-scale information aggregation module includes a detail preservation module and a feature fusion module. The detail preservation module uses the receptive field module to expand the receptive field and enhance the multi-scale feature information. At the same time, the number of channels of the four stages of the auxiliary branch encoder network is changed to make it the same as the number of channels of each stage of the trunk branch encoder network; the feature fusion module uses the temporal feature fusion attention module to fuse the bi-temporal features of the two encoders at the same stage through two main branches: the channel branch and the spatial branch. The channel branch focuses on enhancing the channel information in the feature map, while the spatial branch focuses on strengthening the spatial information. The bi-temporal weights are multiplied and summarized with the bi-temporal features to effectively fuse the bi-temporal features.

[0072] The decoder part consists of three global-local window-multi-head self-attention modules, three multi-scale information aggregation modules and a feature refinement module;

[0073] The traditional Swin Transformer alternately executes the conventional multi-head self-attention module and the shifted window multi-head self-attention module in the continuous Swin Transformer Block; the present invention improves the shifted window self-attention mechanism of the Swin Transformer to obtain a global local window-multi-head self-attention module, and the specific improvements are as follows:

[0074] a. Replace the alternating execution of the regular multi-head self-attention module of Swin Transformer with a local window multi-head self-attention module;

[0075] b. Add a depthwise separable convolution before each of the two consecutive Swin Transformer Blocks to process the spatial and channel information separately, reduce the amount of computation, and improve the efficiency of feature extraction, thereby increasing the convergence speed of the Transformer module;

[0076] c. Replace the four Layer Norms in Swin Transformer Blocks with four Batch Norm normalization operations, which pays more attention to the size relationship between different samples and smoothes the size relationship between different features.

[0077] The alternating execution of the conventional multi-head self-attention module of Swin Transformer is replaced by a local window multi-head self-attention module; specifically, the input sequence is converted into a query vector, a key vector and a value vector through a linear transformation, and the key vector and the value vector are passed through a deep separable convolution layer. The query vector and the value vector are dot-producted and then Softmax normalized. The obtained attention weight is dot-producted with the value vector. Its local window multi-head self-attention function can be described as:

[0078]

[0079] Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively. head is the number of attention heads;

[0080] The shifted window multi-head self-attention module of Swin Transformer is retained to enhance the cross-window information connection for extracting global feature information. Its global window multi-head self-attention function can be described as:

[0081]

[0082] Among them, RPE represents relative position encoding, and this local window multi-head self-attention module limits the receptive field of each query to its neighboring pixels, combining the advantages of local inductive bias of convolution and dynamic feature selection of self-attention; by combining local window multi-head self-attention with global window multi-head self-attention, it can capture local details and global context, and enhance the recognition ability of complex structures.

[0083] The four Layer Norms in Swin Transformer Blocks are replaced by four Batch Norm normalization operations, which pay more attention to the size relationship between different samples and smooth out the size relationship between different features; the function of its global local window-multi-head self-attention module can be described as:

[0084] s″ i-1 =DSConv(s″′) i-1 )+s″′ i-1 ;

[0085] s′ i-1 =LocalAttention(BN(s″) i-1))+s″ i-1 );

[0086] s i-1 =MLP(BN(s′ i-1 ))+s′ i-1 );

[0087] s″ i =DSConv(s i-1 )+s i-1 ;

[0088] s′ i =GlobalAttention(BN(s″) i ))+s″ i )

[0089] s i =MLP(BN(s′ i ))+s′ i );

[0090] Among them, the right side of the equal sign is the input feature of the module, and the left side is the output feature, s″′ i-1 represents the input features of the depthwise separable convolutional layer module, s″ i-1 represents the output feature of the depthwise separable convolutional layer module, s′ i-1 Represents the output features of the LocalAttention module, s i-1 Represents the output features of the multi-layer perceptron module, s″ i represents the output feature of the depthwise separable convolutional layer module, s′ i Represents the output features of the GlobalAttention module, s i Represents the output features of the multi-layer perceptron module, BN is batch normalization, and MLP represents multi-layer perceptron;

[0091] The multi-scale information aggregation module in the decoder is basically the same as that in the encoder, but an upsampling operation based on the bilinear interpolation method is added to the feature fusion module, so that the image after each module of the decoder is expanded to twice the original size and fused with the feature information of the encoder, combining feature information from different scales, thereby improving the segmentation performance of the model;

[0092] The feature refinement module adopts the weighted summation operation of channel path attention and spatial path attention features. The channel path attention includes adaptive average pooling layer, adaptive maximum pooling layer, down-sampling 1×1 convolution layer, ReLu activation function, up-sampling 1×1 convolution layer, Sigmoid activation function, while the spatial path attention includes average pooling, maximum pooling, and Sigmoid activation function. After the weighted summation of the two attention path features, the residual connection is performed with the original features after the 3×3 convolution layer and batch normalization layer to prevent network degradation; then the final segmented image is obtained by the ReLu activation function, 1×1 convolution layer, and four-fold upsampling bilinear interpolation method.

[0093] In step S2: the Vaihingen dataset consists of 33 remote sensing images with an average size of 2500×2000 pixels and a spatial resolution of 9 cm. The TOP images with IDs of 1, 3, 5, 7, 11, 13, 15, 17, 21, 23, 26, 28, 32, 34 and 37 are selected as the training set, the image with ID 30 is used as the validation set, and the remaining 17 images are used as the test set; the Potsdam dataset consists of 38 high-resolution remote sensing images with an average size of 6000×6000 pixels and a spatial resolution of 5 cm. The TOP images with IDs of 2_11, 2_12, 3_10, 3_11, 3_12 are selected. , 4_10, 4_11, 4_12, 5_10, 5_11, 5_12, 6_7, 6_8, 6_9, 6_10, 6_11, 6_12, 7_7, 7_8, 7_9, 7_11 and 7_12 were used for training, the image with ID: 2_10 was used as the validation set, and the remaining 15 images, except for the image with ID: 7_10 which was an incorrectly labeled image, the remaining 14 images were used for testing; the images were cropped using the sliding window cropping method, each image was 512×512 pixels in size, with an overlap of 256 pixels, and the three image enhancement techniques of random scale, random vertical flip and random horizontal flip were used to expand the dataset.

[0094] In step S3: based on the PyTorch deep learning framework programming, the network model is trained and tested; the training set and validation set constructed in step S1 are used to perform parameter training on the semantic segmentation network with dual-branch multi-scale feature fusion, the number of training times is set to 200, the AdamW optimizer is used, the initial learning rate is 6E-4, the weight decay is 0.01, and the learning rate adjustment strategy is the cosine annealing algorithm. The training weights are obtained and used to test the segmentation performance of the model on the test set, and finally the remote sensing segmentation image is obtained.

[0095] In step S4: select a loss function and an evaluation index; due to the high complexity and class imbalance of remote sensing images, traditional classification methods often cannot effectively and accurately capture the rich information in the image, resulting in classification accuracy that cannot meet the needs of practical applications; the loss function is used to quantify the difference between the model prediction and the actual value, and guide parameter optimization to improve model performance; the evaluation index is used to evaluate the segmentation effect of the model and the complexity of the model;

[0096] The joint loss function of the logit adjustment loss function and the dice loss function can be expressed as:

[0097]

[0098] L=r×L logit +(1-r)×L dice ;

[0099] Among them, N and K represent the number of samples and the number of categories respectively, y (n) and represents the encoding of the semantic label and its corresponding network softmax output, n∈[1,…,N], is the confidence of k-type sample n, π k is the prior knowledge of category k, with weight r∈[0, 1]; when the categories are unbalanced, the model tends to predict categories with higher frequency. To solve this problem, the Logit adjustment loss function based on mutual information adjusts the logit of each category and introduces the frequency of occurrence of each category as prior knowledge into the training process to solve the problem of category imbalance; r ranges from 0 to 1, and a weighted method is used to make the final sum equal to 100%, where 0.4 is obtained from multiple experimental tests and the default value is 0.4.

[0100] The average F1 score and average intersection-over-union ratio are selected as the evaluation criteria for the overall accuracy of the algorithm, and the average intersection-over-union ratio is used as the evaluation criteria for the segmentation accuracy of each category; these evaluation indicators can comprehensively evaluate the segmentation effect of the model, among which the average F1 score is suitable for the evaluation of multi-category segmentation models because it considers both precision and recall; the average intersection-over-union ratio is a commonly used indicator in semantic segmentation tasks, which can reflect the contribution of the model to the overall segmentation results; the number of floating-point operations per second and the number of parameters are used to evaluate the complexity of the model. By comprehensively considering all indicators, the performance of the image semantic segmentation model can be more accurately evaluated; the evaluation criteria are described as:

[0101]

[0102] Among them, TP is the positive sample predicted to be correct; FP is the positive sample predicted to be wrong; TN is the negative sample predicted to be correct; FP is the negative sample predicted to be wrong; Precision is the accuracy rate, It indicates the proportion of positive samples among all samples predicted to be positive. Recall is the recall rate. It refers to the proportion of samples predicted to be positive among all samples that are actually positive.

[0103] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network, characterized by: The following steps are involved: S1, using dual encoders and a single decoder to build a network model for remote sensing image semantic segmentation; S2. Building a data set: preprocessing the acquired remote sensing images to be semantically segmented to build an image data set; S3, training and testing network models; S4. Select a loss function to quantify the difference between the model prediction and the actual value, and select an evaluation criterion to evaluate the performance of the network model.

2. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 1, characterized in that: In step S1: the entire network model adopts a dual encoder and single decoder structure. The encoder part uses the ResNet18 residual module as the main branch network, and the SAM2 image encoder is used as the auxiliary branch network. An efficient parameter fine-tuning Adapter module is added to the SAM2 image encoder to form a SAM2Block module. The two encoders are connected layer by layer using a multi-scale information aggregation module. The decoder part includes a global local window-multi-head self-attention module, a multi-scale information aggregation module, and a feature refinement module.

3. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 2, characterized in that: The trunk branch encoder network uses four stages of Res Block in the ResNet18 residual module. Each layer downsamples the feature map by a scaling factor of 2 to capture features at different scales while increasing the receptive field. The auxiliary branch encoder network adopts the four-stage image encoder of SAM2, and deletes its mask decoder module and prompt encoder module; for the difference in migrating from the traditional image pre-training model to the field of remote sensing image semantic segmentation, all parameters of the image encoder are frozen, its original features are retained, and the focus is on training other parts of the network; secondly, the design adds an Adapter module before the Blocks of the image encoder for parameter adjustment, including a down-sampling linear layer and a ReLu activation function, as well as an up-sampling linear layer and a ReLu activation function, and performs residual connection; the SAM2Block module adds parameters and only trains the added parameters in the downstream fine-tuning task, keeping the original parameters of the pre-training model unchanged, including the following calculations: Adapter(f input )=ReLu(Up(ReLu(Down(f input )))); x′ i =Attention(Norm(Adapter(x i-1 )))+x i-1 ; x i =MLP(Norm(x′ i ))+x′ i ; Among them, x i and x i-1 are the output features of the i-th and i-1-th Transformer modules, respectively, x′ i is the intermediate feature of the first sub-block in the i-th Transformer module; Each stage of the two encoders performs feature fusion through a multi-scale information aggregation module; the multi-scale information aggregation module includes a detail preservation module and a feature fusion module. The detail preservation module uses the receptive field module to expand the receptive field and enhance the multi-scale feature information. At the same time, the number of channels of the four stages of the auxiliary branch encoder network is changed to make it the same as the number of channels of each stage of the trunk branch encoder network; the feature fusion module uses the temporal feature fusion attention module to fuse the dual-temporal features of the two encoders at the same stage through two branches: the channel branch and the spatial branch. The channel branch focuses on enhancing the channel information in the feature map, while the spatial branch focuses on strengthening the spatial information. The dual-temporal weights are multiplied by the dual-temporal features and summarized.

4. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 3, characterized in that: The decoder part consists of three global-local window-multi-head self-attention modules, three multi-scale information aggregation modules and a feature refinement module; The shift window self-attention mechanism of Swin Transformer is improved to obtain the global local window-multi-head self-attention module. The specific improvement measures are as follows: a. Replace the alternating execution of the regular multi-head self-attention module of Swin Transformer with a local window multi-head self-attention module; b. Add a depth-wise separable convolution before each of the two consecutive Swin Transformer Blocks; c. Replace the four Layer Norms in Swin Transformer Blocks with four Batch Norm normalization operations.

5. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 4, characterized in that: The alternating execution of the conventional multi-head self-attention module of Swin Transformer is replaced by a local window multi-head self-attention module; specifically, the input sequence is converted into a query vector, a key vector, and a value vector through a linear transformation, and the key vector and the value vector are passed through a depth-separable convolution layer. The query vector and the value vector are dot-producted and then Softmax normalized. The obtained attention weight is dot-producted with the value vector. The local window multi-head self-attention function can be described as: Among them, Q, K, and V are query vectors, key vectors, and value vectors respectively. head is the number of attention heads; The shifted window multi-head self-attention module of Swin Transformer is retained, and the global window multi-head self-attention function can be described as: Among them, RPE represents relative position encoding, and this local window multi-head self-attention module limits the receptive field of each query to its neighboring pixels; by combining the local window multi-head self-attention with the global window multi-head self-attention, both local details and global context are captured.

6. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 4, characterized in that: Replace the four Layer Norms in Swin Transformer Blocks with four Batch Norm normalization operations. The function of the global local window-multi-head self-attention module can be described as: s″ i-1 =DSConv(s″′ i-1 )+s″′ i-1 ; s′ i-1 =LocalAttention(BN(s″ i-1 ))+s″ i-1 ); s i-1 =MLP(BN(s′ i-1 ))+s′ i-1 ); s″ i =DSConv(s i-1 )+s i-1 ; s′ i =GlobalAttention(BN(s″ i ))+s″ i ); s i =MLP(BN(s′ i ))+s′ i ); Among them, the right side of the equal sign is the input feature of the module, and the left side is the output feature; s″′ i-1 represents the input features of the depthwise separable convolutional layer module, s″ i-1 represents the output feature of the depthwise separable convolutional layer module, s′ i-1 Represents the output features of the LocalAttention module, s i-1 Represents the output features of the multi-layer perceptron module, s″ i represents the output feature of the depthwise separable convolutional layer module, s′ i Represents the output features of the GlobalAttention module, s i Represents the output features of the multi-layer perceptron module, DSConv represents the depthwise separable convolutional layer, BN represents batch normalization, and MLP represents the multi-layer perceptron; The multi-scale information aggregation module in the decoder is the same as that in the encoder, but an upsampling operation based on the bilinear interpolation method is added to the feature fusion module, so that the image after each module of the decoder is expanded to twice the original size and fused with the feature information of the encoder; The feature refinement module adopts the weighted summation operation of channel path attention and spatial path attention features. The channel path attention includes adaptive average pooling layer, adaptive maximum pooling layer, down-sampling 1×1 convolution layer, ReLu activation function, up-sampling 1×1 convolution layer, Sigmoid activation function, while the spatial path attention includes average pooling, maximum pooling, and Sigmoid activation function. After the weighted summation of the two attention path features, the residual connection is performed with the original features after the 3×3 convolution layer and batch normalization layer to prevent network degradation; then the final segmented image is obtained by the ReLu activation function, 1×1 convolution layer, and four-fold upsampling bilinear interpolation method.

7. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 1, characterized in that: In step S2: the acquired remote sensing images to be semantically segmented are cropped using the sliding window cropping method. The size of each image is A×A pixels, with an overlap of A / 2 pixels. The three image enhancement techniques of random scaling, random vertical flipping and random horizontal flipping are used to expand the data set and divide it into a training set, a validation set and a test set.

8. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 1, characterized in that: In step S3: the training and testing network model is implemented based on the PyTorch deep learning framework programming; the constructed training set and validation set are used to perform parameter training on the semantic segmentation network with dual-branch multi-scale feature fusion, the number of training times is set to X, the AdamW optimizer is used, the initial learning rate is 6E-4, the weight decay is r, and the learning rate adjustment strategy is the cosine annealing algorithm. The training weights are obtained and used to test the segmentation performance of the model on the test set.

9. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 1, characterized in that: In step S4: selecting a loss function and an evaluation index; the loss function is used to quantify the difference between the model prediction and the actual value, and guide parameter optimization to improve the model performance; the evaluation index is used to evaluate the segmentation effect of the model and the complexity of the model; The joint loss function of the 1ogit adjustment loss function and the dice loss function can be expressed as: L=r×L logit +(1-r)×L dice ; Among them, L logit represents the logit adjusted loss function, L dice represents the dice loss function, L represents the joint loss function of the logit adjusted loss function and the dice loss function, N and K represent the number of samples and the number of categories respectively, y (n) and represents the encoding of the semantic label and its corresponding network softmax output, n∈[1,…,N], is the confidence of k-type sample n, π k is the prior knowledge of category k, with weight r∈[0, 1]. When the categories are unbalanced, the model tends to predict categories with higher frequencies. To solve this problem, the mutual information-based Logit adjustment loss function adjusts the logit of each category and introduces the frequency of occurrence of each category as prior knowledge into the training process to solve the category imbalance problem.

10. The remote sensing image semantic segmentation method based on a dual-branch multi-scale fusion network according to claim 9, characterized in that: The average F1 score and average intersection-over-union ratio were selected as the evaluation criteria for the overall accuracy of the algorithm, and the average intersection-over-union ratio was used as the evaluation criterion for the segmentation accuracy of each category; The average F1 score is suitable for the evaluation of multi-category segmentation models; the average intersection-over-union ratio is a commonly used indicator in semantic segmentation tasks; the number of floating-point operations per second and the number of parameters are used to evaluate the complexity of the model; the evaluation criteria are described as follows: Among them, F1-score represents the average F1 score; mIoU represents the average intersection over union ratio; TP represents a positive sample that is predicted to be correct; FP represents a positive sample that is predicted to be incorrect; TN represents a negative sample that is predicted to be correct; FP represents a negative sample that is predicted to be incorrect; Precision is the accuracy rate, It indicates the proportion of positive samples among all samples predicted to be positive. Recall is the recall rate. It refers to the proportion of samples predicted to be positive among all samples that are actually positive.

Images