Image segmentation method based on deep learning remote sensing image

Abstract

The application provides a kind of image segmentation method based on deep learning remote sensing image, it is related to image processing field, comprising: step S1, acquisition remote sensing image, data set construction is carried out using remote sensing image, and data preprocessing is completed;Step S2, remote sensing image semantic segmentation model is constructed;Step S3, the remote sensing image semantic segmentation model constructed in step S2 is trained, verified and optimized using the data set in step S1, and the model construction is completed.The method improves the consistency of segmentation accuracy and expression by multi-branch deep collaborative modeling, thereby improving the robustness of the model in the boundary blur, small target and label defect area;At the same time, the method uses multi-scale path segmentation, adaptive attention fusion and residual inverse MLP structure to realize more efficient multi-level feature representation and achieve dynamic balance of global and local modeling relationship.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a deep learning-based remote sensing image image segmentation method. BACKGROUND

[0002] Remote sensing image is a digital record of electromagnetic wave reflection or radiation information of the earth's surface obtained by non-contact sensors (such as equipment carried by satellite, airplane, unmanned aerial vehicle, etc. platform); it takes pixel as the basic unit, reacts the spectral characteristics of ground objects through the combination of different wave bands, and is widely used in the fields of agriculture, environmental monitoring, urban planning, disaster management, etc. At present, remote sensing image detection not only leads to detection accuracy decline due to light disturbance (i.e. the same scene will exhibit different characteristics under different light conditions), but also faces the problems of complex regional changes, irregular size and quantity. Deep learning has strong feature extraction method and nonlinear expression ability, and has been widely applied to change detection tasks and has shown good performance. At present, there are some literatures that apply deep learning to remote sensing images, for example, Chinese patent document CN119625553A discloses a vegetation coverage estimation method and device based on remote sensing semantic segmentation, which adopts lightweight design, reduces the calculation complexity of the network while ensuring the segmentation accuracy, makes it suitable for devices with limited resources, and improves the flexibility of practical application; secondly, the Block module combines deep separable convolution and residual connection, which not only reduces the model parameter quantity and calculation cost, but also enhances the feature extraction ability, improves the accuracy and detail retention ability of the segmentation result; through the introduction of multi-path feature fusion and SiLU activation function, the network shows good nonlinear expression ability when processing multi-scale remote sensing images, effectively improving the estimation accuracy of complex ground objects and vegetation coverage; in addition, the application of layer normalization significantly enhances the stability of the model, ensuring its efficient convergence under different data distribution. However, there are natural scale differences in remote sensing scenes, making it difficult for single-scale models to effectively capture diversified target features; at the same time, complex ground cover structure and transition often lead to target boundary blur, which restricts the segmentation performance and image boundary area discrimination; in addition, the semantic relationship in remote sensing image often spans a wide spatial range, which needs to ensure long-range dependence and avoid context information loss; in the above application of deep learning and remote sensing image, although the recognition accuracy can be improved to a certain extent and the calculation complexity can be reduced, there are still problems such as inability to efficiently and continuously fuse multi-scale representations, inability to ensure boundary clarity in noise and detail-rich high-resolution images, and inability to balance calculation efficiency and segmentation accuracy when processing large-scale data and complex shape targets. SUMMARY

[0003] In view of the problems in the prior art, the present application aims to provide an image segmentation method based on deep learning remote sensing image, which improves the segmentation accuracy and expression consistency through multi-branch deep collaborative modeling, thereby improving the robustness of the model in the boundary blur, small target and label defect area. Meanwhile, the method uses multi-scale path segmentation, adaptive attention fusion and residual inverse MLP structure to realize more efficient multi-level feature representation, thereby dynamically balancing the global and local modeling relationship to solve the problems existing in the combination of deep learning and remote sensing image in the prior art.

[0004] The object of the present application is achieved by the following technical solutions:

[0005] An image segmentation method based on deep learning remote sensing image, comprising:

[0006] Step S1, collect remote sensing images, use remote sensing images to build a data set, and complete data preprocessing;

[0007] Step S2, build a remote sensing image semantic segmentation model: first, build a convolutional neural network (CNN) and SwiftFormer encoder; then, build a three-branch channel and use a three-branch mutual induction fusion mechanism to realize dynamic interactive modeling of local and global information; then, use a region perception gating mechanism to realize region adaptive feature information flow regulation; then, use a hierarchical feature fusion module to realize multi-scale semantic information fusion; finally, use a residual reconstruction learning and self-repair mechanism (Residual Reconstruction Learning for Self-Repair) to complete model self-recognition, predict errors, and perform self-repair;

[0008] Step S3, use the data set in step S1 to train, verify and optimize the remote sensing image semantic segmentation model built in step S2, and complete model building.

[0009] Based on further optimization of the above scheme, the remote sensing images in step S1 use the LoveDA dataset, which contains large-scale, high-resolution remote sensing images, providing a total of 5,987 remote sensing images with a resolution of 1024×1024 and their corresponding pixel-level semantic labels. The dataset covers multiple regions in China, including urban, rural, and mixed areas, with significant urban-rural distribution differences, and can truly reflect the common inter-domain offset problem in actual remote sensing applications. The dataset is constructed using remote sensing images as follows: the dataset is cropped with a step size of 512 pixels to obtain images of 512×512 pixels each, and the images are preprocessed to ignore classes with pixel values ​​of 0. The remaining seven ground objects include background, building, road, water, barren soil, agriculture, and forest. The images are randomly rotated, translated, flipped, cropped, and other operations are performed to further increase the data volume to improve the generalization ability of the model. The dataset is then divided into a training set, a validation set, and a test set.

[0010] Based on the further optimization of the above solution, the "construction of convolutional neural network and SwiftFormer encoder" in step S2 is specifically as follows:

[0011] The cascaded SwiftFormer encoder architecture uses an efficient attention mechanism to gradually extract multi-scale features at different network stages while maintaining the ability to model global context. This architecture enables the joint modeling of local detail features and long-range dependencies.

[0012] For a given input feature , the SwiftFormer encoder first adopts a 3x3 depthwise separable convolution ( DWConv ) captures the spatial structure and is then passed through a 1x1 convolution ( Conv ) transforms the channel dimension, thereby compressing features while retaining local structural information, specifically:

[0013]

[0014] At the same time, the SwiftFormer encoder's efficient additive attention mechanism (EAA) significantly reduces computational costs while maintaining the ability to capture global context information. The specific process is as follows:

[0015] First, for the input features , obtain its query matrix through two linear projections Q and bond matrix K :

[0016]

[0017] Where: , represents the learning parameters; n represents the sequence length, d represents the embedding dimension;

[0018] Then, each query matrix is ​​calculated Q and learning vectors The scaled dot product between them generates a global attention score for each position :

[0019]

[0020] Afterwards, the global attention score is used to aggregate the query matrix to obtain the global query vector q :

[0021]

[0022] The global query vector is then fused with the key matrix through element-wise multiplication to encode the interaction between all spatial locations:

[0023]

[0024] Where: Linear represents a linear transformation; Represents element-wise multiplication;

[0025] Finally, the global context features are combined with the normalized query representation through a residual connection to generate the final output of the encoder:

[0026]

[0027] Where: Norm () represents the normalization function.

[0028] Based on the further optimization of the above solution, the step S2 of "constructing a three-branch channel and using a three-branch mutual guidance fusion mechanism to realize dynamic interactive modeling of local and global information" is specifically as follows:

[0029] First, a unified feature map is extracted from the backbone network of the convolutional neural network and the SwiftFormer encoder. F shared , and input it into three branches respectively:

[0030] Detail branch ( Detail Branch ): , used to extract texture details;

[0031] Boundary branch ( Boundary Branch ): , for extracting edge line information;

[0032] global branch ( Global Branch ): , for extracting context;

[0033] In the formula: indicates a feature mapping function of the detail branch, composed of convolution, ReLU , BN and other local extraction modules; indicates a boundary feature extraction module, including Sobe convolution, edge enhancement layer, etc.; indicates Transformer , a global modeling module composed of attention mechanism, biased towards semantic grouping and region perception;

[0034] After that, mutual guidance in the three branches is established through the guidance map respectively:

[0035]

[0036] In the formula: , , , , , respectively indicate the global branch guiding the detail branch map, the boundary branch guiding the detail branch map, the global branch guiding the boundary branch map, the detail branch guiding the boundary branch map, the detail branch guiding the global branch map, and the boundary branch guiding the global branch map. indicates a Sigmoid activation function, whose range is [0, 1]; indicates channel dimension splicing;

[0037] Finally, the outputs of the detail branch, the boundary branch and the global branch are updated respectively through the guidance map:

[0038]

[0039] In the formula: , , respectively indicate the updated detail branch feature map, the boundary branch feature map and the global branch feature map. , , , , , respectively indicate learning parameters.

[0040] Based on the further optimization of the above scheme, the "realizing region adaptive feature information flow regulation with region perception gating mechanism" in step S2 is specifically:

[0041] First, the region structure feature extraction module is set, each pixel in the region has a set of values, which reflect whether the current pixel position is in the edge, corner or texture region of sharp change through the structural complexity S(x,y)

[0042]

[0043]

[0044] In the formula: G x The point I(x,y) In the x axis direction gradient; G y The point I(x,y) In the y axis direction gradient;

[0045] The semantic confidence of the current region C(x,y) To determine the accuracy of the prediction:

[0046]

[0047] In the formula: Softmax() The Softmax function; F pred (x,y) The predicted feature vector of the model at the pixel point (x,y) ;

[0048] The texture intensity of the current region is determined by the local gray standard deviation T(x,y)

[0049]

[0050] In the formula: I i The pixel value of the small window centered on (x,y) ; The pixel mean value; N The number of pixels in the current region;

[0051] The region perception gating module is deployed in the detail branch, boundary branch and global branch, and the structural complexity S(x,y) , semantic confidence C(x,y) And texture intensity T(x,y) , and the information passing weight is dynamically adjusted through the gating function, so that the region level difference modeling is completed before feature fusion, and the fusion effect and segmentation accuracy are improved, which is:

[0052] Detail branch: ​​

[0053]

[0054]

[0055] Boundary branches:

[0056]

[0057]

[0058] Global branch:

[0059]

[0060]

[0061] Where: W d 、 W b 、 W g Represent the weight parameters corresponding to the detail branch, boundary branch, and global branch respectively; B d 、 B b 、 B g Represent the bias items corresponding to the detail branch, boundary branch, and global branch respectively; R d (x,y) 、 R b (x,y) 、 R g (x,y) Respectively represent the corresponding structural complexity in the detail branch, boundary branch, and global branch S(x,y) , semantic confidence C(x,y) and texture strength T(x,y) Combined into a regional structure information vector.

[0062] Based on the further optimization of the above solution, the step S2 of "using the hierarchical feature fusion module to realize multi-scale semantic information fusion" is specifically as follows:

[0063] Introduce the hierarchical feature fusion module at the 1 / 8 and 1 / 16 resolution stages of the backbone network: Assume that the input features of the current backbone network, that is, the global branch features, are ,in, 、 Represent the edge semantic features of the boundary branch and the local structural features of the detail branch respectively, then:

[0064]

[0065] wherein: AvgPool represents average pooling; MaxPool represents max pooling; MLP represents a multi-layer perceptron with shared weights;

[0066] Channel attention weights are regenerated by element-wise multiplication of global features:

[0067]

[0068] A spatial attention mechanism is adopted to highlight the structural salient regions of the detail branch:

[0069]

[0070]

[0071] Finally, the attention-enhanced features , are concatenated with the input features F i-1 along the channel dimension to form the fused input features:

[0072] .

[0073] Further optimization based on the above scheme, the step S2 "adopting residual reconstruction learning and self-repair mechanism, completing model self-identification, predicting errors and self-repairing" is specifically:

[0074] Two decoding paths are set, the main path SegHead generates a regular prediction, and the reconstruction path ReconHead outputs a pseudo-supervised prediction; after the fusion of the output features of the three branches (i.e., the detail branch, the boundary branch, and the global branch), the fused features F fused are sent to two decoders to obtain the corresponding prediction probabilities P seg (x,y) , P recon (x,y) :

[0075]

[0076] Subsequently, the KL divergence between the two prediction probabilities is calculated to obtain the residual map R KL (x,y) :

[0077]

[0078] After that, the local filter is used to smooth the residual map to avoid the interference caused by isolated points:

[0079]

[0080] Then, the mask is dynamically generated:

[0081]

[0082] In the formula: The 90% quantile of the whole image smooth residual is represented, that is, only the top 10% residual area is reserved for repair; ReLU The nonlinear activation function is represented;

[0083] The self-repair loss is constructed L repair Taking the reconstruction path as the target, the main path performs reverse repair in the high residual area:

[0084]

[0085] Finally, the optimization loss model is constructed:

[0086]

[0087] In the formula: CE The cross-entropy loss function is used to improve the pixel-level classification accuracy; IoU The IoU loss function is used to optimize the overlap between the predicted area and the real mask; The weight coefficient of the self-repair loss in the total loss is represented; L repair It is used to guide the main prediction path to pay attention to the unstable area and correct the prediction deviation.

[0088] Based on the further optimization of the above scheme, the step S3 is specifically: training and verification are performed on a single GPU, CUDNN Benchmark is enabled to improve performance, and 6 concurrent data loading threads are set; the pre-training weight is initialized based on ImageNet, OHEM (Online Hard Example Mining) is enabled to improve the ability of difficult example learning, wherein: the OHEM threshold is set to 0.9, and the number of reserved samples is 131072; the input and the benchmark size are 512x512, the number of GPU processing samples per batch is 6, the total training round is 300, the optimizer is SGD, the initial learning rate is 0.005, the weight decay is 0.0005, and the momentum is 0.9; data augmentation is enabled, including random flipping and multi-scale training (Multi-Scale).

[0089] The following are the effects of the technical scheme of the present application:

[0090] The application simultaneously plays the representation ability and structural robustness of the model through the hybrid feature framework composed of the convolutional neural network (CNN) and the SwiftFormer encoder, has good engineering universality, and can adapt to the segmentation task requirements of different remote sensing data (city, countryside, natural features, etc.). Meanwhile, the application designs a multi-scale feature extraction module based on the CNN and the backbone network of the SwiftFormer encoder, realizes the decoupling of semantic modeling and boundary perception through three parallel branches of boundary, global and detail, introduces an information mutual induction mechanism in the structures of the detail branch, the boundary branch and the global semantic branch, realizes mutual guidance and feedback update through dynamic guidance of information flow, breaks the static isolation between the branches, makes the fusion of the branch features in the semantic space more sufficient, improves the consistency and expression diversity of the feature fusion semantics, and significantly improves the boundary analysis and small target restoration ability; then, through the regional perception gating mechanism, the regional vector constructed by combining the structural complexity, the semantic confidence and the texture change is used to realize the local adaptive regulation of the feature flow, the adaptive feature reservation strategy can be used for different regions, so that the response ability of the model to the segmentation difficulty of different regions is enhanced. In addition, through the double decoding paths of the main prediction path and the pseudo reconstruction path, the application constructs a prediction stability residual graph and dynamically generates a repair mask, implements pseudo-supervised optimization on uncertain regions, and effectively improves the fault tolerance of the model to complex or weakly supervised samples.

[0091] Unlike some enhanced methods that require complex auxiliary structures or multi-model reasoning, the self-repairing module proposed in the application is only activated in the training stage and can be completely removed in the inference stage, the model structure is simple, the deployment cost is low, and it is suitable for large-scale and efficient application in the remote sensing scene. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 The structural block diagram of the image segmentation method in the embodiment of the application.

[0093] Figure 2 The overall flowchart of the image segmentation method in the embodiment of the application.

[0094] Figure 3 The flowchart of the convolutional neural network and the SwiftFormer encoder in the embodiment of the application.

[0095] Figure 4 The flowchart of the hierarchical feature fusion module for realizing multi-scale semantic information fusion in the embodiment of the application.

[0096] Figure 5 The flowchart of the residual reconstruction learning and self-repairing mechanism in the embodiment of the application.

[0097] Figure 6 The improved flowchart of another embodiment of the application.

[0098] Figure 7 Comparison effect diagram of the image segmentation method of the present application and the existing segmentation method. DETAILED DESCRIPTION

[0099] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0100] Embodiment 1:

[0101] An image segmentation method based on deep learning remote sensing image, the overall structure block diagram is as shown in Figure 1 , comprising:

[0102] Step S1, collecting remote sensing images, using remote sensing images to construct a data set, and completing data preprocessing; the remote sensing images use LoveDA data set, which contains large-scale and high-resolution remote sensing images, and provides a total of 5987 remote sensing images with a resolution of 1024x1024 and their corresponding pixel-level semantic labels; the data set covers multiple regions in China, covering cities, rural areas and mixed areas, has significant urban-rural distribution differences, and can truly reflect the common inter-field bias problems in actual remote sensing applications;

[0103] The data set construction using remote sensing images is specifically: the data set is cropped according to a step size of 512 pixels to obtain an image of 512x512 pixels, the image is then preprocessed, the class with a pixel value of 0 is ignored, and the remaining seven classes of ground objects include background (Background), building (Building), road (Road), water body (Water), barren soil (Barren), farmland (Agriculture) and forest land (Forest), and the image is further operated to enhance the data volume to improve the generalization ability of the model, such as random rotation, translation, flipping, cropping, etc.; then the data set is divided into a training set, a validation set and a test set (wherein the training set, the validation set and the test set can be randomly divided in a ratio of 7:2:1).

[0104] Step S2, constructing a remote sensing image semantic segmentation model (see Figure 2 ): first, a convolutional neural network (CNN) and a SwiftFormer encoder are constructed, specifically:

[0105] A cascaded SwiftFormer encoder is used, and this architecture gradually extracts multi-scale features at different network stages through an efficient attention mechanism while maintaining the ability of global context modeling; this architecture realizes joint modeling of local detailed features and long-distance dependency relationships;

[0106] For a given input feature , the SwiftFormer encoder first adopts a 3x3 depthwise separable convolution ( DWConv ) to capture spatial structure, and then transforms the channel dimension by a 1x1 convolution ( Conv ) to compress the feature while preserving local structure information (see Figure 3 (a)), specifically:

[0107]

[0108] Meanwhile, the efficient additive attention mechanism (EAA) of the SwiftFormer encoder significantly reduces the computational cost while maintaining the ability to capture global context information, see Figure 3 (b), and the specific process is:

[0109] First, for the input feature , its query matrix Q and key matrix K are obtained through two linear projections:

[0110]

[0111] where denotes the learning parameters; n denotes the sequence length, d denotes the embedding dimension;

[0112] Then, the scaled dot-product between each query matrix Q and the learned vector is calculated to generate global attention scores for each position:

[0113]

[0114] After that, the global query vector q is obtained by aggregating the query matrix using the global attention scores:

[0115]

[0116] The global query vector is then fused with the key matrix through element-wise multiplication to encode the interaction between all spatial positions:

[0117]

[0118] where Linear denotes linear transformation; denotes element-wise multiplication;

[0119] Finally, the global context features are combined with the normalized query representation through a residual connection to generate the final output of the encoder:

[0120]

[0121] wherein: Norm () denotes a normalization function.

[0122] Then, a three-branch channel is constructed, and a three-branch mutual guidance fusion mechanism is used to realize dynamic interaction modeling of local and global information, specifically:

[0123] First, a unified feature map is extracted from the backbone network of the convolutional neural network and the SwiftFormer encoder F shared It is input into three branches respectively:

[0124] The detail branch ( Detail Branch ): is used to extract texture details;

[0125] The boundary branch ( Boundary Branch ): is used to extract edge line information;

[0126] The global branch ( Global Branch ): is used to extract context;

[0127] wherein: denotes a feature mapping function of the detail branch, composed of local extraction modules such as convolution, ReLU , BN , etc. denotes a boundary feature extraction module, including Sobe l convolution, edge enhancement layer, etc. denotes Transformer , a global modeling module composed of attention mechanisms, biased towards semantic grouping and region perception.

[0128] After that, mutual guidance in the three branches is established through the guidance map respectively:

[0129]

[0130] wherein: , , , , , denote the global branch guiding the detail branch map, the boundary branch guiding the detail branch map, the global branch guiding the boundary branch map, the detail branch guiding the boundary branch map, the detail branch guiding the global branch map, and the boundary branch guiding the global branch map, respectively. denotes a Sigmoid activation function with a range of [0, 1]; denotes channel dimension concatenation;

[0131] Finally, the outputs of the detail branch, the boundary branch and the global branch are updated respectively by guiding maps:

[0132]

[0133] wherein: , , denote the updated feature maps of the detail branch, the boundary branch and the global branch respectively; , , , , , denote learning parameters.

[0134] After that, a region-aware gating mechanism is adopted to realize region-adaptive feature information flow regulation, specifically:

[0135] First, a region structure feature extraction module is set, each pixel in the region has a set of values, which reflect whether the current pixel position is in the edge, corner or texture dramatic change area through the structural complexity S(x,y) :

[0136]

[0137]

[0138] wherein: G x denotes the gradient of the point I(x,y) in the x axis direction; G y denotes the gradient of the point I(x,y) in the y axis direction;

[0139] The accuracy of prediction is determined by the semantic confidence C(x,y) of the current region:

[0140]

[0141] wherein: Softmax() denotes Softmax function; F pred (x,y) denotes the predicted feature vector of the model at the pixel point (x,y) ;

[0142] By local gray scale standard deviation T(x,y) Discriminate the texture intensity of the current region:

[0143]

[0144] In the formula: I i Indicates the pixel value of the small window centered on (x,y) ; Indicates the pixel mean value; N Indicates the number of pixels of the current region;

[0145] The region perception gating module is deployed in the detail branch, the boundary branch and the global branch, and the structural complexity S(x,y) , the semantic confidence C(x,y) and the texture intensity T(x,y) are adaptively selected, and the information passing weight is dynamically adjusted through the gating function, so that the region level difference modeling is completed before feature fusion, and the fusion effect and the segmentation accuracy are improved, specifically:

[0146] Detail branch:

[0147]

[0148]

[0149] Boundary branch:

[0150]

[0151]

[0152] Global branch:

[0153]

[0154]

[0155] In the formula: W d , W b , W g respectively indicate the weight parameters corresponding to the detail branch, the boundary branch and the global branch; B d , B b , B g respectively indicate the bias terms corresponding to the detail branch, the boundary branch and the global branch; R d (x,y) , Rb (x,y) 、 R g (x,y) Respectively represent the corresponding structural complexity in the detail branch, boundary branch, and global branch S(x,y) , semantic confidence C(x,y) and texture strength T(x,y) Combined into a regional structure information vector.

[0156] Then, a hierarchical feature fusion module is used to achieve multi-scale semantic information fusion, see Figure 4 As shown, specifically:

[0157] Introduce the hierarchical feature fusion module at the 1 / 8 and 1 / 16 resolution stages of the backbone network: Assume that the input features of the current backbone network, that is, the global branch features, are ,in, 、 Represent the edge semantic features of the boundary branch and the local structural features of the detail branch respectively, then:

[0158]

[0159] Where: AvgPool represents average pooling; MaxPool represents maximum pooling; MLP represents a multilayer perceptron with shared weights;

[0160] The channel attention weights are multiplied element-wise to regenerate the global features:

[0161]

[0162] A spatial attention mechanism is used to highlight the structural salient areas of detail branches:

[0163]

[0164]

[0165] Finally, the attention enhancement feature 、 Along the channel dimension and input features F i-1 Splice to form fused input features:

[0166] .

[0167] Finally, the residual reconstruction learning for self-repair is adopted to complete the self-recognition of the model, predict errors, and perform self-repair. See Figure 5 As shown in the following specific embodiments:

[0168] Two decoding paths are set, a main path SegHead generates a regular prediction, and a reconstruction path ReconHead outputs a pseudo-supervised prediction; after the feature fusion of the three branches (i.e., the detail branch, the boundary branch, and the global branch), the fused features F fused are simultaneously sent to two decoders to obtain corresponding prediction probabilities P seg (x,y) , P recon (x,y) :

[0169]

[0170] Subsequently, the KL divergence between the two prediction probabilities is calculated to obtain a residual map R KL (x,y) :

[0171]

[0172] Then, the residual map is smoothed by using a local filter to avoid interference caused by isolated points with no difference:

[0173]

[0174] A mask is dynamically generated again:

[0175]

[0176] In the formula: represents the 90% quantile value of the full map smoothed residual, that is, only the highest 10% residual area is reserved for repair; ReLU represents a nonlinear activation function;

[0177] The self-repair loss is constructed L repair The main path performs reverse repair in the high residual area with the reconstruction path as the target:

[0178]

[0179] Finally, an optimization loss model is constructed:

[0180]

[0181] In the formula: CE Cross-entropy loss function for improving pixel-level classification accuracy; IoU IoU loss function for optimizing the overlap between the predicted region and the real mask; Self-repairing loss weight coefficient in the total loss; L repair For guiding the main prediction path to pay attention to unstable areas and correcting prediction deviation.

[0182] Step S3, using the data set in step S1 to train, verify and optimize the remote sensing image semantic segmentation model constructed in step S2, complete the model construction; wherein, the training and verification are carried out on single block GPU, CUDNN Benchmark is enabled to improve the performance, 6 concurrent data loading threads are set; the pre-training weight is initialized based on ImageNet, OHEM (Online Hard Example Mining) is enabled to improve the difficult example learning ability, wherein: the OHEM threshold is set to 0.9, and the number of reserved samples is 131072; the input and the benchmark size are 512x512, the number of GPU processing samples per batch is 6, the total training round is 300, the optimizer is SGD, the initial learning rate is 0.005, the weight decay is 0.0005, and the momentum is 0.9; data enhancement is enabled, including random flip and multi-scale training (Multi-Scale).

[0183] Test size and benchmark size: 512x512, multi-scale and flip test strategy is enabled to improve the inference stability and accuracy, test batch size: 6.

[0184] Embodiment 2:

[0185] As a preferred embodiment of the scheme of the application, on the basis of embodiment 1, in order to further tap the up-down dependency relationship under different receptive fields, in the hierarchical feature fusion module in step S2, a multi-scale splitting and fusion mechanism is further introduced, specifically:

[0186] For the input feature map , it is divided into four channel groups, as shown in Figure 6 (a) shown:

[0187]

[0188] Each group is convolved using multiple sizes of convolution kernels to capture different receptive field ranges, specifically:

[0189]

[0190] Channel group Subsequently, the responses from the four convolution paths are spliced to form a multi-scale representation, specifically:

[0191]

[0192] Finally, the enhanced sub-features are spliced to generate the output of the fusion module:

[0193] .

[0194] To evaluate the semantic segmentation method in the embodiment, the semantic segmentation method in the embodiment is compared with existing methods (including DDRNet, BiSeNetV2 and HRNet) in performance, and the comparison results are shown in the following table:

[0195]

[0196] As shown in the above table, the model in the embodiment has achieved significant performance improvement on all categories, and the overall mIoU reaches 0.627, which is about 14.8% higher than that of the existing technology HRNet; at the same time, the model in the embodiment shows significant advantages in building, road and water body and other key heterogeneous land cover types, and the intersection over union (IoU) scores reach 0.647, 0.651 and 0.758 respectively, indicating that the model in the embodiment has obvious advantages in maintaining the consistency of fine-grained details and global semantics.

[0197] To more intuitively show the performance advantages of the method in the embodiment in the semantic segmentation task, Figure 7 The prediction results of the method of the embodiment and DDRNet and BiSeNetV2 are shown in the following figure: Figure 7 As shown in the figure, DDRNet and BiSeNetV2 have obvious cracking and deformation in the road area, and it is difficult to accurately restore its true form (especially in complex structure areas such as curves and intersections), and the boundary is prone to blur and fusion phenomenon; in addition, these two methods have different degrees of contour distortion and misclassification problems in the edge of the building, and cannot effectively capture the detailed structural features of the compact urban blocks. In contrast, the road prediction generated by the method in the embodiment is not only continuous, but also can effectively follow the curve layout, avoiding the local cracking and boundary offset problems in other methods; in addition, the predicted building area also exhibits clear and coherent contour lines, reflecting the excellent ability of the model in multi-scale feature alignment and fine-grained boundary modeling.

Claims

1. A deep learning based method for image segmentation of remote sensing images, characterized in that: Comprise: Step S1, collecting remote sensing images, using remote sensing images to construct a data set, and completing data preprocessing; Step S2, constructing a remote sensing image semantic segmentation model: first, a convolutional neural network and a SwiftFormer encoder are constructed, specifically: A cascaded SwiftFormer encoder is used, which gradually extracts multi-scale features at different network stages through an efficient attention mechanism while maintaining the ability to model global context; For a given input feature The SwiftFormer encoder first adopts a 3x3 depthwise separable convolution to capture spatial structure, followed by a 1x1 convolution to transform the channel dimension, thus compressing the feature while preserving local structure information, specifically: At the same time, the efficient additive attention mechanism of the SwiftFormer encoder greatly reduces the computational cost while maintaining the ability to capture global context information, and the specific process is as follows: First, for the input features , its query matrix Q and key matrix K are obtained by two linear projections: In the formula: represents a learning parameter; n represents a sequence length, d represents an embedding dimension; Subsequently, the scaled dot product between each query matrix Q and the learned vector is computed to generate global attention scores for each position : After that, the global query vector is obtained by aggregating the query matrix using global attention scores q : Then, the global query vector and the key matrix are fused by element-wise multiplication to encode the interaction between all spatial positions: In the formulae: Linear denotes a linear transformation; denotes an element-wise multiplication; Finally, the global context features are combined with the normalized query representation through residual connection to generate the final output of the encoder: In the formulae: Norm () denotes a normalization function; Then, a three-branch channel is constructed, and a three-branch mutual guidance fusion mechanism is used to realize dynamic interaction modeling of local and global information; then, a region-aware gating mechanism is used to realize region-adaptive feature information flow regulation; Then, a hierarchical feature fusion module is used to realize multi-scale semantic information fusion; finally, a residual reconstruction learning and self-repair mechanism is used to complete model self-recognition, predict errors, and perform self-repair; Step S3, using the data set in step S1 to train, verify and optimize the remote sensing image semantic segmentation model constructed in step S2, and completing model construction. 2.The method of claim 1, wherein: The remote sensing images in step S1 use the LoveDA data set, which contains a large number of high-resolution remote sensing images, providing a total of 5987 remote sensing images with a resolution of 1024x1024 and their corresponding pixel-level semantic labels; using remote sensing images to construct a data set specifically involves: cropping the data set by 512 pixels as a step to obtain an image of 512x512 pixels, then preprocessing the image, ignoring the class with a pixel value of 0, leaving the remaining seven types of ground objects including background, building, road, water, barren soil, farmland and forest, and enhancing the image; then the data set is divided into training set, validation set and test set. 3.The method of claim 2, wherein: In step S2, "constructing a three-branch channel and using a three-branch mutual guidance fusion mechanism to realize dynamic interaction modeling of local and global information" specifically involves: First, a unified feature map is extracted from the backbone network of the convolutional neural network and the SwiftFormer encoder F shared It is input into three branches respectively: Detail branch: for extracting texture details; Boundary branch: for extracting edge line information; Global branches: for extracting context; In the formula: represents the feature mapping function of the detail branch; represents the boundary feature extraction module; represents Transformer , the global modeling module composed of the attention mechanism, biased towards semantic grouping and region perception; Then, the mutual guidance in the three branches is established through the guide map: In the formula: , , , , , respectively represent a global branch guidance detail branch graph, a boundary branch guidance detail branch graph, a global branch guidance boundary branch graph, a detail branch guidance boundary branch graph, a detail branch guidance global branch graph, and a boundary branch guidance global branch graph; represents a Sigmoid activation function, the range of which is [0, 1]; represents a channel dimension splicing; Finally, the outputs of the detail branch, the boundary branch and the global branch are updated through the guide map: In the formula: , , respectively represent the updated detail branch feature map, the boundary branch feature map, and the global branch feature map; , , , , , respectively represent learning parameters.

4. The method of claim 3, wherein the method further comprises: In step S2, "using a region-aware gating mechanism to realize region-adaptive feature information flow regulation" specifically involves: First, set the region structure feature extraction module, each pixel in the region has a set of values, through the structure complexity S (x,y) to reflect whether the current pixel position is in the edge, corner or texture dramatic change region index: In the formula: G x representative point I(x,y) In x gradient in the axis direction; G y representative point I(x,y) In y gradient in the axis direction; determining the accuracy of the prediction by the semantic confidence of the current region C(x,y) : In the formula: Softmax() represents Softmax a function; F pred (x,y) represents the predicted feature vector of the model at the pixel point (x,y) ; By local gray scale standard deviation T(x,y) discriminate the texture intensity of the current region: wherein: I i represents a pixel value of a small window centered at (x,y) represents a pixel mean value; N represents the number of pixels of the current region;​ The region perception gating module is deployed in the detail branch, the boundary branch and the global branch, and structural complexity is adaptively selected S(x,y) , semantic confidence C(x,y) and texture intensity T(x,y) , and information passing weights are dynamically adjusted through a gating function, so that region level difference modeling is completed before feature fusion, and fusion effect and segmentation accuracy are improved. Detail branch: Boundary branch: Global branch: In the formula: W d , W b , W g respectively represent the weight parameters corresponding to the detail branch, the boundary branch and the global branch; B d , B b , B g respectively represent the bias terms corresponding to the detail branch, the boundary branch and the global branch; R d (x,y) , R b (x,y) , R g (x,y) respectively represent the structure complexity S(x,y) , the semantic confidence C(x,y) and the texture intensity T (x,y) in the detail branch, the boundary branch and the global branch combined into the regional structure information vector.

5. The method of claim 4, wherein the method is based on deep learning. The step S2 of "adopting a hierarchical feature fusion module to realize multi-scale semantic information fusion" specifically includes: Introducing a hierarchical feature fusion module in the 1 / 8 and 1 / 16 resolution stages of the backbone network: assuming that the input feature of the current backbone network, i.e., the global branch feature, is wherein, , respectively represent the edge semantic feature of the boundary branch and the local structure feature of the detail branch, then: wherein: AvgPool denotes average pooling; MaxPool denotes max pooling; MLP denotes a multi-layer perceptron with shared weights; The channel attention weight is regenerated by element-wise multiplication: The spatial attention mechanism is adopted to highlight the structural saliency area of the detail branch: Finally attention augmented features , along the channel dimension with input features F i-1 are concatenated to form fused input features: 。 6. The method of claim 5, wherein the method is based on deep learning. The step S2 of "adopting a residual reconstruction learning and self-repair mechanism to complete self-identification of the model, predict errors and perform self-repair" specifically includes: Two decoding paths are set, a main path SegHead , generating a regular prediction, a reconstruction path ReconHead , outputting a pseudo-supervised prediction; After the three branch output features are fused, the fused features are input into a fully connected layer to obtain the final prediction result F fused Meanwhile, the two paths are input into two decoders respectively to obtain the corresponding prediction probabilities P seg (x,y) 、 P recon (x,y) : Subsequently, the KL divergence between the two prediction probabilities is calculated, obtaining a residual map R KL (x,y) : Then, a local filter is used to smooth the residual map to avoid interference caused by isolated points with no difference: A mask is dynamically generated: In the formula: represents the 90th percentile of the full image smooth residual, i.e., only the top 10% of the residual regions are reserved for repair; ReLU represents a nonlinear activation function; Constructing self-repair loss L repair With the goal of reconstructing the path, the main path performs reverse repair in the high residual area: Finally, an optimization loss model is constructed: In the formula: CE a cross-entropy loss function for improving pixel-level classification accuracy; IoU a IoU loss function for optimizing the overlap between the predicted region and the real mask; a weight coefficient of the self-repairing loss in the total loss; L repair for guiding the main prediction path to pay attention to unstable regions and correcting prediction deviation.

7. The method of claim 1, wherein the method is based on deep learning. The step S3 specifically includes: training and verification are performed on a single GPU, CUDNN Benchmark is enabled to improve performance, and 6 concurrent data loading threads are set; pre-training weights are initialized based on ImageNet, and OHEM is enabled, wherein the OHEM threshold is set to 0.9, and the number of retained samples is 131072; the input and the reference size are 512*512, the number of samples processed by the GPU per batch is 6, the total number of training rounds is 300, the optimizer is SGD, the initial learning rate is 0.005, the weight decay is 0.0005, and the momentum is 0.9; data augmentation is enabled, including random flipping and multi-scale training.

Images