A remote sensing image segmentation method and system based on multi-scale context enhancement

Abstract

The application discloses a kind of based on multiscale context enhancement remote sensing image segmentation method and system, it is related to remote sensing image processing technical field, including: obtaining remote sensing image to be handled and input to global-local fusion network, obtain multiscale feature;Based on multiscale feature input to cascade fusion network, obtain first processing feature, second processing feature, third processing feature and integrated feature;Based on integrated feature input to multiscale context fusion network obtains multiscale semantic feature;Based on first processing feature, second processing feature and third processing feature respectively with multiscale semantic feature fusion, corresponding first fusion feature, second fusion feature and third fusion feature are obtained and are fused, obtain cascade feature;Based on cascade feature input to result output network, obtain the segmentation prediction result of remote sensing image.Effectively improve the segmentation precision of complex ground object boundary and small target in remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and more particularly to a remote sensing image segmentation method and system based on multi-scale context enhancement. BACKGROUND

[0002] With the rapid development of remote sensing technology, the wide application of high-resolution, multi-spectral and even hyperspectral satellite and unmanned aerial vehicle platforms provides rich spatial-spectral information for land cover type identification, environmental monitoring, urban planning, agricultural management and other fields. However, remote sensing image segmentation always faces the challenges brought by multi-scale targets, diversified ground object structures and complex background interference.

[0003] In recent years, convolutional neural networks (CNN) and deep learning techniques have made breakthrough progress in the field of natural image semantic segmentation and have been gradually applied to remote sensing image segmentation tasks. However, when facing remote sensing images, such methods still have the following bottlenecks: first, the fusion of shallow high-resolution features and deep high-semantic features is insufficient, and when the input resolution is very high and the target size span is large, the shallow details are difficult to fully utilize; second, the fixed shape of the receptive field limits the response ability to irregular targets and small targets, resulting in roads, canals, small buildings and the like being easily overwhelmed by noise; third, long-range dependency relationships in large-scale scenes are difficult to capture, and the model is difficult to balance local details and global structure at the same time. In order to overcome the above defects, researchers try to introduce attention mechanisms, multi-scale pyramid pooling or dilated convolution to expand the receptive field, in order to improve the perception ability of different scale features. However, these methods usually have large computational overhead, large number of parameters, and limited processing efficiency for high-resolution remote sensing images; at the same time, the accurate segmentation of small targets is still unsatisfactory. In addition, existing multi-scale fusion is usually performed in the spatial or channel dimension, but lacks sequential global-local collaborative modeling, making it difficult to achieve seamless transition from pixel-level details to scene-level structure in a unified framework. In view of the complex and variable ground feature characteristics and large-scale scene structure in remote sensing images, researchers have also carried out global modeling exploration based on frequency domain or state space. For example, using Fourier transform or wavelet transform to map the image to the frequency domain for feature extraction can effectively capture macro textures and periodic structures; state space models simulate long-range dependencies through recursive scanning. However, these techniques often focus on global information or emphasize single modal fusion, and pay insufficient attention to local details and cross-scale interactions, making it difficult to meet the high-precision and high-efficiency requirements of remote sensing segmentation.

[0004] Therefore, how to effectively improve the segmentation accuracy of complex ground object boundaries and small targets in remote sensing images based on the deep coupling of local details and global semantic information is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the application provides a remote sensing image segmentation method and system based on multi-scale context enhancement.

[0006] To achieve the above object, the application adopts the following technical scheme:

[0007] A remote sensing image segmentation method based on multi-scale context enhancement comprises the following steps:

[0008] Acquire a remote sensing image to be processed and input the remote sensing image to a global-local fusion network to obtain multi-scale features;

[0009] Input the multi-scale features to a cascade fusion network to obtain first processing features, second processing features, third processing features and integrated features;

[0010] Input the integrated features to a multi-scale context fusion network to obtain multi-scale semantic features;

[0011] Fuse the first processing features, the second processing features and the third processing features with the multi-scale semantic features respectively to obtain first fusion features, second fusion features and third fusion features respectively;

[0012] Fuse the first fusion features, the second fusion features and the third fusion features to obtain cascade features;

[0013] Input the cascade features to a result output network to obtain a segmentation prediction result of the remote sensing image.

[0014] Preferably, the global-local fusion network comprises a first fusion unit, a second fusion unit, a third fusion unit and a fourth fusion unit.

[0015] Input the remote sensing image to the first fusion unit to obtain initial extraction features;

[0016] Input the initial extraction features to the second fusion unit to obtain shallow features;

[0017] Input the shallow features to the third fusion unit to obtain middle features;

[0018] Input the middle features to the fourth fusion unit to obtain deep features;

[0019] The shallow features, the middle features and the deep features jointly constitute the multi-scale features.

[0020] Preferably, the first fusion unit, the second fusion unit, the third fusion unit and the fourth fusion unit are structurally identical, each comprising:

[0021] a down-sampling layer, a first normalization layer, a domain attention layer, a first two-dimensional selective scanning layer, a first convolutional layer, a second normalization layer and a first feed-forward neural network layer;

[0022] based on the input feature input to the down-sampling layer, a first extraction feature is obtained;

[0023] based on the first extraction feature input to the first normalization layer and the domain attention layer in sequence, a second extraction feature is obtained;

[0024] based on the second extraction feature input to the first two-dimensional selective scanning layer, a third extraction feature is obtained;

[0025] based on the third extraction feature input to the first convolutional layer after being fused with the second extraction feature, a fourth extraction feature is obtained;

[0026] based on the fourth extraction feature being fused with the first extraction feature, a process feature is obtained;

[0027] based on the process feature input to the second normalization layer and the first feed-forward neural network layer in sequence, a fifth extraction feature is obtained;

[0028] based on the fifth extraction feature being fused with the process feature, an output feature is obtained.

[0029] Preferably, the cascaded fusion network comprises: a second convolutional layer, a first up-sampling layer, a third convolutional layer, a second up-sampling layer, a fourth convolutional layer, a fusion layer and a fifth convolutional layer;

[0030] based on the deep feature input to the second convolutional layer and the first up-sampling layer in sequence, the first processing feature is obtained;

[0031] based on the middle layer feature input to the third convolutional layer and the second up-sampling layer in sequence, the second processing feature is obtained;

[0032] based on the shallow layer feature input to the fourth convolutional layer, the third processing feature is obtained;

[0033] based on the first processing feature, the second processing feature and the third processing feature input to the fusion layer and then input to the fifth convolutional layer, the integrated feature is obtained.

[0034] Preferably, the multi-scale context fusion network comprises: a multi-scale splicing unit, a feature enhancement unit and a feature recovery unit;

[0035] Based on the integrated feature input to the multi-scale splicing unit, a multi-scale splicing feature is obtained.

[0036] Based on the multi-scale splicing feature input to the feature enhancement unit, a multi-scale enhanced feature is obtained.

[0037] Based on the multi-scale enhanced feature and the integrated feature input to the feature recovery unit, the multi-scale semantic feature is obtained.

[0038] Preferably, the multi-scale splicing unit comprises a third up-sampling layer, a sixth convolutional layer, an expanded convolutional layer, a seventh convolutional layer, a fourth up-sampling layer, an eighth convolutional layer, and a windmill convolutional layer.

[0039] Based on the integrated feature input to the third up-sampling layer, the sixth convolutional layer, and the expanded convolutional layer in sequence, a first scale feature is obtained.

[0040] Based on the integrated feature input to the seventh convolutional layer and the fourth up-sampling layer in sequence, a second scale feature is obtained.

[0041] Based on the integrated feature input to the eighth convolutional layer and the windmill convolutional layer in sequence, a third scale feature is obtained.

[0042] Based on the first scale feature, the second scale feature, and the third scale feature splicing, the multi-scale splicing feature is obtained.

[0043] Preferably, the feature enhancement unit comprises a dynamically deformable convolutional layer, a second two-dimensional selective scanning layer, and a second feedforward neural network layer.

[0044] Based on the multi-scale splicing feature input to the dynamically deformable convolutional layer, the second two-dimensional selective scanning layer, and the second feedforward neural network layer in sequence, the multi-scale enhanced feature is obtained.

[0045] The feature recovery unit comprises a depth separable convolutional layer, a fifth up-sampling layer, and a ninth convolutional layer.

[0046] Based on the integrated feature and the multi-scale enhanced feature fusion, a complementary fusion feature is obtained.

[0047] Based on the integrated feature input to the depth separable convolutional layer, a channel weight feature is obtained.

[0048] Based on the complementary fusion feature and the channel weight feature fusion and then input to the fifth up-sampling layer and the ninth convolutional layer in sequence, the multi-scale semantic feature is obtained.

[0049] Preferably, the result output network comprises a multi-layer perception, a sixth up-sampling layer, and a segmentation head.

[0050] The cascade feature is sequentially input to the multi-layer perception, the sixth up-sampling layer and the segmentation head to obtain the segmentation prediction result.

[0051] Preferably, the global-local fusion network, the cascade fusion network, the multi-scale context fusion network and the result output network jointly constitute an image segmentation model.

[0052] The image segmentation model is optimized and trained based on a total loss function to obtain a trained image segmentation model.

[0053] The trained image segmentation model is input based on a to-be-processed remote sensing image to obtain a corresponding segmentation prediction result.

[0054] The total loss function comprehensively considers cross-entropy loss, Dice loss and boundary perception loss.

[0055] A remote sensing image segmentation system based on multi-scale context enhancement comprises a first feature extraction module, a second feature extraction module, a third feature extraction module, a first feature fusion module, a second feature fusion module and a result output module.

[0056] The first feature extraction module is configured to acquire a to-be-processed remote sensing image and input the image to a global-local fusion network to obtain multi-scale features.

[0057] The second feature extraction module is configured to input the multi-scale features to a cascade fusion network to obtain first processing features, second processing features, third processing features and integrated features.

[0058] The third feature extraction module is configured to input the integrated features to a multi-scale context fusion network to obtain multi-scale semantic features.

[0059] The first feature fusion module is configured to fuse the first processing features, the second processing features and the third processing features with the multi-scale semantic features respectively to obtain first fusion features, second fusion features and third fusion features.

[0060] The second feature fusion module is configured to fuse the first fusion features, the second fusion features and the third fusion features to obtain a cascade feature.

[0061] The result output module is configured to input the cascade feature to a result output network to obtain a segmentation prediction result of the remote sensing image.

[0062] Compared with the prior art, the remote sensing image segmentation method and system based on multi-scale context enhancement provided by the present disclosure have the following advantages:

[0063] 1、The present application realizes seamless connection of cross-scale features by capturing high-resolution details (such as road edges and building outlines) in shallow layers and integrating wide-area semantics (such as farmland texture and urban structure) in deep layers, greatly improving the segmentation performance of multi-class and different size ground objects.

[0064] 2、The present application has stronger sensitivity and robustness to small targets (such as water channels and narrow roads) and irregular ground object boundaries in complex backgrounds of remote sensing images by introducing neighborhood attention and two-dimensional selective scanning blocks, combining distance weighted boundary loss and Dice loss optimization, and significantly reducing the problems of missed and over-segmentation.

[0065] 3、The present application realizes deep interaction of local and global information at each level by connecting modules such as normalization, neighborhood attention, and selective scanning blocks; at the same time, the designed multi-scale context fusion network realizes adaptive response to heterogeneous ground object regions through multi-branch parallel decomposition, dynamic deformable convolution, and channel re-labeling technology; finally, based on the optimization strategy of composite loss function, the cross-entropy, Dice, and distance weighted boundary loss are combined to effectively improve the segmentation accuracy of complex texture and small targets.

[0066] 4、The present application not only has higher accuracy than existing multi-scale segmentation networks, but also has significant advantages in computational efficiency and model generalization ability, providing strong technical support for remote sensing image automatic analysis and application. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only the embodiments of the present application, and for those skilled in the art, without creative labor, other drawings can also be obtained according to the provided drawings.

[0068] Figure 1 A flow chart of a remote sensing image segmentation method based on multi-scale context enhancement is provided.

[0069] Figure 2 A global-local fusion network structure diagram is provided.

[0070] Figure 3 A multi-scale context fusion network structure diagram is provided.

[0071] Figure 4 A remote sensing image segmentation system structure diagram based on multi-scale context enhancement is provided. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] Example 1

[0074] like Figure 1 As shown, this embodiment of the invention discloses a remote sensing image segmentation method based on multi-scale context enhancement, including:

[0075] The remote sensing image to be processed is acquired and input into a global-local fusion network to obtain multi-scale features;

[0076] Based on the multi-scale feature input to the cascaded fusion network, the first processed feature, the second processed feature, the third processed feature and the integrated feature are obtained;

[0077] Multi-scale semantic features are obtained by integrating the features and inputting them into a multi-scale context fusion network.

[0078] Based on the fusion of the first processing feature, the second processing feature, and the third processing feature with the multi-scale semantic feature, the first fusion feature, the second fusion feature, and the third fusion feature are obtained respectively;

[0079] The cascaded features are obtained by fusing the first, second, and third fusion features.

[0080] The segmentation prediction results of remote sensing images are obtained by using a cascaded feature input to the output network.

[0081] Example 2

[0082] This invention discloses a remote sensing image segmentation method based on multi-scale context enhancement, comprising:

[0083] The remote sensing image to be processed is acquired and input into a global-local fusion network to obtain multi-scale features.

[0084] Preferably, the remote sensing image to be processed is preprocessed before input, including denoising, geometric correction and spectral correction, and the preprocessed remote sensing image is input into the global-local fusion network.

[0085] Preferred, such as Figure 2 As shown, the global-local fusion network includes: a first fusion unit, a second fusion unit, a third fusion unit, and a fourth fusion unit;

[0086] Based on the remote sensing image input to the first fusion unit, an initial extraction feature is obtained;

[0087] Based on the initial extraction feature input to the second fusion unit, a shallow layer feature is obtained;

[0088] Based on the shallow layer feature input to the third fusion unit, a middle layer feature is obtained;

[0089] Based on the middle layer feature input to the fourth fusion unit, a deep layer feature is obtained;

[0090] The shallow layer feature, the middle layer feature and the deep layer feature jointly constitute a multi-scale feature.

[0091] Preferably, the first fusion unit, the second fusion unit, the third fusion unit and the fourth fusion unit are the same in structure, and each includes:

[0092] a down-sampling layer, a first normalization layer, a domain attention layer, a first two-dimensional selective scanning layer, a first convolutional layer, a second normalization layer and a first feedforward neural network layer;

[0093] Based on the input feature input to the down-sampling layer, a first extraction feature is obtained;

[0094] Based on the first extraction feature input to the first normalization layer and the domain attention layer in sequence, a second extraction feature is obtained;

[0095] Based on the second extraction feature input to the first two-dimensional selective scanning layer, a third extraction feature is obtained;

[0096] Based on the third extraction feature input to the first convolutional layer after being fused with the second extraction feature, a fourth extraction feature is obtained;

[0097] Based on the fourth extraction feature being fused with the first extraction feature, a process feature is obtained;

[0098] Based on the process feature input to the second normalization layer and the first feedforward neural network layer in sequence, a fifth extraction feature is obtained;

[0099] Based on the fifth extraction feature being fused with the process feature, an output feature is obtained.

[0100] Preferably, when the input feature is a remote sensing image, an initial extraction feature, a shallow layer feature or a middle layer feature, the corresponding obtained output feature is an initial extraction feature, a shallow layer feature, a middle layer feature or a deep layer feature.

[0101] Preferably, based on the input feature input to the down-sampling layer, a first extraction feature F o .

[0102] Preferably, the first extracted feature is input into a first normalization layer to standardize the feature distribution, slow down the internal covariate shift, enhance the training stability of the model, obtain normalized features, and input the normalized features into a domain attention layer to obtain a second extracted feature.

[0103] Preferably, a domain attention layer is introduced to extract local spatial dependencies; the domain attention layer captures local structural features in high-resolution remote sensing images by sliding a fixed-size window over the feature map without introducing large-scale computational overhead, and the specific operation is as follows:

[0104] Based on the normalized features, three independent linear transformations (such as 1x1 convolution) are used to map the normalized features into query vectors q i , key vectors k j , and value vectors v j For each pixel position i in the local window, the attention weight with other positions j is defined as:

[0105]

[0106] where d represents the feature dimension.

[0107] Based on the attention weight, the value vector is weighted aggregated to obtain the updated feature, i.e., the second extracted feature F Nat .

[0108] The domain attention layer effectively enhances the modeling capability of edge, texture and other fine-grained local information in remote sensing images.

[0109] Preferably, in order to further capture long-range dependencies and semantic associations in wide-area scenes, a two-dimensional selective scanning layer is introduced to realize global feature modeling. The two-dimensional selective scanning layer is based on the state space modeling idea, and uses a dynamic weight mechanism to integrate information across channels in the feature map. The specific process is as follows:

[0110] State initialization: assign and initialize hidden states for each channel;

[0111] Selective scanning: sequentially scan the feature map according to a preset path (such as horizontal direction, vertical direction, diagonal line, etc.), and update the hidden state step by step;

[0112] Global integration: aggregate the information collected in the scanning path into the feature expression of each pixel to form a feature map containing global context information as the third extracted feature F GL .

[0113] This global modeling mechanism effectively improves the understanding ability of large-scale structures in complex scenes (such as urban buildings, road networks, farmland textures, etc.) in remote sensing images.

[0114] Preferably, the third extraction feature F GL is fused with the second extraction feature F Nat to obtain a process feature F o . c :

[0115] F c = F o ⊕Conv(F GL ⊕F Nat ).

[0116] Wherein, ⊕ represents feature splicing or element-wise addition, and Conv represents convolution operation.

[0117] Preferably, all the fusion units effectively improve the modeling capability of complex object boundary and semantic continuity in remote sensing image segmentation through the synergistic mechanism of local structure reinforcement and global semantic aggregation, and are important components to realize fine segmentation.

[0118] Preferably, in order to realize multi-scale feature expression and further enhance the recognition ability of the model to different spatial scale objects, a multi-level fusion structure is designed: based on the obtained initial extraction feature F input into the second fusion unit, a shallow feature F 1 is obtained; based on the shallow feature F 1 input into the third fusion unit, a middle feature F 2 is obtained; based on the middle feature F 2 input into the fourth fusion unit, a deep feature F 3 is obtained; through three layers of downsampling and fusion operation, the network can capture structure information from fine to coarse in remote sensing image layer by layer, and realize unified semantic modeling across scales.

[0119] Based on the multi-scale feature, a first processing feature, a second processing feature, a third processing feature and an integrated feature are obtained by inputting into the cascaded fusion network.

[0120] Preferably, the cascaded fusion network comprises a second convolutional layer, a first upsampling layer, a third convolutional layer, a second upsampling layer, a fourth convolutional layer, a fusion layer and a fifth convolutional layer.

[0121] Based on the deep feature, the second convolutional layer and the first upsampling layer are input in sequence to obtain the first processing feature;

[0122] Based on the middle feature, the third convolutional layer and the second upsampling layer are input in sequence to obtain the second processing feature;

[0123] Based on the shallow feature, the fourth convolutional layer is input to obtain the third processing feature;

[0124] The integrated feature is obtained by inputting the first processing feature, the second processing feature and the third processing feature to the fifth convolution layer after fusion.

[0125] Preferably, the first processing feature F de1 = Upsample(Conv(F 3 )); the second processing feature F de2 = Upsample(Conv(F 2 )); and the third processing feature F de3 = Conv(F 1 ). 4 = Concat(F de1 , F de2 , F de3 ); wherein Upsample represents an up-sampling operation (such as bilinear interpolation).

[0126] The multi-scale semantic feature is obtained by inputting the integrated feature to the multi-scale context fusion network.

[0127] Preferably, as shown in Figure 3 , the multi-scale context fusion network comprises a multi-scale concatenation unit, a feature enhancement unit and a feature recovery unit.

[0128] The multi-scale concatenation feature is obtained by inputting the integrated feature to the multi-scale concatenation unit.

[0129] The multi-scale enhancement feature is obtained by inputting the multi-scale concatenation feature to the feature enhancement unit.

[0130] The multi-scale semantic feature is obtained by inputting the multi-scale enhancement feature and the integrated feature to the feature recovery unit.

[0131] Preferably, the multi-scale concatenation unit comprises a third up-sampling layer, a sixth convolution layer, a dilated convolution layer, a seventh convolution layer, a fourth up-sampling layer, an eighth convolution layer and a windmill convolution layer.

[0132] The first scale feature is obtained by inputting the integrated feature to the third up-sampling layer, the sixth convolution layer and the dilated convolution layer in sequence.

[0133] The second scale feature is obtained by inputting the integrated feature to the seventh convolution layer and the fourth up-sampling layer in sequence.

[0134] The third scale feature is obtained by inputting the integrated feature to the eighth convolution layer and the windmill convolution layer in sequence.

[0135] The multi-scale concatenation feature is obtained by concatenating the first scale feature, the second scale feature and the third scale feature.

[0136] Preferably, the multi-scale concatenation unit includes three parallel branches, aiming to mine spatial context information from multiple scales and receptive fields to enhance the expression ability of feature boundaries in remote sensing images:

[0137] The first branch (shallow global perception path): the integrated feature β 0 is up-sampled (e.g. bilinear interpolation) to restore spatial resolution, then channel adjustment is performed using a 1x1 convolution, and finally the receptive field is enhanced through a dilated convolution layer to obtain the first scale feature β 1 .

[0138] The second branch (medium-scale context fusion path): the integrated feature β 0 is subjected to a standard 3x3 convolution operation to extract local texture features, followed by up-sampling to maintain the same spatial size as the first branch, to obtain the second scale feature β 2 .

[0139] The third branch (deep spatial structure modeling path): in this branch, a 5x5 large kernel convolution layer is used to capture wider spatial structure features, and a windmill convolution mechanism is introduced to simulate the rotational symmetry of features, improving the model's adaptability to complex targets such as building groups and road intersections. The output of this branch is the third scale feature β 3 .

[0140] Preferably, the first scale feature β 1 , the second scale feature β 2 and the third scale feature β 3 are concatenated along the channel dimension to fuse multi-scale semantic information and obtain a multi-scale concatenated feature β * .

[0141] Preferably, the feature enhancement unit includes a dynamic deformable convolution layer, a second two-dimensional selective scanning layer and a second feedforward neural network layer.

[0142] Based on the multi-scale concatenated feature, the dynamic deformable convolution layer, the second two-dimensional selective scanning layer and the second feedforward neural network layer are sequentially input to obtain a multi-scale enhanced feature.

[0143] Preferably, the dynamic deformable convolution layer (Dynamic Deformable Convolution): enhances the model's adaptability to target geometric deformation and boundary blur by dynamically adjusting the sampling position through learning the offset to generate irregular receptive fields.

[0144] Preferably, the second two-dimensional selective scanning layer has the same structure as the first two-dimensional selective scanning layer, both of which adopt a two-dimensional selective scanning block (2D Selective Scan Block): by simulating a spatial attention mechanism, the importance of information at different positions in a two-dimensional plane is modeled, and the response characteristics of the salient target area are strengthened.

[0145] Preferably, the first feed-forward neural network layer and the second feed-forward neural network layer have the same structure, both of which adopt a feed-forward neural network (FFN): by further purifying high-order semantic information through nonlinear transformation, multi-scale enhanced features β ** are generated. ** The multi-scale, dynamic structure and spatial attention mechanism are fused, providing a more rich and robust semantic expression for subsequent remote sensing image segmentation tasks.

[0146] Preferably, the feature recovery unit includes a depth separable convolution layer, a fifth upsampling layer and a ninth convolution layer.

[0147] Based on the integration of features and multi-scale enhanced features, complementary fusion features are obtained.

[0148] Based on the input of the integration features to the depth separable convolution layer, channel weight features are obtained.

[0149] Based on the fusion of the complementary fusion features and the channel weight features, the fifth upsampling layer and the ninth convolution layer are sequentially input, and multi-scale semantic features are obtained.

[0150] Preferably, based on the integration of features β 0 and multi-scale enhanced features β ** , complementary fusion features β r are obtained. r : β 0 ⊕ β ** ; based on the input of the integration features β 0 to the depth separable convolution layer (Depthwise Separable Convolution, DWConv) for weight modeling, this operation can be decomposed into depthwise convolution (Depthwise) and pointwise convolution (Pointwise), which has stronger channel-specific expression ability, and outputs channel weight features β w : β w = DWConv(β 0 ).

[0151] Preferably, based on the fusion of the complementary fusion features β r and the channel weight features β w , enhanced expression features β g are obtained. : βg = β r ⊕ β w ; realizes the enhanced expression based on the original feature adaptive weighting, which is equivalent to introducing a lightweight channel attention mechanism, which can automatically adjust the response strength of the fused features according to the feature distribution of the input remote sensing image, highlight important areas, and suppress redundant backgrounds.

[0152] Preferably, in order to further improve the segmentation accuracy and reduce the subsequent calculation overhead, the fused enhanced expression feature β g is upsampled (Upsample) to restore the original image or the target output size, so that the spatial dimension matches the segmentation requirement, and then a 1*1 convolution is applied for channel compression and mapping to obtain a multi-scale semantic feature β + : β + = Conv1*1 (Upasamplr (β g )), which has the advantages of compact structure, rich semantics and adaptive response. +

[0153] Based on the first processing feature, the second processing feature and the third processing feature, the first fusion feature, the second fusion feature and the third fusion feature are obtained respectively.

[0154] Based on the first fusion feature, the second fusion feature and the third fusion feature, the cascade feature is obtained.

[0155] Based on the cascade feature input to the result output network, the segmentation prediction result of the remote sensing image is obtained.

[0156] Preferably, the result output network includes: a multi-layer perceptron, a sixth up-sampling layer and a segmentation head;

[0157] The cascade feature is input to the multi-layer perceptron, the sixth up-sampling layer and the segmentation head in sequence to obtain the segmentation prediction result.

[0158] Preferably, the cascade feature is input to the multi-layer perceptron for nonlinear transformation to enhance the feature discriminability, and then passes through the sixth up-sampling layer and the segmentation head to obtain the pixel-level segmentation prediction result P of the remote sensing image.

[0159] Preferably, in this embodiment, the segmentation head adopts a lightweight structure composed of a 1*1 convolution layer, an up-sampling layer and a softmax activation function, which is used to map the fused multi-scale feature to a pixel-level classification result consistent with the resolution of the original image.

[0160] Preferably, the global-local fusion network, the cascade fusion network, the multi-scale context fusion network and the result output network jointly constitute an image segmentation model. ​

[0161] The image segmentation model is trained based on the total loss function to obtain a trained image segmentation model.

[0162] The trained image segmentation model is input with the to-be-processed remote sensing image to obtain a corresponding segmentation prediction result.

[0163] The total loss function integrates the cross-entropy loss, the Dice loss and the boundary perception loss.

[0164] Preferably, in the training stage, the total loss function is used to supervise the final output segmentation result P and the real label G. The total loss function is specifically:

[0165] L total total = a·L CE cross(P, G) + b·L Dice dice(P, G) + c·L Boundary boundary(P, G);

[0166] Wherein, P represents the final predicted segmentation probability map, and G represents the corresponding pixel-level feature class label; a, b and c each represent a loss weight coefficient, L CE cross(P, G) represents the cross-entropy loss, L Dice dice(P, G) represents the Dice loss, and L Boundary boundary(P, G) represents the boundary perception loss.

[0167] Preferably, the cross-entropy loss L CE cross(P, G) is:

[0168]

[0169] Wherein, N represents the total number of pixels in the image, C represents the total number of classes, P i,c represents the predicted probability that the ith pixel belongs to the cth class, G i represents the real class label of the ith pixel, and Ι(G i =c) represents an indicator function, which takes the value 1 when G i =c and 0 otherwise.

[0170] Preferably, the Dice loss L Dice dice(P, G) is:

[0171]

[0172] Wherein, G i,c represents the real label of the ith pixel in the cth class, which is usually one-hot encoded (i.e. 1 when the pixel belongs to the cth class, otherwise 0); ε represents a smoothing term to prevent division by zero, which is usually a very small positive number.

[0173] The boundary-aware loss L is preferably Boundary (P, G) is:

[0174]

[0175] wherein D i represents the distance of the i-th pixel to the real boundary, P i represents the predicted probability value of the i-th pixel by the model, G i represents the real label value of the i-th pixel.

[0176] Preferably, the segmentation result image is finally obtained by performing a pixel-by-pixel maximum probability class selection operation on the segmentation probability map P, and common remote sensing image segmentation evaluation indexes such as IoU (Intersection over Union), mIoU (mean IoU), F1-score and Pixel Accuracy (Pixel Accuracy) are used to quantitatively evaluate the performance of the model, so as to ensure stable segmentation effect in multi-class ground object scenarios.

[0177] Embodiment 3

[0178] As shown in Figure 4 , a remote sensing image segmentation system based on multi-scale context enhancement includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a first feature fusion module, a second feature fusion module, and a result output module.

[0179] The first feature extraction module is configured to acquire a remote sensing image to be processed and input the remote sensing image to a global-local fusion network to obtain multi-scale features.

[0180] The second feature extraction module is configured to input the multi-scale features to a cascade fusion network to obtain first processing features, second processing features, third processing features, and integrated features.

[0181] The third feature extraction module is configured to input the integrated features to a multi-scale context fusion network to obtain multi-scale semantic features.

[0182] The first feature fusion module is configured to fuse the first processing features, the second processing features, and the third processing features with the multi-scale semantic features, respectively, to obtain first fusion features, second fusion features, and third fusion features.

[0183] The second feature fusion module is configured to fuse the first fusion features, the second fusion features, and the third fusion features to obtain cascade features.

[0184] The result output module is configured to input the cascade features to a result output network to obtain a segmentation prediction result of the remote sensing image.

[0185] Preferably, the functions of the various functional modules in the embodiments correspond one by one to the above method content, and will not be described one by one here.

[0186] Embodiment 4

[0187] Based on the same inventive concept, the present application also provides a computer device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus;

[0188] The memory is used for storing a computer program.

[0189] The processor is used for executing the program stored on the memory, and can realize the multi-scale context enhancement based remote sensing image segmentation method in one of the embodiments 1 or 2.

[0190] The electronic device can include a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus. The processor can call the logical instructions in the memory to execute the multi-scale context enhancement based remote sensing image segmentation method in one of the embodiments 1 or 2.

[0191] In addition, the logical instructions in the memory described above can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0192] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0193] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-scale context enhancement based remote sensing image segmentation method, characterized in that, The method comprises the following steps: acquiring a remote sensing image to be processed and inputting the remote sensing image into a global-local fusion network to obtain a multi-scale feature; inputting the multi-scale feature into a cascaded fusion network to obtain a first processing feature, a second processing feature, a third processing feature, and an integrated feature; inputting the integrated feature into a multi-scale context fusion network to obtain a multi-scale semantic feature; fusing the first processing feature, the second processing feature, and the third processing feature with the multi-scale semantic feature respectively to obtain a first fusion feature, a second fusion feature, and a third fusion feature; fusing the first fusion feature, the second fusion feature, and the third fusion feature to obtain a cascaded feature; inputting the cascaded feature into a result output network to obtain a segmentation prediction result of the remote sensing image. 2.The method of claim 1, wherein, The global-local fusion network comprises a first fusion unit, a second fusion unit, a third fusion unit, and a fourth fusion unit; inputting the remote sensing image into the first fusion unit to obtain an initial extraction feature; inputting the initial extraction feature into the second fusion unit to obtain a shallow feature; inputting the shallow feature into the third fusion unit to obtain a middle feature; inputting the middle feature into the fourth fusion unit to obtain a deep feature; the shallow feature, the middle feature, and the deep feature jointly constitute the multi-scale feature.

3. The method of claim 2, wherein, The first fusion unit, the second fusion unit, the third fusion unit, and the fourth fusion unit are the same in structure and each comprises: a down-sampling layer, a first normalization layer, a domain attention layer, a first two-dimensional selective scanning layer, a first convolutional layer, a second normalization layer, and a first feedforward neural network layer; inputting an input feature into the down-sampling layer to obtain a first extraction feature; inputting the first extraction feature into the first normalization layer and the domain attention layer in sequence to obtain a second extraction feature; inputting the second extraction feature into the first two-dimensional selective scanning layer to obtain a third extraction feature; inputting the third extraction feature into the first convolutional layer after being fused with the second extraction feature to obtain a fourth extraction feature; fusing the fourth extraction feature with the first extraction feature to obtain a process feature; inputting the process feature into the second normalization layer and the first feedforward neural network layer in sequence to obtain a fifth extraction feature; fusing the fifth extraction feature and the process feature to obtain an output feature.

4. The method of claim 2, wherein, The cascaded fusion network comprises a second convolutional layer, a first up-sampling layer, a third convolutional layer, a second up-sampling layer, a fourth convolutional layer, a fusion layer, and a fifth convolutional layer; inputting the deep feature into the second convolutional layer and the first up-sampling layer in sequence to obtain the first processing feature; inputting the middle feature into the third convolutional layer and the second up-sampling layer in sequence to obtain the second processing feature; inputting the shallow feature into the fourth convolutional layer to obtain the third processing feature; The integrated feature is input into the fifth convolutional layer based on the first processing feature, the second processing feature and the third processing feature, and the integrated feature is obtained.

5. The method of claim 1, wherein, The multi-scale context fusion network comprises a multi-scale splicing unit, a feature enhancement unit and a feature recovery unit. The multi-scale splicing feature is obtained based on the integrated feature being input into the multi-scale splicing unit. The multi-scale enhanced feature is obtained based on the multi-scale splicing feature being input into the feature enhancement unit. The multi-scale semantic feature is obtained based on the multi-scale enhanced feature and the integrated feature being input into the feature recovery unit.

6. The method of claim 5, wherein, The multi-scale splicing unit comprises a third up-sampling layer, a sixth convolutional layer, an expanded convolutional layer, a seventh convolutional layer, a fourth up-sampling layer, an eighth convolutional layer and a windmill-shaped convolutional layer. The first scale feature is obtained based on the integrated feature being sequentially input into the third up-sampling layer, the sixth convolutional layer and the expanded convolutional layer. The second scale feature is obtained based on the integrated feature being sequentially input into the seventh convolutional layer and the fourth up-sampling layer. The third scale feature is obtained based on the integrated feature being sequentially input into the eighth convolutional layer and the windmill-shaped convolutional layer. The multi-scale splicing feature is obtained based on the first scale feature, the second scale feature and the third scale feature being spliced.

7. The method of claim 5, wherein the method further comprises: The feature enhancement unit comprises a dynamic deformable convolutional layer, a second two-dimensional selective scanning layer and a second feedforward neural network layer. The multi-scale enhanced feature is obtained based on the multi-scale splicing feature being sequentially input into the dynamic deformable convolutional layer, the second two-dimensional selective scanning layer and the second feedforward neural network layer. The feature recovery unit comprises a depth separable convolutional layer, a fifth up-sampling layer and a ninth convolutional layer. The complementary fusion feature is obtained based on the integrated feature and the multi-scale enhanced feature being fused. The channel weight feature is obtained based on the integrated feature being input into the depth separable convolutional layer. The multi-scale semantic feature is obtained based on the complementary fusion feature and the channel weight feature being fused and sequentially input into the fifth up-sampling layer and the ninth convolutional layer.

8. The method of claim 1, wherein, The result output network comprises a multi-layer perception, a sixth up-sampling layer and a segmentation head. The segmentation prediction result is obtained based on the global-local fusion network, the cascaded fusion network, the multi-scale context fusion network and the result output network jointly constituting an image segmentation model.

9. The method of claim 1, wherein, The total loss function is used to optimize and train the image segmentation model, and a trained image segmentation model is obtained. The total loss function is used to optimize and train the image segmentation model, and a trained image segmentation model is obtained. The total loss function is used to optimize and train the image segmentation model, and a trained image segmentation model is obtained. The total loss function comprises a cross-entropy loss, a Dice loss and a boundary perception loss.

10. A multiscale context enhancement based remote sensing image segmentation system for performing a multiscale context enhancement based remote sensing image segmentation method according to any one of claims 1-9, characterized in that, The total loss function comprises a cross-entropy loss, a Dice loss and a boundary perception loss. The total loss function comprises a cross-entropy loss, a Dice loss and a boundary perception loss. The first feature extraction module is configured to acquire a remote sensing image to be processed and input the remote sensing image to a global-local fusion network to obtain multi-scale features. The second feature extraction module is configured to input the multi-scale features to a cascade fusion network to obtain first processing features, second processing features, third processing features, and integrated features. The third feature extraction module is configured to input the integrated features to a multi-scale context fusion network to obtain multi-scale semantic features. The first feature fusion module is configured to fuse the first processing features, the second processing features, and the third processing features with the multi-scale semantic features respectively to obtain first fusion features, second fusion features, and third fusion features. The second feature fusion module is configured to fuse the first fusion features, the second fusion features, and the third fusion features to obtain cascade features. The result output module is configured to input the cascade features to a result output network to obtain a segmentation prediction result of the remote sensing image.

Images