Semantic segmentation method and device for flood area in SAR image, storage medium and terminal

Abstract

The present invention provides a semantic segmentation method and device for flood areas in SAR images, a storage medium and a terminal, wherein the method comprises using a preset encoder to perform semantic extraction on the SAR image to obtain a multi-stage semantic feature map; performing a preset process on each feature map in the shallow feature map area and then performing noise filtering to obtain a noise-filtered feature map of the corresponding stage; performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map; performing a preset process on the feature map in the intermediate feature map area to obtain a corresponding pre-processed intermediate feature map, splicing the pre-processed intermediate feature map, the noise-filtered feature map and the processed deep feature map to obtain an optimized feature map; inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image. The present invention can effectively improve the semantic segmentation effect of flood areas in SAR images.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing, and relates to a semantic segmentation method for flood areas in SAR images, and in particular to a semantic segmentation method and device, storage medium and terminal for flood areas in SAR images. Background Art

[0002] Synthetic Aperture Radar (SAR) is not limited by daytime or weather conditions and can penetrate clouds. It plays a key role in remote sensing image processing. Flood segmentation based on SAR images is an important task in earth observation missions. Traditional SAR image flood segmentation methods, such as conditional random fields (CRF) and support vector machines (SVM), usually rely on tedious parameter adjustments based on professional knowledge. In recent years, convolutional neural networks (CNN) have been widely used in flood semantic segmentation of SAR images.

[0003] However, in the SAR image flood segmentation task, the existing technology has the following three unresolved problems. First, when SAR images show significant changes in texture, shape and features, it is difficult to model long-range relationships because the CNN convolution kernel can only focus on a limited number of local pixels and semantic information within its receptive field. Although some methods incorporate other mechanisms into the network design to improve the ability of CNN to capture long-range relationships, they still do not directly encode global dependencies. Therefore, this method does not completely solve the limitations of CNN, and the model effect is still limited by the localization characteristics of the convolution operation. Second, the semantic segmentation method based on the encoder-decoder architecture reduces the spatial dimension of features through continuous convolution and pooling operations, resulting in discontinuous segmentation of flood edges in SAR images and loss of small water body information. To solve this problem, it is necessary to introduce jump connections in the decoder to fuse low-level features with high-level semantic information to improve the model's semantic segmentation performance for small targets. However, the existing information fusion is only performed by element-by-element summation and simple combination of features of different scales. This method may not only cause feature confusion and interfere with the segmentation of small targets, but also the low-level feature maps involved in the fusion usually contain a lot of noise, which will also affect the final segmentation performance. Third, the distribution of floods is affected by various factors including topography and landforms, and the scale and form of floods vary. These complex geographical features lead to significant scale changes of floods in SAR images, which greatly increases the difficulty of semantic segmentation. For the above reasons, the effect of existing SAR image flood segmentation is poor. Summary of the invention

[0004] The object of the present invention is to provide a semantic segmentation method and device for flood areas in SAR images, a storage medium and a terminal, so as to solve the technical problem that the semantic segmentation effect of flood areas in SAR images in the prior art is poor.

[0005] In a first aspect, the present invention provides a semantic segmentation method for flood areas in SAR images, comprising:

[0006] Using a preset encoder to perform semantic extraction on the SAR image to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map;

[0007] Each feature map in the shallow feature map area is processed by a preset process and then subjected to noise filtering to obtain a noise-filtered feature map of the corresponding stage;

[0008] Performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map;

[0009] The feature map in the intermediate feature map area is processed in advance to obtain a corresponding preprocessed intermediate feature map, and the preprocessed intermediate feature map, the noise filtered feature map and the processed deep feature map are spliced ​​to obtain an optimized feature map;

[0010] Inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image;

[0011] Wherein, the shallow feature map area includes at least a first-stage feature map and a second-stage feature map.

[0012] In one embodiment of the present invention, each feature map in the shallow feature map area is subjected to a preset process and then subjected to noise filtering to obtain a noise-filtered feature map of the corresponding stage, including:

[0013] Performing preset processing on each feature map in the shallow feature map area to obtain a preprocessed feature map of the corresponding stage;

[0014] Based on the feature map after preprocessing in the odd-numbered stage and the feature map after preprocessing in the next stage of the odd-numbered stage, obtain the corresponding channel attention map;

[0015] The odd-numbered stage preprocessed feature map, the next stage preprocessed feature map of the odd-numbered stage and the corresponding channel attention map are fused to obtain the corresponding noise-filtered feature map.

[0016] In one embodiment of the present invention, based on the feature map after preprocessing in the odd stage and the feature map after preprocessing in the next stage of the odd stage, obtaining the corresponding channel attention map includes:

[0017] Obtain the reshaped result of the feature map after preprocessing in the odd-numbered stage as the front feature map, and obtain the reshaped result of the feature map after preprocessing in the next stage of the odd-numbered stage as the back feature map;

[0018] Performing matrix multiplication on the rear feature map and the transpose of the front feature map to obtain a multiplication result;

[0019] A softmax operation is performed on the multiplication result to obtain the corresponding channel attention map.

[0020] In one embodiment of the present invention, performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map includes:

[0021] Extracting local context information of the feature map of the last stage using a preset depth convolution operation to obtain a feature map containing local context information;

[0022] Extracting multi-scale context information of the feature map containing local context information by using multi-branch depth-separable strip convolution to obtain multiple depth feature maps;

[0023] The multiple depth feature maps are spliced ​​along the channel dimension to obtain a depth splicing result, the depth splicing result is subjected to 1x1 convolution processing and then element-by-element multiplication with the last stage feature map to obtain a processed deep feature map.

[0024] In one embodiment of the present invention, each branch of the depthwise separable stripe convolution includes a first depthwise separable stripe convolution and a second depthwise separable stripe convolution;

[0025] The convolution kernel size of the first depth-separable strip convolution is 1 times the preset size;

[0026] The convolution kernel size of the second depth-wise separable strip convolution is a preset size multiplied by 1;

[0027] The preset size corresponds to a branch depth separable strip convolution, and different branch depth separable strip convolutions correspond to different preset sizes.

[0028] In one embodiment of the present invention, the preset sizes corresponding to the multi-branch depthwise separable strip convolution are 7, 11, 15, and 21 respectively.

[0029] In one embodiment of the present invention, the preset encoder is a hybrid transformer encoder MiT.

[0030] In a second aspect, the present invention further provides a semantic segmentation device for flood areas in SAR images, characterized in that it comprises:

[0031] A main encoding module, used for performing semantic extraction on the SAR image using a preset encoder to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map;

[0032] A noise filtering module, used for performing a preset process on each feature map in the shallow feature map area and then performing noise filtering to obtain a noise-filtered feature map of the corresponding stage;

[0033] A multi-scale convolution module, used for performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map;

[0034] A fusion module is used to perform preset processing on the feature map in the intermediate feature map area to obtain a corresponding preprocessed intermediate feature map, and to splice the preprocessed intermediate feature map, the noise filtered feature map and the processed deep feature map to obtain an optimized feature map;

[0035] A classification module, used for inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image;

[0036] Wherein, the shallow feature map area includes at least a first-stage feature map and a second-stage feature map.

[0037] In a third aspect, the present invention further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic segmentation method for flood areas in SAR images as described above.

[0038] In a fourth aspect, the present invention further provides a terminal, comprising a processor and a memory, wherein the memory is communicatively connected to the processor;

[0039] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the semantic segmentation method of flood areas in SAR images as described above.

[0040] As described above, the semantic segmentation method and device, storage medium and terminal of the SAR image flood area of ​​the present invention have the following beneficial effects:

[0041] 1. The present invention removes redundant semantic information by performing noise filtering on the feature maps in the shallow feature map area, thereby enhancing the semantic segmentation performance of small objects and edges.

[0042] 2. The present invention performs multi-scale convolution processing on the feature map of the last stage and then fuses it with other feature maps, which not only enhances the model's ability to represent multi-scale features, but also reduces the number of parameters and computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic flow chart of a method for semantic segmentation of flood areas in SAR images according to an embodiment of the present invention is shown.

[0044] Figure 2 A schematic diagram of data flow of the semantic segmentation method for flood areas in SAR images according to an embodiment of the present invention is shown.

[0045] Figure 3 A schematic diagram of the process of noise filtering in the semantic segmentation method of flood areas in SAR images according to an embodiment of the present invention is shown.

[0046] Figure 4 A schematic diagram of the process of multi-scale convolution in the semantic segmentation method of flood areas in SAR images according to an embodiment of the present invention is shown.

[0047] Figure 5 A visual comparison diagram of the semantic segmentation method described in an embodiment of the present invention and the prior art is shown.

[0048] Figure 6 A schematic diagram of the structure of a semantic segmentation device for flood areas in SAR images according to an embodiment of the present invention is shown.

[0049] Figure 7 A schematic diagram of the structure of a terminal according to an embodiment of the present invention is shown.

[0050] Component number description:

[0051] E1-E4 are feature maps from the first stage to the fourth stage, MDCM is multi-scale convolution, NFM is noise filtering, D1 is the preprocessed feature map corresponding to the first stage feature map, D2 is the preprocessed feature map corresponding to the second stage feature map, D3 is the preprocessed feature map corresponding to the third stage feature map, D4 is the processed deep feature map, Cat is feature map splicing, and XX is the optimized feature map. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] The following will describe in detail the principles and implementation methods of the semantic segmentation method and device for flood areas in SAR images, storage media, and terminals of this embodiment, so that those skilled in the art can understand the semantic segmentation method, device, storage medium, and terminal for flood areas in SAR images of this embodiment without creative work.

[0054] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention provides a semantic segmentation method for flood areas in SAR images.

[0055] Figure 1 FIG. 1 is a flow chart showing a semantic segmentation method for flood areas in SAR images according to an embodiment of the present invention, referring to FIG. Figure 1 As shown, the semantic segmentation method of flood area in SAR image according to the embodiment of the present invention mainly includes steps S100 to S500.

[0056] Figure 2 FIG. 1 is a schematic diagram showing the data flow of the semantic segmentation method for flood areas in SAR images according to an embodiment of the present invention, referring to FIG. Figure 2 As shown, a preset encoder (which is equivalent to the Encoder process) is used to extract the first-stage feature map E1 to the fourth-stage feature map E4 of the SAR image, and E1 and E2 are used as shallow feature map areas. After E1 is subjected to preset processing, the corresponding pre-processed first-stage feature map D1 is obtained, and after E2 is subjected to preset processing, the corresponding pre-processed second-stage feature map D2 is obtained. D1 and D2 are subjected to noise filtering to obtain the corresponding noise-filtered feature map; the last-stage feature map E4 is subjected to multi-scale convolution processing MDCM to obtain the processed deep feature map D4; E3 is used as the intermediate feature map area and E3 is subjected to preset processing to obtain D3, and the noise-filtered feature map is spliced ​​with D4 and D3 to obtain the optimized feature map XX (noise filtering NFM, multi-scale convolution MDCM and feature map splicing together constitute the Decoder process). The optimized feature map XX is input into the preset classifier to obtain the semantic segmentation result of the SAR image. It should be noted that Figure 2Only the feature maps of 4 stages are taken as an example for explanation. According to the actual situation, the preset encoder can be used to extract feature maps of different numbers of stages. For example, if the feature maps of 10 stages are extracted, the feature map of the last stage is the feature map of the tenth stage. At this time, the middle feature map area is the feature map of the third stage to the feature map of the ninth stage, and the pre-processed third stage feature map to the pre-processed ninth stage feature map are obtained after the preset processing. The embodiment of the present invention does not limit the number of feature map stages to be extracted.

[0057] Step S100: using a preset encoder to perform semantic extraction on the SAR image to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map.

[0058] Specifically, in this embodiment, the Encoder-Decoder structure is used to construct the network model, wherein the Encoder is an encoder and the Decoder is a decoder. In the encoder, this embodiment uses a preset encoder to extract the semantics of the SAR image to obtain semantic feature maps of multiple stages. These semantic feature maps are divided into a shallow feature map area, an intermediate feature map area, and a feature map of the last stage. The feature map in the shallow feature map area refers to the feature map of the previous stage that mainly extracts shallow features, such as edges and textures in SAR images; the feature map of the last stage is the feature map of the last stage (that is, the deepest layer) extracted by the preset encoder. The feature map has a high degree of semantic abstraction and captures high-level information of the image, such as the shape and distribution of the flood area; the intermediate feature map area refers to all feature maps except the shallow feature map area and the feature map of the last stage. For example, there are ten stages of feature maps in total. If the shallow feature map area includes feature maps of the first to fourth stages, then the intermediate feature map area is the feature map of the fifth to ninth stages. Reference Figure 1 As shown, in this embodiment, the shallow feature map area includes at least the first stage feature map E1 and the second stage feature map E2. According to actual conditions, feature maps of more stages can also be selected to form the shallow feature map area. It is only necessary that the number of feature maps in the shallow feature map area is an even number and the stages to which the feature maps belong are continuous. The number of feature maps in the intermediate feature map area is at least 1. For example, when six stage feature maps are extracted, the first stage to the fourth stage feature maps can be set as the shallow feature map area.

[0059] Optionally, the preset encoder is a hybrid transformer encoder MiT. MiT (Mixture of Transformer) is a new type of hierarchical transformer encoder designed within the SegFormer framework, which outputs multi-scale features at different layers, and unlike the early transformer architecture, MiT does not rely on position encoding. MiT also has an efficient self-attention mechanism to reduce computational complexity. MiT can reduce the amount of computation while maintaining the ability of the self-attention mechanism to capture long-distance dependencies. The embodiment of the present invention uses MiT as the preset encoder. MiT is a Transformer-based encoder that can avoid the limitations in global modeling caused by the introduction of CNN, so that the model can effectively distinguish between flood areas and background objects with similar features.

[0060] Step S200: each feature map in the shallow feature map area is subjected to a preset process and then subjected to noise filtering to obtain a noise-filtered feature map of the corresponding stage.

[0061] The prior art often processes and fuses the feature maps of each stage in the same way, but this application considers that the shallow feature map area of ​​the SAR image contains a large amount of redundant information, which has an adverse effect on the final semantic segmentation effect. Therefore, the feature map in the shallow feature map area is subjected to noise filtering to filter out the redundant information and improve the semantic segmentation effect. The preset processing is to process the number of channels of each processed feature map to be the same as the number of channels of the feature map of the last stage, referring to Figure 1 As shown, the number of channels of the processed feature map is processed to 256. According to actual needs, 256 can be modified to other channel numbers. Optionally, the preset processing is a 1×1 convolution operation. Furthermore, the preset processing is to first perform a 1×1 convolution operation, then input the BN layer, and finally use the ReLU activation function for processing. By combining the 1×1 convolution operation with the BN layer (Batch Normalization) and the ReLU activation function, this combination can effectively improve the representation ability of the feature map, while stabilizing the training process and accelerating convergence.

[0062] Optionally, Figure 3 FIG. 4 is a flow chart showing the noise filtering process in the semantic segmentation method of flood area in SAR image according to an embodiment of the present invention, referring to FIG. Figure 3 As shown, each feature map in the shallow feature map area is subjected to a preset process and then subjected to noise filtering to obtain a feature map after noise filtering at the corresponding stage, including steps S201 to S203:

[0063] Step S201: Perform preset processing on each feature map in the shallow feature map area to obtain a preprocessed feature map of the corresponding stage.

[0064] Specifically, a preset processing is performed on each feature map in the shallow feature map area (first-stage feature map E1, second-stage feature map E2) to modify its number of channels to a preset number of channels, where the preset number of channels is the number of channels of the last-stage feature map, and a preprocessed feature map D1 corresponding to the first-stage feature map E1 is obtained, and a preprocessed feature map D2 corresponding to the second-stage feature map E2 is obtained.

[0065] Step S202: Based on the feature map after preprocessing in the odd stage and the feature map after preprocessing in the next stage of the odd stage, obtain the corresponding channel attention map.

[0066] Since the number of feature maps in the shallow feature map area is even, this embodiment uses each odd-numbered stage preprocessed feature map and its next stage preprocessed feature map as a pair, and obtains a channel attention map based on a pair of feature maps, that is, the channel attention maps are obtained in pairs. For example, there are 4 feature maps in the shallow feature map area, the first to fourth stage feature maps, then a corresponding channel attention map is obtained based on the first stage preprocessed feature map and the second stage preprocessed feature map, and another corresponding channel attention map is obtained based on the third stage preprocessed feature map and the fourth stage preprocessed feature map. Specifically, refer to Figure 3 As shown, based on the feature map D1 after preprocessing in the odd stage and the feature map D2 after preprocessing in the next stage of the odd stage, a channel attention map A corresponding to the two feature maps is obtained, and the channel attention map is used to characterize the similarity between the two feature maps.

[0067] Optionally, based on the feature map after preprocessing in the odd stage and the feature map after preprocessing in the next stage of the odd stage, the corresponding channel attention map is obtained, referring to the following formula:

[0068]

[0069] Among them, A represents the channel attention map, Represents a matrix multiplication operation, and R() represents a reshape (i.e. Figure 3In reshape, reshaping is used to adjust the shape of the feature map to adapt to specific operations or structural requirements; S() represents the softmax function, which is used to convert a vector into a probability distribution so that all output values ​​are non-negative and the sum is 1. T represents the transpose operation, which is used to reverse the rows and columns of a matrix or tensor. D1 (i.e., the first stage feature map after preprocessing) represents the feature map after preprocessing of the odd stage, and D2 (i.e., the second stage feature map after preprocessing) represents the feature map after preprocessing of the next stage of the odd stage. When there are more feature maps in the shallow feature map area, they are all processed in the above manner. For example, the shallow feature map area includes the feature maps of the first stage to the fourth stage, then the feature map of the third stage after preprocessing can be used as the feature map of the odd stage, and the feature map of the fourth stage after preprocessing can be used as the feature map of the next stage of the odd stage, and the processing can be performed in the above manner.

[0070] Referring to the above formula, based on the feature map after preprocessing in the odd stage and the feature map after preprocessing in the next stage of the odd stage, obtaining the corresponding channel attention map includes steps S2021 to S2023:

[0071] Step S2021: Obtain the reshaped result of the feature map after preprocessing in the odd stage as the front feature map, and obtain the reshaped result of the feature map after preprocessing in the next stage of the odd stage as the back feature map.

[0072] Specifically, in this step, the feature map D1 after preprocessing in the odd stage and the feature map D2 after preprocessing in the next stage of the odd stage are used as inputs. The purpose of reshaping D1 and D2 is to meet the shape requirements of subsequent matrix multiplication. Figure 3 As shown, the initial shapes of D1 and D2 are both c×h×w. The feature map D1 is adjusted from the shape c×h×w to a new shape hw×c (that is, R(D1) in the above formula) as the front feature map, and the feature map E2 is adjusted from the shape c×h×w to a new shape c×hw (that is, R(D2) in the above formula) as the back feature map.

[0073] Step S2022: Perform matrix multiplication on the transposed rear feature map and the front feature map to obtain a multiplication result.

[0074] Specifically, use the later feature map R(D2) to multiply the transposed matrix R(D1) of the previous feature map T The purpose of this operation is to calculate the similarity relationship between channels. The result of matrix multiplication is a new matrix in which each element represents the degree of correlation between the corresponding channels in the feature maps of the previous and next stages.

[0075] Step S2023: Perform a softmax operation on the multiplication result to obtain the corresponding channel attention map.

[0076] Specifically, performing a softmax operation is to use the softmax function to process the multiplication result. After obtaining the matrix multiplication result, the values ​​in the matrix multiplication result are normalized to probability values ​​between 0 and 1 through the Softmax operation, thereby generating a channel attention map A. The Softmax operation ensures the relative importance of each channel in different feature maps, so that the model can focus on important channel features. Therefore, by calculating the channel similarity of the previous feature map and the subsequent feature map, a channel attention map A is generated for subsequent operations.

[0077] Step S203: Fuse the feature map after preprocessing in the odd-numbered stage, the feature map after preprocessing in the next stage of the odd-numbered stage, and the corresponding channel attention map to obtain the corresponding feature map after noise filtering.

[0078] Specifically, the channel attention map A is fused with the original two pre-processed feature maps (D1 and D2). Since the channel attention map A captures the similarity of the original processed feature maps, this fusion method amplifies the most relevant feature information of D1 and D2 and indirectly weakens irrelevant redundant noise.

[0079] Optionally, the odd-numbered stage preprocessed feature map, the next stage preprocessed feature map of the odd-numbered stage, and the corresponding channel attention map are fused to obtain the corresponding noise-filtered feature map, referring to the following formula:

[0080]

[0081] Among them, N represents the feature map after noise filtering, w represents a 3×3 convolution operation, represents the element-by-element addition operation, represents an element-wise subtraction operation, Represents a matrix multiplication operation. That is, multiply the reshaped result of D1 by the attention map to obtain the channel resampling result; reshape the channel resampling result to obtain the resampled reshaped result, add the resampled reshaped result, D2, and the result of element-by-element subtraction of D2 and D1 element-by-element; process the result of element-by-element addition by convolution operation to obtain the final noise-filtered feature map N. Using the attention map A to perform channel resampling on the reshaped result of D1 can increase the weight of effective features and reduce the interference of irrelevant or redundant features on fusion; by subtracting D2 from D1 element-by-element and then adding D2 element-by-element, the advantageous features of D2 can be more highlighted, while suppressing redundant information in D2 that is similar to D2, so as to provide a more balanced feature map fusion result, and then adding the above result to the resampled reshaped result element-by-element, which can not only enhance the information interaction between feature maps in the shallow feature map area, but also effectively reduce information redundancy and noise.

[0082] Step S300: Perform multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map.

[0083] Since the feature map of the last stage has undergone multiple stages of feature extraction and is the deepest feature map in the encoder, it contains rich semantic information. In the embodiment of the present application, multi-scale convolution performs depth-separable convolution on the feature map of the last stage at different scales, and different scales correspond to different convolution kernel sizes. By processing the feature map of the last stage through multi-scale convolution, details and contextual information can be captured at different scales, thereby further enhancing the expressiveness of the feature map of the last stage.

[0084] Optionally, performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map includes steps S301 to S303, and specifically refers to the following formula:

[0085]

[0086] Among them, X represents the feature map of the last stage, X' represents the deep feature map after processing, and Concat() represents the channel concatenation operation, which is used to merge feature maps of different scales along the channel dimension; Represents an element-by-element multiplication operation, which is used to multiply the attention weight by the feature map element-by-element. represents a 1×1 convolution operation, X 1 -X 4 Represents multiple depth feature maps, and Att represents the result of 1x1 convolution processing on the depth splicing result.

[0087] Step S301: Use a preset deep convolution operation to extract local context information of the feature map of the last stage to obtain a feature map containing local context information.

[0088] refer to Figure 4 As shown, a 3×3 preset depth convolution operation is used to extract the local context information of the feature map of the last stage to obtain a feature map containing local context information.

[0089] Step S302: Use multi-branch depth-separable strip convolution to extract multi-scale context information of the feature map containing local context information to obtain multiple depth feature maps.

[0090] Specifically, through multi-branch depth-separable strip convolution, multiple depth feature maps X of different scales corresponding to the feature map containing local context information are obtained. 1 , X 2 , X 3 , X 4 .

[0091] Step S303: Multiple depth feature maps are spliced ​​along the channel dimension to obtain a depth splicing result, and the depth splicing result is subjected to 1x1 convolution processing and then element-wise multiplication with the feature map of the last stage to obtain a processed deep feature map.

[0092] refer to Figure 4 As shown in the figure, all the deep feature maps are concatenated along the channel dimension, and the concatenated result is used as the deep concatenation result C. The number of channels of the deep concatenation result C is adjusted using a 1×1 convolution. This step reduces the computational complexity of the model parameters while ensuring that important contextual information is retained.

[0093] Optionally, each branch depth-separable strip convolution includes a first depth-separable strip convolution and a second depth-separable strip convolution; the convolution kernel size of the first depth-separable strip convolution is 1 times a preset size; the convolution kernel size of the second depth-separable strip convolution is the preset size times 1; wherein the preset size corresponds to the branch depth-separable strip convolution, and different branch depth-separable strip convolutions correspond to different preset sizes. Figure 4 As shown in the figure, the depthwise separable strip convolution of each branch further reduces the computational complexity and improves the feature extraction capability through two convolution operations: the first depthwise separable strip convolution and the second depthwise separable strip convolution. The first depthwise separable strip convolution has a convolution kernel size of 1 times the preset size, and has a larger convolution kernel (e.g., 1×7, 1×11, 1×15, 1×21) in only one direction, which is used to capture the features of the image in one dimension (e.g., height). The second depthwise separable strip convolution has a convolution kernel size of 1 times the preset size, and has a larger convolution kernel (e.g., 7×1, 11×1, 15×1, 21×1) in only another direction, focusing on the features of the image in another dimension (e.g., width). Through the above two-step depthwise separable strip convolution, more spatial information can be obtained at a lower computational cost, and directional information such as stripes and edges in the image can be captured and processed more sensitively. In addition, by using different preset sizes in each branch, the model can extract features at multiple scales and enhance the richness of features. It should be noted that when the preset encoder is the hybrid transformer encoder MiT, this embodiment fully combines the advantages of multi-scale convolution and self-attention mechanism by applying MDCM to the Transformer-based encoder of MiT, makes up for the shortcomings of CNN in modeling long-distance dependencies, and realizes more flexible multi-scale feature fusion. This design can not only improve the global semantic understanding ability of the model, but also retain the ability to capture local features in detail. Therefore, it has better effect in the semantic segmentation task of flood areas in SAR images.

[0094] Optionally, the multi-branch depthwise separable strip convolution has a total of 4 branches of depthwise separable strip convolution, and the preset sizes corresponding to the multi-branch depthwise separable strip convolution are 7, 11, 15, and 21. 7, 11, 15, and 21 are selected as the preset sizes mainly to achieve balance in multi-scale feature extraction and avoid the problem of convolution kernel being too large or too small, so as to capture multi-scale features and control the computational overhead of the model.

[0095] Step S400: pre-process the feature map in the intermediate feature map area to obtain the corresponding pre-processed intermediate feature map, and splice the pre-processed intermediate feature map, the noise filtered feature map and the processed deep feature map to obtain the optimized feature map.

[0096] Specifically, refer to Figure 2 As shown in the figure, after these intermediate feature maps are preset, the number of channels is the same as that of the feature map of the last stage. The splicing operation Cat can integrate features at different levels, so that the model can use a variety of information from shallow, intermediate and deep layers to improve its representation ability. The feature map after noise filtering is a streamlined version that removes irrelevant information and emphasizes effective features; the processed deep feature map contains multi-scale information extracted by deep convolution; the pre-processed intermediate feature map serves as a feature supplement; therefore, the optimized feature map after splicing the three integrates features from different levels and after processing, has strong expression ability, and can help the model achieve higher accuracy and robustness in subsequent tasks. At the same time, by fusing features from different stages, the optimized feature map can better cope with changes and noise interference in complex scenes, helping the model to make more accurate judgments when facing complex tasks. Reference Figure 2 As shown, optionally, it also includes upsampling the splicing result to the same size as the first stage feature map after preprocessing, and obtaining the upsampled feature map as the optimized feature map, for example Figure 2 The size of the optimized feature map XX in is 1 / 4 in height, 1 / 4 in width, and 256 in channel dimension, as well as the size of the first stage feature map D1 after preprocessing.

[0097] Step S500: inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image.

[0098] The optimized feature map contains rich information that has been noise filtered and multi-scale processed, and has good semantic information expression capabilities. The optimized feature map is used as the input of the classifier to obtain the final semantic segmentation result. The embodiment of the present invention does not limit the preset classifier, and common classifiers can be applied here. Optionally, the preset classifier includes a convolution layer and an argmax operation. Specifically, after the optimized feature map is input into the preset classifier, it is first input into the convolution layer for processing, and then the processing result of the convolution layer is subjected to the argmax operation to obtain the semantic segmentation result, wherein the argmax operation is an operation of taking the maximum value in the channel dimension to output the index of the channel to which the maximum value of each pixel belongs in all corresponding channels, and the index represents the predicted category. The number of channel dimensions of the convolution layer is equal to the number of target classification categories. Inputting the optimized feature map into the convolution layer can generate a probability map of each pixel belonging to different categories, and the argmax operation is used to obtain the predicted category of each pixel.

[0099] In order to fully verify the effect of the semantic segmentation method of the embodiment of the present invention, it is compared with U-Net, DeepLabV3+ and DANet in the prior art in terms of accuracy (taking the quantitative results on the Sen1Floods11 dataset as an example). Figure 5 A visual comparison diagram of the semantic segmentation method according to the embodiment of the present invention and the prior art is shown. Figure 5 As shown, Figure 5 The figure shows visualization images of two images Image and their corresponding original labels Label, as well as visualization images of the prediction results of semantic segmentation of Image using U-Net, DeepLabV3+, DANet and the method of the present invention OURs. It can be clearly seen that our method (that is, the OURs image in the figure) effectively distinguishes targets with a high similarity to flood areas during the segmentation process compared with other algorithms (that is, U-Net, DeepLabV3+ and DANet), and is significantly better than several other algorithms in segmenting flood edges, small flood branches and flood areas of different sizes.

[0100] The protection scope of the semantic segmentation method of flood areas in SAR images according to the embodiment of the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art according to the principles of the present invention are included in the protection scope of the present invention.

[0101] The semantic segmentation method of flooded areas in SAR images of the embodiment of the present invention removes redundant semantic information by filtering noise from feature maps in shallow feature map areas, thereby enhancing the semantic segmentation performance of small targets and edges; after multi-scale convolution processing of the feature map in the last stage, it is fused with other feature maps, which not only enhances the model's ability to represent multi-scale features, but also reduces the number of parameters and computational complexity. In addition, the multi-branch deep separable strip convolution is combined with the Transformer-based encoder MiT to help the model obtain richer semantic information, further improving the semantic segmentation effect of the model.

[0102] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a semantic segmentation device for flood areas in SAR images.

[0103] Figure 6 FIG. 1 is a schematic diagram showing a structure of a semantic segmentation device for flood areas in a SAR image according to an embodiment of the present invention, referring to FIG. Figure 6 As shown, the semantic segmentation device of the flood area of ​​the SAR image according to the embodiment of the present invention includes:

[0104] A main encoding module, used for performing semantic extraction on the SAR image using a preset encoder to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map;

[0105] A noise filtering module is used to perform a preset process on each feature map in the shallow feature map area and then perform noise filtering to obtain a noise-filtered feature map of the corresponding stage;

[0106] The multi-scale convolution module is used to perform multi-scale convolution processing on the feature map of the last stage to obtain the processed deep feature map;

[0107] A fusion module is used to pre-process the feature map in the intermediate feature map area to obtain the corresponding pre-processed intermediate feature map, and to splice the pre-processed intermediate feature map, the noise filtered feature map and the processed deep feature map to obtain the optimized feature map;

[0108] The classification module is used to input the optimized feature map into a preset classifier to obtain the semantic segmentation result of the SAR image;

[0109] Among them, the shallow feature map area includes at least the first stage feature map and the second stage feature map.

[0110] The semantic segmentation device for flooded areas in SAR images of the embodiment of the present invention removes redundant semantic information by filtering noise from feature maps in shallow feature map areas, thereby enhancing the semantic segmentation performance of small targets and edges; after multi-scale convolution processing of the feature map in the last stage, it is fused with other feature maps, which not only enhances the model's ability to represent multi-scale features, but also reduces the number of parameters and computational complexity. In addition, the multi-branch deep separable strip convolution is combined with the Transformer-based encoder MiT to help the model obtain richer semantic information, further improving the semantic segmentation effect of the model.

[0111] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, all steps of the semantic segmentation method of flood areas in SAR images of the embodiment are implemented.

[0112] The specific steps of the semantic segmentation method for flood areas in SAR images and the beneficial effects obtained by applying the readable storage medium provided by the embodiment of the present invention are the same as those in the above embodiment, and will not be described in detail here.

[0113] A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0114] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a terminal. Figure 7 FIG. 1 shows a schematic diagram of the structure of a terminal according to an embodiment of the present invention, referring to FIG. Figure 7As shown, the terminal of the embodiment of the present invention includes a processor and a memory, and the memory is communicatively connected to the processor; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes all steps of the semantic segmentation method of the flood area of ​​the SAR image in the above embodiment.

[0115] The specific steps of the semantic segmentation method for flood areas in SAR images and the beneficial effects obtained by applying the terminal provided by the embodiment of the present invention are the same as those in the above embodiment, and will not be described in detail here.

[0116] It should be noted that the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. Similarly, the processor may also be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0117] Although the embodiments disclosed in the present invention are as above, the contents described are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A semantic segmentation method for flood areas in SAR images, comprising: Using a preset encoder to perform semantic extraction on the SAR image to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map; Each feature map in the shallow feature map area is processed by a preset process and then subjected to noise filtering to obtain a noise-filtered feature map of the corresponding stage; Performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map; The feature map in the intermediate feature map area is processed in advance to obtain a corresponding preprocessed intermediate feature map, and the preprocessed intermediate feature map, the noise filtered feature map and the processed deep feature map are spliced ​​to obtain an optimized feature map; Inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image; Wherein, the shallow feature map area includes at least the first stage feature map and the second stage feature map; Each feature map in the shallow feature map area is subjected to a preset process and then subjected to noise filtering to obtain a feature map after noise filtering at a corresponding stage, including: Performing preset processing on each feature map in the shallow feature map area to obtain a preprocessed feature map of the corresponding stage; Based on the feature map after preprocessing in the odd-numbered stage and the feature map after preprocessing in the next stage of the odd-numbered stage, obtain the corresponding channel attention map; The odd-numbered stage preprocessed feature map, the next stage preprocessed feature map of the odd-numbered stage and the corresponding channel attention map are fused to obtain the corresponding noise-filtered feature map.

2. The semantic segmentation method according to claim 1, characterized in that: Based on the feature map after preprocessing in the odd stage and the feature map after preprocessing in the next stage of the odd stage, obtaining the corresponding channel attention map includes: Obtain the reshaped result of the feature map after preprocessing in the odd-numbered stage as the front feature map, and obtain the reshaped result of the feature map after preprocessing in the next stage of the odd-numbered stage as the back feature map; Performing matrix multiplication on the rear feature map and the transpose of the front feature map to obtain a multiplication result; A softmax operation is performed on the multiplication result to obtain the corresponding channel attention map.

3. The semantic segmentation method according to claim 1, characterized in that: The multi-scale convolution processing is performed on the feature map of the last stage to obtain the processed deep feature map including: Extracting local context information of the feature map of the last stage using a preset depth convolution operation to obtain a feature map containing local context information; Extracting multi-scale context information of the feature map containing local context information by using multi-branch depth-separable strip convolution to obtain multiple depth feature maps; The multiple depth feature maps are spliced ​​along the channel dimension to obtain a depth splicing result, the depth splicing result is subjected to 1x1 convolution processing and then element-by-element multiplication with the last stage feature map to obtain a processed deep feature map.

4. The semantic segmentation method according to claim 3, characterized in that: Each branch of the depth-separable strip convolution includes a first depth-separable strip convolution and a second depth-separable strip convolution; The convolution kernel size of the first depth-separable strip convolution is 1 times the preset size; The convolution kernel size of the second depth-wise separable strip convolution is a preset size multiplied by 1; The preset size corresponds to a branch depth separable strip convolution, and different branch depth separable strip convolutions correspond to different preset sizes.

5. The semantic segmentation method according to claim 4, characterized in that: The preset sizes corresponding to multi-branch depthwise separable strip convolution are 7, 11, 15, and 21 respectively.

6. The semantic segmentation method according to claim 1, characterized in that: The preset encoder is a hybrid transformer encoder MiT.

7. A semantic segmentation device for flood areas in SAR images, comprising: A main encoding module, used for performing semantic extraction on the SAR image using a preset encoder to obtain a multi-stage semantic feature map; the multi-stage semantic feature map includes a shallow feature map area, a last stage feature map, and an intermediate feature map area except the shallow feature map area and the last stage feature map; A noise filtering module, used for performing a preset process on each feature map in the shallow feature map area and then performing noise filtering to obtain a noise-filtered feature map of the corresponding stage; A multi-scale convolution module, used for performing multi-scale convolution processing on the feature map of the last stage to obtain a processed deep feature map; A fusion module is used to perform preset processing on the feature map in the intermediate feature map area to obtain a corresponding preprocessed intermediate feature map, and to splice the preprocessed intermediate feature map, the noise filtered feature map and the processed deep feature map to obtain an optimized feature map; A classification module, used for inputting the optimized feature map into a preset classifier to obtain a semantic segmentation result of the SAR image; Wherein, the shallow feature map area includes at least the first stage feature map and the second stage feature map; Each feature map in the shallow feature map area is subjected to a preset process and then subjected to noise filtering to obtain a feature map after noise filtering at a corresponding stage, including: Performing preset processing on each feature map in the shallow feature map area to obtain a preprocessed feature map of the corresponding stage; Based on the feature map after preprocessing in the odd-numbered stage and the feature map after preprocessing in the next stage of the odd-numbered stage, obtain the corresponding channel attention map; The odd-numbered stage preprocessed feature map, the next stage preprocessed feature map of the odd-numbered stage and the corresponding channel attention map are fused to obtain the corresponding noise-filtered feature map.

8. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the semantic segmentation method according to any one of claims 1 to 6 is implemented.

9. A terminal, characterized in that: It includes a processor and a memory, and the memory is communicatively connected to the processor; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the semantic segmentation method as described in any one of claims 1 to 6.

Images