Water semantic segmentation method and system based on sandwich decoding network, and readable storage medium

Abstract

The invention discloses a water body semantic segmentation method and system based on a sandwich decoding network, and a readable storage medium, belongs to the technical field of remote sensing image segmentation, and solves the problem that the existing neural network neglects the segmentation precision and calculation amount balance optimization of water bodies with different resolutions. The segmentation method comprises the steps of performing feature aggregation on remote sensing water body images, performing vector coding on feature maps, performing vector coding on feature maps of different layers in a backbone network, coding output of convolution down-sampling, retaining original information of the images in a dimension reduction process, and finally obtaining image features and vector codes of the image features. And step-by-step sending into a sandwich decoder for decoding, and sending a final result into a semantic segmentation head to realize semantic recognition and segmentation of each main body part in the remote sensing image. Based on the two-step self-attention strategy and the multi-scale fusion convolution module, spatial feature aggregation is realized, and the segmentation precision and the expansion capability of a computer vision model on a water body extraction task are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image segmentation, and more specifically, relates to a water body semantic segmentation method, system and readable storage medium based on a sandwich decoding network. Background Art

[0002] With the rapid development of sensor technology and observation platforms, a large number of remote sensing images with different temporal and spatial resolutions have been obtained, and these data are widely used in various downstream task research. Among the many remote sensing image processing tasks, water body segmentation tasks are closely related to vegetation growth, disaster prediction, environmental protection, urban planning, etc., especially to surface mapping and disaster emergency response. Therefore, they are closely watched by researchers.

[0003] However, the high resolution of water remote sensing images, as well as the diversity and differences in the shape, size, color and texture of water bodies, pose great challenges to the accuracy of water recognition and segmentation and the computational efficiency of algorithms in machine vision. In the semantic segmentation task of water targets in remote sensing images, the accuracy of neural networks in the recognition and segmentation of water bodies with different resolutions is low. The existing neural networks ignore the balanced optimization of the segmentation accuracy and computational complexity of water bodies with different resolutions, and there are problems with the large amount of computation and high computational overhead of traditional self-attention algorithms. Summary of the invention

[0004] The present invention aims at solving the technical problems existing in the prior art and provides a method, system and readable storage medium for water body semantic segmentation based on a sandwich decoding network.

[0005] In order to solve the above technical problems, the present invention first provides a water body semantic segmentation method based on a sandwich decoding network, comprising: Feature aggregation of remote sensing water body images; construct a deep learning model SDNet, using convolutional networks or self-attention networks as the backbone network and using them as the feature extraction part of SDNet to extract deep information of images; Vector encoding of feature maps; the feature extraction part of SDNet will perform hierarchical downsampling of the input original image. Each downsampling will reduce the resolution of the output feature map and increase the number of channels of the feature map. The output of each downsampling level is vector encoded through the fully connected layer to retain and align the original information of the image during the dimensionality reduction process. The sandwich decoder decodes; the image features and the vector encoding of the image features in the above steps are obtained, and are sent to the sandwich decoder for decoding step by step, and the final result is sent to the semantic segmentation head to realize the semantic recognition and segmentation of each main part in the remote sensing image.

[0006] Preferably, in the feature extraction process, four-times distributed downsampling is used to obtain semantic information of multiple scales. The four-times convolution sampling results extracted in the backbone network are subjected to two operations. The first operation is to use them as inputs to the next level one by one, and the second operation is to input the feature maps into the fully connected layer for vector encoding. The obtained feature vectors are input into the middle layer of the sandwich decoder of the corresponding level, and the original information of the image is added.

[0007] Preferably, the sandwich decoder has three layers, namely the first layer, the middle layer and the third layer. The first layer is a two-step self-attention strategy, and the third layer is a multi-scale fusion convolution module.

[0008] Preferably, the two-step self-attention strategy includes: Similar feature selection; including: Use small-scale convolution kernels to perform convolution and split operations to obtain Q, K, V tensors of the same shape as the feature map. ; In the formula, is the input feature map, They are the convolution kernels for splitting the Q, K, and V tensors respectively; Set a non-overlapping window of size w×w, use average pooling and maximum pooling respectively, add the corresponding channels of the feature maps obtained by the two different pooling operations, and realize regional information aggregation of the tensor on the whole image. After different pooling, Q and K are used to calculate the semantic similarity between regions; ; ; ; In the formula, Q , K The shape of the tensor is , all values ​​are real numbers, , Respectively represent the average pooling and maximum pooling operations on the tensor, and They respectively represent the results of channel addition after different pooling; In all the similarity matrices calculated, select the k with the largest values ​​as the regions of interest. For each non-overlapping window , only focus on the k most closely related , and use the small window for fine-grained feature differentiation: ; In the formula, It means selecting the k results with the largest similarity matrix values. Indicates that these selected results will be used as subsequent candidate inputs.

[0009] Preferably, the two-step self-attention strategy further includes: Detailed feature differentiation; including: After obtaining the k most interesting regions of each area, the original image is directly mapped to obtain Q, and the candidate regions are mapped to obtain key-value pairs. The original image mapping retains all the information of the original image, and the candidate area ROI mapping reflects some of the most concerned information of the original image. Then, a multi-head attention calculation is performed element by element to distinguish the differences in detail. ; In the formula, Represents the result of multi-head attention calculation on the tensor, the formula is based on Q , K Tensor to calculate similarity, divided by And apply the softmax function to obtain the weights and apply them to the required tensor V; After calculating the regional attention, convolution is performed layer by layer to improve the rank of the original feature map; ; In the formula, M out is the output of the first layer of the sandwich decoder, is the result of layer-by-layer convolution of the original feature map. The result of multi-head attention calculation on the attention area; The obtained results are sent to the middle layer of the sandwich decoder for calculation.

[0010] Preferably, the middle layer of the sandwich decoder is a semantic coding layer, which fuses the feature information extracted by decoding with the original information extracted by downsampling convolution to perform semantic compression mapping.

[0011] Preferably, it also includes: The sandwich decoder undergoes information fusion at the middle layer and inputs the fusion result into the multi-scale fusion convolution module at the third layer of the decoder. The multi-scale fusion convolution module constructs a dual-branch convolutional neural network. This network module has a bottleneck structure and compresses and reuses the input features. For a feature map with an input size of H×W and a number of channels of C Perform convolution operations and aggregate information through normalized pooling; A dual-branch operation is performed, and the feature maps obtained after information aggregation are subjected to layer-by-layer convolution and dilated convolution using convolution kernels of different sizes. Small-sized layer-by-layer convolution is used to extract local information, and large-sized dilated convolution is used to supplement surrounding semantics.

[0012] Preferably, it also includes: After performing the two-way convolution, the number of channels is compressed to half of the original number: ; ; ; In the formula, represents the feature map obtained by information aggregation, Represents the feature map obtained after layer-by-layer convolution and channel compression, represents the feature map obtained after dilated convolution and channel compression, represents the expansion rate; The calculation formula for the change in perceived visual field caused by the expansion rate is as follows: ; In the formula, RF represents the receptive field of view in the convolutional layer, and k represents the conventional convolution kernel size; The feature maps obtained by convolution are split in the channel dimension, and the features of the two branches are reorganized in the channel to construct multiple recognition modes for the model: ; ; ; ; In the formula, They represent the feature maps after convolution of different scales and channel splitting. Represents the feature map after channel reorganization; Perform a convolution on the reorganized feature map and activate it with an activation function to achieve information fusion between different branches: ; In the formula, Indicates that the result after convolution and activation of the original feature map is updated; The dual-branch feature map is summed and then dimensionally upgraded to restore the original number of channels. The original input identity mapping is introduced for residual connection to obtain the final semantic fusion result: ; In the formula, represents the final semantic fusion result, represents the feature map of the original input, They represent the results of information fusion respectively.

[0013] Furthermore, the present invention also provides a water body semantic segmentation system based on a sandwich decoding network. The segmentation system is provided with a program module corresponding to any of the above steps, and executes the steps in the semantic segmentation method based on the sandwich decoding network during operation.

[0014] Furthermore, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of any of the above-mentioned semantic segmentation methods based on the sandwich decoding network when called by a processor.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a new deep learning model SDNet. In the process of vector encoding, some important feature blocks are selected to realize spatial feature aggregation. The decoding part as a whole forms a network structure with a sandwich shape, and can be combined with the continuously developing mainstream encoders to update the performance. Compared with the existing methods, SDNet containing a sandwich decoder shows better detection sensitivity to tiny water bodies. The experimental results on the data set show that SDNet surpasses the existing SOTA methods, improves the average segmentation intersection-to-union ratio, and the computational overhead is also lower than the existing segmentation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is a schematic diagram of a network structure of a segmentation method according to a first embodiment of the present invention; Figure 2 A schematic diagram of the structure of a sandwich decoder according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a two-step self-attention strategy in an embodiment of the present invention; Figure 4 Schematic diagram of the structure of a multi-scale fusion convolution module in an embodiment of the present invention; Figure 5 It is a schematic diagram of the segmentation effect of the present invention on a large-scale river system water body; Figure 6 It is a schematic diagram of the segmentation effect of the present invention on a small-scale field pond water body. DETAILED DESCRIPTION

[0018] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the following is a further detailed description of a method, system and readable storage medium for water body semantic segmentation based on a sandwich decoding network provided by the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] See also Figure 1 The present invention provides a water body semantic segmentation method based on a sandwich decoding network, comprising: Feature aggregation of remote sensing water body images; construct a deep learning model SDNet, using convolutional networks or self-attention networks as the backbone network and using them as the feature extraction part of SDNet to extract deep information of images; Vector encoding of feature maps; the feature extraction part of SDNet will perform hierarchical downsampling of the input original image, and the output of each level of downsampling will be vector encoded through the fully connected layer to retain and restore the original information of the image during the dimensionality reduction process; The sandwich decoder decodes; the image features and the vector encoding of the image features in the above steps are obtained, and are sent to the sandwich decoder for decoding step by step, and the final result is sent to the semantic segmentation head to realize the semantic recognition and segmentation of each main part in the remote sensing image.

[0020] The present invention provides a new deep learning model SDNet. In the process of vector encoding, some important feature blocks are selected to realize spatial feature aggregation. The decoding part as a whole forms a network structure with a sandwich shape, and can be combined with the continuously developing mainstream encoders to update the performance. Compared with the existing methods, SDNet containing a sandwich decoder shows better detection sensitivity to tiny water bodies. The experimental results on the data set show that SDNet surpasses the existing SOTA methods, improves the average segmentation intersection-to-union ratio, and the computational overhead is also lower than the existing segmentation methods.

[0021] Embodiment 1 This embodiment provides a water body semantic segmentation method based on a sandwich decoding network. The overall network structure and process are as follows: Figure 1 As shown, it specifically includes the following steps: S1, feature aggregation of remote sensing water images; For the input water remote sensing images, mainstream convolutional networks with better performance, such as ConvNeXt or self-attention networks such as SegFormer, are selected as the backbone network to extract deep information of the image; at the same time, as the feature extraction part of SDNet, in the feature extraction process, quartic distribution downsampling is used to obtain semantic information at multiple scales and obtain feature maps at all levels required by the network.

[0022] S2, vector encoding of feature maps; The feature maps of different layers in the backbone network are vectorized and the output of convolutional downsampling is encoded to retain the original information of the image during the dimensionality reduction process.

[0023] S2.1. In the feature extraction process, the four convolution sampling results extracted in the backbone network are backbone network module 1, backbone network module 2, backbone network module 3, and backbone network module 4. After the feature aggregation network obtains four feature maps with different depths, different operations are taken on the feature maps and divided into two outputs; S2.2. The first path is to use the downsampled feature map as the input of the next sampling layer step by step. Each downsampling will reduce the resolution of the output feature map and increase the number of channels of the feature map. The second path is to input the feature map into the fully connected layer for vector encoding, and the obtained feature vectors are input into the middle layer of the sandwich decoder of the corresponding level to add the original information of the image.

[0024] This step adds more original image information to the back-end decoder through the input of feature vectors.

[0025] S3, sandwich decoder decoding module; The vector coding of the image features in the above steps is obtained, and the received vector coding of the image features is sent to the sandwich decoder for decoding step by step, and the final result is sent to the semantic segmentation head to realize the semantic recognition and segmentation of each main part in the remote sensing image.

[0026] In this embodiment, the sandwich decoder has three layers, namely, the first layer, the middle layer and the third layer. The first layer and the third layer are the lower layer and the upper layer respectively. The specific structure of the sandwich decoder is as follows: Figure 2 As shown in the figure, the first layer is the two-step self-attention strategy (TSSA), the third layer is the multi-scale fusion convolution module (MSFC), and the middle layer is the semantic encoding layer (Semantic Embedding).

[0027] In this embodiment, the SDNet network is a water body semantic segmentation method (Sandwich Decoder Block) based on a sandwich decoding network of the present invention, the TSSA (Two-Step Self-Attention) algorithm is a new self-attention mechanism proposed by the present invention, and the MSFC (Multi-Scale Fusion Convolution) is a new feature convolution fusion module proposed by the present invention.

[0028] Specifically include: S3.1 Self-attention strategy The advantage of the traditional self-attention mechanism is that it can find global semantic associations, but its disadvantage is that the amount of calculation is large. Remote sensing images generally have the characteristics of large resolution. The exponential increase in image resolution will bring quadratic computational consumption to the traditional self-attention mechanism. Based on this, in order to save some redundant information interaction expenses, this embodiment adopts a two-step self-attention strategy, firstly selecting similar features, and then calculating detailed feature distinction. The module structure of the two-step self-attention strategy is as follows: Figure 3 As shown, it includes the following steps.

[0029] S3.1.1, similar feature selection; like Figure 3 As shown, first, the size is H × W The feature map is cut into pieces of size S × S Then, we use a small-scale convolution kernel to perform convolution and split operations to obtain Q, K, and V tensors of the same shape as the feature map: ; In the formula, is the input feature map, These are the convolution kernels for splitting the Q, K, and V tensors respectively.

[0030] Then, a non-overlapping window of size w×w is set, and average pooling and maximum pooling are used respectively. The corresponding channels of the feature maps obtained by the two different pooling operations are added together to realize regional information aggregation of the tensor on the whole image. Q and K after different pooling are used to calculate the semantic similarity between regions. ; ; ; In the formula, Q , K The shape of the tensor is , all values ​​are real numbers, , Respectively represent the average pooling and maximum pooling operations on the tensor, and They respectively represent the results of channel addition after different pooling.

[0031] Finally, select the k regions with the largest values ​​from all the similarity matrices calculated as the regions of interest. , only focus on the k most closely related , and use the small window for subsequent fine-grained feature differentiation: ; In the formula, It means selecting the k results with the largest similarity matrix values. Indicates that these selected results will be used as subsequent candidate inputs.

[0032] All tensors calculated Q i , K i , tensor similarity, select the k with the largest similarity, and get Q j , K j , V j .

[0033] S3.1.2, detailed feature distinction; After obtaining the k candidate most interesting regions for each area, the original image is directly mapped to obtain Q (query), and the candidate regions are mapped to obtain key-value pairs. The original image mapping retains all the information of the original image, while the candidate area ROI mapping reflects some of the most concerned information of the original image. Then, an element-by-element multi-head attention calculation is performed, and regularization and softmax activation are performed to distinguish the differences in detail.

[0034] ; In the formula, Represents the result of multi-head attention calculation on the tensor, the formula is based on Q j , K j Tensor to calculate similarity, divided by And apply the softmax function to obtain the weights and apply them to the required tensors V j superior.

[0035] Consider the region of interest window size as , global MSA and ROI-MSA are based on a size of The feature map calculation consumption comparison is as follows: ; ; In the formula, H , W , C Represent the size, width, height and number of channels of the feature map respectively. and They represent the computational cost of global attention calculation and attention calculation on candidate parts respectively.

[0036] The above formula satisfies When the equal sign is taken, the amount of calculation is the least.

[0037] By setting appropriate parameters, a large number of redundant calculations can be removed, taking into account both segmentation accuracy and computational efficiency, and the original square complexity Reduce to .

[0038] After calculating the regional attention, convolution is performed layer by layer to improve the rank of the original feature map, further improving the network's modeling ability and differentiated expression ability.

[0039] ; In the formula, M out is the output of the first layer of the sandwich decoder, is the result of layer-by-layer convolution of the original feature map. The result of multi-head attention calculation on the attention area.

[0040] Finally, the obtained result is sent to the middle layer of the sandwich decoder for calculation.

[0041] S3.2. The feature information extracted by the lower layer decoding and the original information extracted by the downsampling convolution are fused in the middle layer of the sandwich decoder.

[0042] By fusing the feature information extracted by decoding with the original information extracted by downsampling convolution and performing semantic compression mapping, the perception information of different fields of view of remote sensing water images can be enriched, and then the fused results are input into the multi-scale fusion convolution module on the upper layer of the decoder.

[0043] S3.3. The sandwich decoder undergoes information fusion at the middle layer and inputs the fused result into the multi-scale fusion convolution module (MSFC) of the third layer of the decoder to compress and reuse the input features.

[0044] The multi-scale fusion convolution module constructs a dual-branch convolutional neural network. The network module presents a bottleneck structure. The structure is as follows Figure 4 shown.

[0045] First, the feature map with input size H×W and number of channels C is Perform convolution operations and aggregate information through normalized pooling.

[0046] Then, a dual-branch operation is performed. For the feature map obtained after information aggregation, convolution kernels of different sizes are used for layer-by-layer convolution and dilated convolution. Small-sized layer-by-layer convolution is used to extract local information, and large-sized dilated convolution is used to supplement the surrounding semantics.

[0047] After performing the two-way convolution, the number of channels is compressed to half of the original number: ; ; ; In the formula, represents the feature map obtained by information aggregation, Represents the feature map obtained after layer-by-layer convolution and channel compression, represents the feature map obtained after dilated convolution and channel compression, represents the expansion rate; The calculation formula for the change in perceived visual field caused by the expansion rate is as follows: ; In the formula, RF represents the receptive field of view in the convolutional layer, and k represents the conventional convolution kernel size.

[0048] In this embodiment, the dilated convolution can effectively consider slightly distant spatial features, which is convenient for establishing long-range dependencies on the feature map. However, too large a dilation rate will lead to too much zero padding, while too small a dilation rate is not conducive to long-range dependency modeling. Therefore, a suitable k value can be selected according to the characteristics of the data set and the requirements of the segmentation task.

[0049] The feature maps obtained by convolution are split in the channel dimension, and the features of the two branches are reorganized in the channel to construct multiple recognition modes for the model: ; ; ; ; In the formula, Respectively represent the features after layer-by-layer convolution and dilated convolution, They represent the features after channel splitting, They represent the characteristics after channel splicing and reorganization.

[0050] Perform a regular convolution on the reorganized feature map and activate it with an activation function to achieve information fusion between different branches: ; In the formula, Indicates updating the result of convolution and activation of the original feature map.

[0051] The obtained dual-branch feature maps are summed and then dimensionally upgraded to restore the original number of channels. The original input identity mapping is introduced for residual connection to obtain the final semantic fusion result: ; In the formula, represents the final semantic fusion result, represents the feature map of the original input, They represent the results of information fusion respectively.

[0052] The obtained results are sent to the segmentation head to obtain the results of semantic segmentation of water bodies in remote sensing images.

[0053] Specifically, Figure 4 As shown in the figure, first, a convolution operation is performed on the input feature map to obtain a 3×3 convolution; then a dual-branch operation is performed, and the feature map obtained after information aggregation is convolved and expanded layer by layer using 3×3 convolution and 7×7 dilated convolution respectively. After the two-way convolution is completed, the number of channels is compressed to half of the original; the feature map obtained after convolution is divided in the channel dimension, and the features of the two branches are reorganized through channels. A conventional convolution is performed on the reorganized feature map to achieve information fusion between different branches, obtain a 3×3 convolution, and perform regularization and activation; finally, the obtained dual-branch feature map is summed and then dimensionally upgraded to restore the original number of channels to obtain the final semantic fusion result.

[0054] The present invention proposes a water body semantic segmentation method based on a sandwich decoding network, and proposes a new deep learning model SDNet. It improves on the traditional self-attention mechanism and proposes a two-step self-attention strategy and a multi-scale fusion convolution module. Based on the two-step self-attention strategy and the multi-scale fusion convolution module, the present invention selects some important feature blocks, realizes spatial feature aggregation, and obtains semantic information from different levels of receptive fields; and realizes axial information interaction through channel segmentation and reorganization, which improves the segmentation accuracy and expansion ability of the SDNet model in water body extraction tasks, while maintaining a small amount of network parameters.

[0055] Embodiment 2 Furthermore, the present invention also provides a water body semantic segmentation system based on a sandwich decoding network. The segmentation system is provided with a program module corresponding to any of the above steps, and executes the steps in the above semantic segmentation method based on the sandwich decoding network during operation.

[0056] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment, and each step in the second embodiment is consistent with that of the first embodiment.

[0057] Embodiment 3 Furthermore, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of any of the above-mentioned semantic segmentation methods based on the sandwich decoding network when called by a processor.

[0058] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment, and each step in the third embodiment is consistent with that of the first embodiment.

[0059] Experimental environment and configuration: The present invention also constructs an implementation environment, configures an experimental segmentation system, and simulates a segmentation method. All models mentioned in the embodiments of the present invention are built using the pytorch framework and deployed on a single NVIDIA RTX 3080Ti 12GB GPU. In order to sensitively adjust parameters and converge the model faster, the AdamW optimizer and cosine annealing learning strategy are used to train all models. The initial learning rate of the optimizer is not fixed and depends on the specific performance of the network in the experiment. Other hyperparameters are kept at empirical values. , .

[0060] The experimental results show that compared with the existing influential methods, the SDNet containing the sandwich decoder shows better detection sensitivity for tiny water bodies. The SDNet of the present invention surpasses the existing SOTA methods, with an average segmentation intersection-over-union index improved by 2%-10%, and an F1 score improved by 3%-9%. The computational overhead is also lower than the existing better methods, showing better accuracy and completeness.

[0061] like Figure 5 , Figure 6 As shown in the figure, the segmentation method provided by the present invention improves the segmentation accuracy and expansion capability of the computer vision model in the water body extraction task, and shows better segmentation effect in both large-scale river system water bodies and small-scale field pond water bodies. Figure 5 , Figure 6 In the figure, the white area of ​​the image processing result is the water area. It can be found that the network model and entity segmentation method used in the present invention can accurately locate the existence of large-scale continuous water bodies or small-scale independent water bodies, accurately identify their ranges, and effectively propose water body areas. The present invention has a high water body recognition and segmentation accuracy rate in river water systems and field pond water bodies at different resolutions, can more accurately identify water body areas, and obtain more accurate water body target extraction results. The extraction accuracy is basically in line with the actual situation, and at the same time, the amount of calculation is reduced compared with other algorithms.

[0062] The present invention proposes a method, system and readable storage medium for semantic segmentation of water bodies based on a sandwich decoding network. Aiming at the characteristics of remote sensing water bodies such as large resolution, bird's-eye view, and small-sized targets, the present invention solves the problem that existing neural networks ignore the balanced optimization of segmentation accuracy and computational complexity of water bodies with different resolutions. In order to improve the segmentation accuracy of computer vision models in water body extraction tasks and reduce the occupation of computing resources, the present invention proposes a new deep learning model SDNet, improves the traditional self-attention technology and convolutional network, and proposes a sandwich decoder network based on a two-step self-attention strategy and a multi-scale fusion convolution module. Based on the two-step self-attention strategy and the multi-scale fusion convolution module, the sandwich decoder network selects some important feature blocks, realizes spatial feature aggregation, optimizes segmentation accuracy, and obtains semantic information from different levels of receptive fields; and realizes axial information interaction through channel segmentation and reorganization, which improves the segmentation accuracy and expansion capability of the SDNet model in water body extraction tasks, while maintaining a small amount of network parameters.

[0063] The segmentation method of the present invention adopts the pytorch deep learning framework to extract the semantic features of the water body remote sensing image, and fuses the features through the newly proposed sandwich decoder to finally achieve the segmentation effect. The decoding part of the present invention forms a network structure with a sandwich shape as a whole, and can be combined with the continuously developing mainstream encoder to update the performance. The present invention improves the accuracy of the water body segmentation task of remote sensing images and improves the computational efficiency of the algorithm. Compared with the existing methods, SDNet containing the sandwich decoder shows better detection sensitivity for tiny water bodies; the experimental results on the data set show that SDNet surpasses the existing SOTA method, the average segmentation intersection-to-union ratio is improved by 2%-10%, the F1 score is improved by 3%-9%, and the computational overhead is also lower than the existing better segmentation method, showing better accuracy and completeness.

[0064] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0065] In addition, in the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0066] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A water body semantic segmentation method based on a sandwich decoding network, characterized in that: include: Feature aggregation of remote sensing water body images; Build a deep learning model SDNet, use the convolutional network or self-attention network as the backbone network, and use it as the feature extraction part of SDNet to extract the deep information of the image; Vector encoding of feature maps; the feature extraction part of SDNet will perform hierarchical downsampling on the input original image, and the output of each level of downsampling will be vector encoded through the fully connected layer to retain and align the original information of the image during the dimensionality reduction process; The sandwich decoder decodes; the image features and the vector encoding of the image features in the above steps are obtained, and are sent to the sandwich decoder for decoding step by step, and the final result is sent to the semantic segmentation head to realize the semantic recognition and segmentation of each main part in the remote sensing image.

2. According to the method for water body semantic segmentation based on sandwich decoding network in claim 1, it is characterized in that: In the feature extraction process, four-times distributed downsampling is used to obtain semantic information at multiple scales. The four-times convolution sampling results extracted in the backbone network are subjected to two operations. The first operation is to use them as the input of the next layer one by one. The second operation is to input the feature maps into the fully connected layer for vector encoding. The obtained feature vectors are input into the middle layer of the sandwich decoder of the corresponding layer to add the original information of the image.

3. According to the method for water body semantic segmentation based on sandwich decoding network in claim 1, it is characterized in that: The sandwich decoder has three layers, namely the first layer, the middle layer and the third layer. The first layer is a two-step self-attention strategy, and the third layer is a multi-scale fusion convolution module.

4. The method for water body semantic segmentation based on sandwich decoding network according to claim 3 is characterized in that: The two-step self-attention strategy includes: Similar feature selection; including: Use small-scale convolution kernels to perform convolution and split operations to obtain Q, K, V tensors of the same shape as the feature map. ； In the formula, is the input feature map, They are the convolution kernels for splitting the Q, K, and V tensors respectively; Set a non-overlapping window of size w×w, use average pooling and maximum pooling respectively, add the corresponding channels of the feature maps obtained by the two different pooling operations, and realize regional information aggregation of the tensor on the whole image. After different pooling, Q and K are used to calculate the semantic similarity between regions; ； ； ； In the formula, Q , K The shape of the tensor is , all values ​​are real numbers, , Respectively represent the average pooling and maximum pooling operations on the tensor, and They respectively represent the results of channel addition after different pooling; In all the similarity matrices calculated, select the k with the largest values ​​as the regions of interest. For each non-overlapping window , only focus on the k most closely related , and use the small window for fine-grained feature differentiation: ； In the formula, It means selecting the k results with the largest similarity matrix values. Indicates that these selected results will be used as subsequent candidate inputs.

5. The method for water body semantic segmentation based on sandwich decoding network according to claim 4, characterized in that: The two-step self-attention strategy also includes: Detailed feature differentiation; including: After obtaining the k most interesting regions of each area, the original image is directly mapped to obtain Q, and the candidate regions are mapped to obtain key-value pairs. The original image mapping retains all the information of the original image, and the candidate area ROI mapping reflects some of the most concerned information of the original image. Then, a multi-head attention calculation is performed element by element to distinguish the differences in detail. ； In the formula, Represents the result of multi-head attention calculation on the tensor, the formula is based on Q , K Tensor to calculate similarity, divided by And apply the softmax function to obtain the weights and apply them to the required tensor V; After calculating the regional attention, convolution is performed layer by layer to improve the rank of the original feature map; ； In the formula, M out is the output of the first layer of the sandwich decoder, is the result of layer-by-layer convolution of the original feature map. The result of multi-head attention calculation on the attention area; The obtained results are sent to the middle layer of the sandwich decoder for calculation.

6. The method for water body semantic segmentation based on sandwich decoding network according to claim 3, characterized in that: The middle layer of the sandwich decoder is a semantic coding layer, which fuses the feature information extracted by decoding with the original information extracted by downsampling convolution to perform semantic compression mapping.

7. The method for water body semantic segmentation based on sandwich decoding network according to claim 6, characterized in that: Also includes: The sandwich decoder undergoes information fusion at the middle layer and inputs the fusion result into the multi-scale fusion convolution module at the third layer of the decoder. The multi-scale fusion convolution module constructs a dual-branch convolutional neural network. The network module presents a bottleneck structure and compresses and reuses the input features. For a feature map with an input size of H×W and a number of channels of C Perform convolution operations and aggregate information through normalized pooling; A dual-branch operation is performed, and the feature maps obtained after information aggregation are subjected to layer-by-layer convolution and dilated convolution using convolution kernels of different sizes. Small-sized layer-by-layer convolution is used to extract local information, and large-sized dilated convolution is used to supplement surrounding semantics.

8. The method for water body semantic segmentation based on sandwich decoding network according to claim 7, characterized in that: Also includes: After performing the two-way convolution, the number of channels is compressed to half of the original number: ； ； ； In the formula, represents the feature map obtained by information aggregation, Represents the feature map obtained after layer-by-layer convolution and channel compression, represents the feature map obtained after dilated convolution and channel compression, represents the expansion rate; The calculation formula for the change in perceived visual field caused by the expansion rate is as follows: ； In the formula, RF represents the receptive field of view in the convolutional layer, and k represents the conventional convolution kernel size; The feature maps obtained by convolution are split in the channel dimension, and the features of the two branches are reorganized in the channel to construct multiple recognition modes for the model: ； ； ； ； In the formula, They represent the feature maps after convolution of different scales and channel splitting. Represents the feature map after channel reorganization; Perform a convolution on the reorganized feature map and activate it with an activation function to achieve information fusion between different branches: ； In the formula, Indicates that the result after convolution and activation of the original feature map is updated; The dual-branch feature map is summed and then dimensionally upgraded to restore the original number of channels. The original input identity mapping is introduced for residual connection to obtain the final semantic fusion result: ； In the formula, represents the final semantic fusion result, represents the feature map of the original input, They represent the results of information fusion respectively.

9. A water body semantic segmentation system based on a sandwich decoding network, characterized in that: The segmentation system is provided with a program module corresponding to any one of the steps in claims 1-8, and executes the steps in the above-mentioned semantic segmentation method based on the sandwich decoding network when running.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a computer program, and the computer program is configured to implement the steps in the semantic segmentation method based on the sandwich decoding network according to any one of claims 1 to 8 when called by a processor.

Images