Coastal culture pond semantic segmentation method based on multi-task learning

Abstract

The invention discloses a coastal culture pond semantic segmentation method based on multi-task learning, which belongs to the technical field of remote sensing image processing, is used for intelligent extraction of a coastal culture pond, and comprises the following steps of: simultaneously learning masks, edges and distances of the coastal culture pond under a unified framework, and extracting the masks, the edges and the distances of the coastal culture pond through weight sharing among tasks; the accuracy of the extraction result of the coastal culture pond is effectively improved; an improved ConvNeXt V2 block is used as a network construction unit to form a multi-task network of a single encoder-decoder, so that the performance of the multi-task network is fully exerted while the light weight of the network is ensured; the hierarchical multi-head attention oriented to the multi-task convolutional network is designed by taking a Transform multi-head attention mechanism as a reference, and the features are effectively enhanced according to the features of different tasks. Compared with the prior art, the method has the advantages that the performance of a single task is improved, the problem of boundary adhesion is solved, and the extraction accuracy and the model generalization are improved.

Description

Technical Field

[0001] The invention discloses a semantic segmentation method of a coastal aquaculture pond based on multi-task learning, and belongs to the technical field of remote sensing image processing. Background Art

[0002] Intelligent extraction of coastal aquaculture ponds not only provides important data support for marine resource management and environmental protection, but also provides a scientific basis for the planning and management of the aquaculture industry. The difficulty in extracting coastal aquaculture ponds lies in the fact that coastal aquaculture ponds are distributed in coastal areas with complex environments, and are closely arranged, large in number, and non-uniform in shape. Therefore, traditional methods such as object-based image analysis perform poorly in terms of extraction accuracy and reproducibility. Deep learning, as a learning method under the current wave of artificial intelligence, has greatly promoted the development of image semantic segmentation technology. By building a network model, the computer can autonomously learn the features of land objects in remote sensing images through the model and have good expression capabilities.

[0003] Existing semantic segmentation methods for aquaculture ponds in coastal zones mainly focus on improving the accuracy and stability of coastal aquaculture pond extraction by improving single models and introducing complex feature fusion techniques. However, these methods usually require complex network structures and feature fusion strategies, which increase the complexity and training difficulty of the model, and due to design limitations, these models are often difficult to remain stable in different scenarios. In response to these problems, multi-task learning provides an effective solution. By designing several different tasks in a unified network framework, based on information sharing between tasks, the model can learn richer and more stable feature expressions, thereby improving extraction accuracy and generalization ability. Therefore, the present invention provides a semantic segmentation method for coastal aquaculture ponds based on multi-task learning, corresponding to a lightweight network structure, while learning the mask, boundary and distance features of coastal aquaculture ponds, effectively improving the boundary adhesion problem of adjacent coastal aquaculture ponds, and improving the migration ability of the model in different scenarios. Summary of the invention

[0004] The object of the present invention is to provide a semantic segmentation method for coastal aquaculture ponds based on multi-task learning to solve the problem of boundary adhesion of adjacent aquaculture ponds in intelligent extraction of coastal aquaculture ponds in the prior art.

[0005] A semantic segmentation method for coastal aquaculture ponds based on multi-task learning, including:

[0006] S1 is a dataset produced based on high-resolution remote sensing images of the coastal zone, including images, masks, edges and distances, and divides the coastal aquaculture pond vector files into training areas and test areas;

[0007] S2 built a coastal aquaculture pond semantic segmentation model based on multi-task learning, including a multi-task network architecture based on encoder-decoder, improved ConvNeXt V2 blocks, and hierarchical multi-head attention;

[0008] S3 builds a multi-task loss function based on the task type;

[0009] S4 uses a new training method to train the model to obtain prediction results;

[0010] S5 repairs the segmentation results based on morphological operations, including clearing broken blocks and filling holes, and converts the final processed results into a vector format used in engineering.

[0011] S1 includes: S1.1 obtaining vector files of coastal aquaculture ponds in high-resolution remote sensing images of the coastal zone, and dividing them into training areas and test areas; processing the images and vector files based on the data processing method of the gdal library, and generating mask label sets and image sets with fixed sizes and overlapping rates.

[0012] S1 includes, S1.2 generates boundary label set and distance label set based on canny edge detection and Euclidean distance, and the Euclidean distance changes from each pixel to the nearest non-zero pixel:

[0013]

[0014] Where (x1, y1) and (x2, y2) represent two points in space, and d(x, y) represents the Euclidean distance between the two points.

[0015] S2 includes, S2.1 adopts a multi-task network architecture to establish a multi-task weight sharing network for mask extraction, edge detection and distance estimation.

[0016] S2 includes, S2.2 develops a hierarchical multi-head attention HM-HA suitable for multi-task convolutional networks at the decoder end of the network architecture. For the input feature X∈[B,C,H,W], B is the batch size, C is the number of channels, H and W are the height and width of the feature map, respectively, HM-HA calculates the maximum pooling and average pooling results of the input feature, and connects along the channel dimension to obtain the output feature Z:

[0017] Z=cat(max(X),mean(X),dim=1);

[0018] Where cat(·, dim=1) is the output feature of all attention heads connected along the channel dimension, max(X) is the maximum pooling result of X, and mean(X) is the average pooling result of X;

[0019] For each task, calculate the output Y t :

[0020]

[0021] Where σ(·) is the Sigmoid activation function, which is used to perform nonlinear transformation on the convolution result; Xonv h,t (·) is the h-th convolution operation of the t-th task, used to calculate attention; ⊙ is the element-wise multiplication of attention and input features, It is calculated for all attention heads of the task.

[0022] The multi-task network architecture in S2.1 is a single encoder-decoder architecture composed of improved ConvNeXt V2 blocks as basic building blocks;

[0023] The improved ConvNeXt V2 block adds a 3×3 convolution layer, an InstanceNorm normalization layer, and a GELU activation function based on the ConvNeXt V2 block;

[0024] The encoder uses 5 improved ConvNeXt V2 blocks to increase the feature dimension. The first output is A1, and it is downsampled 4 times. The output of each time is d1, d2, d3 and d4. The number of channels varies from 32-64-128-256-512, and the downsampling parameters are 2-2-2-4.

[0025] After the decoder upsamples the spatial resolution of d4 output to the size of d3, it is fused with d3 in the channel dimension, and the fused features are compressed using the improved ConvNeXt V2 block. The fusion and processing process is repeated until d2 and A1, the number of channels changes to 256-128-64-32, and the upsampling parameters are 4-2-2-2;

[0026] After the hierarchical multi-head attention performs feature reweighting on the mask prediction, edge detection and distance estimation tasks, the mask prediction, edge detection and distance estimation tasks are output processed respectively, and the number of output channels is 2-2-1 respectively.

[0027] S3 includes, S3.1 uses the negative log-likelihood loss function for loss calculation in the classification task. For N samples, the negative log-likelihood loss l nll The pixel-level classification loss is calculated as:

[0028]

[0029] In the formula, p(y i |x i ) is a given input x i The actual label y calculated by the softmax activation function is iThe predicted probability of , i represents the i-th sample.

[0030] S3 includes, S3.2 uses mean square error to obtain loss, N sample distance estimation task mean square error l dist for:

[0031]

[0032] In the formula, y i and are the true distance and the predicted distance respectively.

[0033] S3 includes, S3.3 weighted loss for classification and regression tasks l total calculate:

[0034] l total =ω1l mask +ω2l contour +ω3l dist ;

[0035] Where ω1, ω2 and ω3 correspond to the weight coefficients of mask prediction, edge detection and distance estimation tasks, respectively, with a ratio of 1:1:1; l mask is the mean square error of the mask prediction task, l contour is the mean square error of the edge detection task.

[0036] S4 includes:

[0037] S4.1 optimizer uses AdamW and sets momentum β1, β2, β1 = 0.9, β2 = 0.999;

[0038] S4.2 weight decay coefficient is set to 0.05;

[0039] S4.3 Learning scheduler uses CosineAnnealing and sets the base learning rate to 1e -3 ;

[0040] In the S4.4 basic block, the random depths are set to 0, 0, 0, 0, 0.1, 0.1, 0.2, and the layer scaling factor is 1e -6 ;

[0041] S4.5 weight initialization;

[0042] S4.6 uses automatic mixed precision for training;

[0043] S4.7 uses exponential moving average for training and sets the parameter to 0.9999;

[0044] S4.8 enhances the data using color jitter, vertical flip, diagonal flip, and horizontal flip.

[0045] Compared with the existing technology, the present invention has the following beneficial effects: it makes full use of the multi-level information of remote sensing images, improves the performance of a single task by sharing weights between tasks, solves the boundary adhesion problem well, and improves the extraction accuracy and model generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a technical flow chart of the present invention;

[0047] Figure 2 This is an architecture diagram of a semantic segmentation method for coastal aquaculture ponds based on multi-task learning proposed by the present invention;

[0048] Figure 3 Attention feature map generated by the hierarchical multi-head attention designed for the present invention for coastal aquaculture ponds;

[0049] Figure 4 This is the result diagram of mask prediction in the multi-task network;

[0050] Figure 5 It is the result graph of edge detection in multi-task network;

[0051] Figure 6 This is the result image after morphological restoration of mask prediction. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] A semantic segmentation method for coastal aquaculture ponds based on multi-task learning constructs a multi-task semantic segmentation network by constructing a dataset and establishing a multi-task loss function to intelligently extract coastal aquaculture ponds, such as Figure 1 shown.

[0054] A semantic segmentation method for coastal aquaculture ponds based on multi-task learning, including:

[0055] S1 is a dataset produced based on high-resolution remote sensing images of the coastal zone, including images, masks, edges and distances, and divides the coastal aquaculture pond vector files into training areas and test areas;

[0056] S2 built a coastal aquaculture pond semantic segmentation model based on multi-task learning, including a multi-task network architecture based on encoder-decoder, improved ConvNeXt V2 blocks, and hierarchical multi-head attention;

[0057] S3 builds a multi-task loss function based on the task type;

[0058] S4 uses a new training method to train the model to obtain prediction results;

[0059] S5 repairs the segmentation results based on morphological operations, including clearing broken blocks and filling holes, and converts the final processed results into a vector format used in engineering.

[0060] S1 includes: S1.1 obtaining vector files of coastal aquaculture ponds in high-resolution remote sensing images of the coastal zone, and dividing them into training areas and test areas; processing the images and vector files based on the data processing method of the gdal library, and generating mask label sets and image sets with fixed sizes and overlapping rates.

[0061] S1 includes, S1.2 generates boundary label set and distance label set based on canny edge detection and Euclidean distance, and the Euclidean distance changes from each pixel to the nearest non-zero pixel:

[0062]

[0063] Where (x1, y1) and (x2, y2) represent two points in space, and d(x, y) represents the Euclidean distance between the two points.

[0064] S2 includes, S2.1 adopts a multi-task network architecture to establish a multi-task weight sharing network for mask extraction, edge detection and distance estimation.

[0065] S2 includes, S2.2 develops a hierarchical multi-head attention HM-HA suitable for multi-task convolutional networks at the decoder end of the network architecture. For the input feature X∈[B,C,H,W], B is the batch size, C is the number of channels, H and W are the height and width of the feature map, respectively, HM-HA calculates the maximum pooling and average pooling results of the input feature, and connects along the channel dimension to obtain the output feature Z:

[0066] Z=cat(max(X),mean(X),dim=1);

[0067] Where cat(·, dim=1) is the output feature of all attention heads connected along the channel dimension, max(X) is the maximum pooling result of X, and mean(X) is the average pooling result of X;

[0068] For each task, calculate the output Y t :

[0069]

[0070] Where σ(·) is the Sigmoid activation function, which is used to perform nonlinear transformation on the convolution result; Conv h,t (·) is the h-th convolution operation of the t-th task, used to calculate attention; ⊙ is the element-wise multiplication of attention and input features, It is calculated for all attention heads of the task.

[0071] The multi-task network architecture in S2.1 is a single encoder-decoder architecture composed of improved ConvNeXt V2 blocks as basic building blocks;

[0072] The improved ConvNeXt V2 block adds a 3×3 convolution layer, an InstanceNorm normalization layer, and a GELU activation function based on the ConvNeXt V2 block;

[0073] The encoder uses 5 improved ConvNeXt V2 blocks to increase the feature dimension. The first output is A1, and it is downsampled 4 times. The output of each time is d1, d2, d3 and d4. The number of channels varies from 32-64-128-256-512, and the downsampling parameters are 2-2-2-4.

[0074] After the decoder upsamples the spatial resolution of d4 output to the size of d3, it is fused with d3 in the channel dimension, and the fused features are compressed using the improved ConvNeXt V2 block. The fusion and processing process is repeated until d2 and A1, the number of channels changes to 256-128-64-32, and the upsampling parameters are 4-2-2-2;

[0075] After the hierarchical multi-head attention performs feature reweighting processing on the mask prediction, edge detection and distance estimation tasks, the mask prediction, edge detection and distance estimation tasks are output respectively, and the number of output channels is 2-2-1 respectively. The result of mask prediction is as follows: Figure 4 As shown, the results of edge detection are as follows Figure 5 shown.

[0076] S3 includes, S3.1 uses the negative log-likelihood loss function for loss calculation in the classification task. For N samples, the negative log-likelihood loss l nll The pixel-level classification loss is calculated as:

[0077]

[0078] In the formula, p(y i |x i ) is a given input x i The actual label y calculated by the softmax activation function is i The predicted probability of , i represents the i-th sample.

[0079] S3 includes, S3.2 uses mean square error to obtain loss, N sample distance estimation task mean square error l dist for:

[0080]

[0081] In the formula, y i and are the true distance and the predicted distance respectively.

[0082] S3 includes, S3.3 weighted loss for classification and regression tasks l total calculate:

[0083] l total =ω1l mask +ω2l contour +ω3l dist ;

[0084] Where ω1, ω2 and ω3 correspond to the weight coefficients of mask prediction, edge detection and distance estimation tasks, respectively, with a ratio of 1:1:1; l mask is the mean square error of the mask prediction task, l contour is the mean square error of the edge detection task.

[0085] S4 includes:

[0086] S4.1 optimizer uses AdamW and sets momentum β1, β2, β1 = 0.9, β2 = 0.999;

[0087] S4.2 weight decay coefficient is set to 0.05;

[0088] S4.3 Learning scheduler uses CosineAnnealing and sets the base learning rate to 1e -3 ;

[0089] In the S4.4 basic block, the random depths are set to 0, 0, 0, 0, 0.1, 0.1, 0.2, and the layer scaling factor is 1e -6 ;

[0090] S4.5 weight initialization;

[0091] S4.6 uses automatic mixed precision for training;

[0092] S4.7 uses exponential moving average for training and sets the parameter to 0.9999;

[0093] S4.8 uses color jitter, vertical flip, diagonal flip, and horizontal flip to enhance the data. The results after morphological repair of the mask prediction are shown in the figure below. Figure 6shown.

[0094] The architecture of the semantic segmentation method of coastal aquaculture ponds based on multi-task learning proposed in this paper is as follows: Figure 2 As shown in the figure, downsample and upsample are respectively downsampling and upsampling of the previous layer features; GRN (Global Response Normalization Layer) is a regularization layer used to improve the training and performance of neural networks. Its main purpose is to enhance the generalization ability and stability of the network by standardizing and normalizing the global response; LN (LayerNorm) and IN (InstanceNorm) are two normalization techniques that help the network to converge more stably and faster during training and improve the performance of the model; Maxpool and AvgPool are maximum pooling and average pooling of input features; Sigmoid is an activation function; Cat means stacking features in the channel dimension; Mean is the average of the elements in the same position of a set of feature matrices.

[0095] The hierarchical multi-head attention designed in this invention generates attention features for coastal aquaculture ponds as follows Figure 3 As shown in Figure 1, the attention visualization of the HM-HA module. The top column is the input image, the middle column is the attention map without the HM-HA module, and the bottom column is the attention map generated by the HM-HA module. It can be seen that the HM-HA module effectively distinguishes the boundaries ( Figure 3 The red circles a, b, c, and f in the figure also enhance the extraction capacity for irregular coastal aquaculture ponds ( Figure 3 The red circle d) and the ability to distinguish easily confused objects ( Figure 3 Red circle e).

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for coastal aquaculture ponds based on multi-task learning, It is characterized in that include: S1 is a dataset produced based on high-resolution remote sensing images of the coastal zone, including images, masks, edges and distances, and divides the coastal aquaculture pond vector files into training areas and test areas; S2 built a coastal aquaculture pond semantic segmentation model based on multi-task learning, including an encoder-decoder based multi-task network architecture, an improved ConvNeXt V2 block, and hierarchical multi-head attention; S3 builds a multi-task loss function based on the task type; S4 uses a new training method to train the model to obtain prediction results; S5 repairs the segmentation results based on morphological operations, including clearing broken blocks and filling holes, and converts the final processed results into a vector format used in engineering.

2. A semantic segmentation method for coastal aquaculture ponds based on multi-task learning according to claim 1, It is characterized in that S1 includes, S1.1 obtaining the vector file of coastal aquaculture ponds in the high-resolution remote sensing image of the coastal zone, and dividing it into training area and test area; Based on the data processing method of the GDAL library, the images and vector files are processed to generate mask label sets and image sets with fixed size and overlap rate.

3. A coastal aquaculture pond semantic segmentation method based on multi-task learning according to claim 2, It is characterized in that S1 includes, S1.2 generates boundary label set and distance label set based on canny edge detection and Euclidean distance, and the Euclidean distance changes from each pixel to the nearest non-zero pixel: In the formula, (x 1 ,y 1 ) and (x 2 ,y 2 ) represents two points in space, and d(x,y) represents the Euclidean distance between the two points.

4. A coastal aquaculture pond semantic segmentation method based on multi-task learning according to claim 3, It is characterized in that S2 includes, S2.1 adopts a multi-task network architecture to establish a multi-task weight sharing network for mask extraction, edge detection and distance estimation.

5. A coastal aquaculture pond semantic segmentation method based on multi-task learning according to claim 4, It is characterized in that S2 includes, S2.2 develops a hierarchical multi-head attention HM-HA suitable for multi-task convolutional networks at the decoder end of the network architecture. For the input feature X∈[B,C,H,W], B is the batch size, C is the number of channels, H and W are the height and width of the feature map, respectively, HM-HA calculates the maximum pooling and average pooling results of the input feature, and connects along the channel dimension to obtain the output feature Z: Z =cat(max(X),mean(X),dim=1); Where cat(·, dim=1) is the output feature of all attention heads connected along the channel dimension, max(X) is the maximum pooling result of X, and mean(X) is the average pooling result of X; For each task, calculate the output Y t : Where σ(·) is the Sigmoid activation function, which is used to perform nonlinear transformation on the convolution result; Conv h,t (·) is the h-th convolution operation of the t-th task, used to calculate attention; ⊙ is the element-wise multiplication of attention and input features, It is calculated for all attention heads of the task.

6. A method for semantic segmentation of coastal aquaculture ponds based on multi-task learning according to claim 5, It is characterized in that The multi-task network architecture in S2.1 is a single encoder-decoder architecture composed of improved ConvNeXt V2 blocks as basic building blocks; The improved ConvNeXt V2 block adds a 3×3 convolution layer, an InstanceNorm normalization layer, and a GELU activation function based on the ConvNeXt V2 block; The encoder uses 5 improved ConvNeXt V2 blocks to increase the feature dimension. The first output is A1, and it is downsampled 4 times. The output of each time is d1, d2, d3 and d4. The number of channels varies from 32-64-128-256-512, and the downsampling parameters are 2-2-2-4. After the decoder upsamples the spatial resolution of d4 output to the size of d3, it is fused with d3 in the channel dimension, and the fused features are compressed using the improved ConvNeXt V2 block. The fusion and processing process is repeated until d2 and A1, the number of channels changes to 256-128-64-32, and the upsampling parameters are 4-2-2-2; After the hierarchical multi-head attention performs feature reweighting on the mask prediction, edge detection and distance estimation tasks, the mask prediction, edge detection and distance estimation tasks are output processed respectively, and the number of output channels is 2-2-1 respectively.

7. A method for semantic segmentation of coastal aquaculture ponds based on multi-task learning according to claim 6, It is characterized in that S3 includes, S3.1 uses the negative log-likelihood loss function for loss calculation in the classification task. For N samples, the negative log-likelihood loss l nll The pixel-level classification loss is calculated as: In the formula, p(y i |x i ) is a given input x i The actual label y calculated by the softmax activation function is i The predicted probability of , i represents the i-th sample.

8. A method for semantic segmentation of coastal aquaculture ponds based on multi-task learning according to claim 7, It is characterized in that S3 includes, S3.2 uses mean square error to obtain loss, N sample distance estimation task mean square error l dist for: In the formula, y i and are the true distance and the predicted distance respectively.

9. A method for semantic segmentation of coastal aquaculture ponds based on multi-task learning according to claim 8, It is characterized in that S3 includes, S3.3 weighted loss for classification and regression tasks l total calculate: l total =ω 1 l mask +oh 2 l contour +oh 3 l dist ; In the formula, ω 1 ,ω 2 and ω 3 The weight coefficients corresponding to the mask prediction, edge detection and distance estimation tasks are 1:1:1 respectively; l mask is the mean square error of the mask prediction task, l contour is the mean square error of the edge detection task.

10. A method for semantic segmentation of coastal aquaculture ponds based on multi-task learning according to claim 9, It is characterized in that S4 include: S4.1 optimizer uses AdamW and sets momentum β1, β2, β1 = 0.9, β2 = 0.999; S4.2 weight decay coefficient is set to 0.05; S4.3 Learning scheduler uses CosineAnnealing and sets the base learning rate to 1e -3 ; In the S4.4 basic block, the random depths are set to 0, 0, 0, 0, 0.1, 0.1, 0.2, and the layer scaling factor is 1e -6 ; S4.5 weight initialization; S4.6 uses automatic mixed precision for training; S4.7 uses exponential moving average for training and sets the parameter to 0.9999; S4.8 enhances the data using color jitter, vertical flip, diagonal flip, and horizontal flip.

Images