A lightweight semantic segmentation method applied to underwater obstacle detection

Abstract

The application discloses a lightweight semantic segmentation method applied to underwater obstacle detection, and comprises the following steps: S1, constructing a semantic segmentation network overall framework model; the overall network framework adopts a pyramid pooling architecture of PSPNet, and features in an image are extracted in parallel by using multiple pooling kernels; S2, replacing a backbone feature extraction network of the model in the overall network framework with a bottleneck structure in Mobilenetv2, so as to reduce the parameter quantity of the model; S3, introducing dilated convolution in the bottleneck structure, increasing the receptive field of the convolution kernel, and strengthening feature extraction; S4, adopting auxiliary branch training to avoid local optimization; S5, adopting a focal loss loss function to increase the loss value of a difficult-to-detect part in an image, so that the model focuses on the difficult-to-detect part in the image; S6, adopting a transformer attention mechanism to increase the connection of each part in the image, and outputting a predicted image. The application has high detection accuracy while ensuring real-time detection, and improves the water surface obstacle detection capability of an unmanned ship.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning, and particularly relates to a lightweight semantic segmentation method applied to water obstacle detection. BACKGROUND

[0002] As a kind of water robot, the surface unmanned vehicle obtains the surrounding environment information through various sensors. When the unmanned vehicle is autonomously moving, it needs to distinguish obstacles and drivable areas. At this time, not only the information of objects with targets such as surrounding ships and buoys needs to be obtained, but also the information of objects without targets such as water surface, shore and wharf needs to be obtained. The existing water surface unmanned vehicle environment perception method is mostly based on target detection or coastline detection. They cannot analyze the information of all objects in the water surface scene. Image semantic segmentation is a pixel-level classification of images, which is not affected by the appearance shape of objects and can well meet the demand of environment perception of the water surface unmanned vehicle when autonomously moving.

[0003] In order to evaluate the performance of the semantic segmentation network applied to ground obstacle detection in water obstacle detection. Cane et al. studied the performance of SegNet, ESPNet and Eet on the marine dataset. They divided the image into four categories: sky, water surface, obstacle and others. Considering the accuracy and real-time performance of obstacle detection, they did not subdivide the obstacle category. Bovcon et al. also studied the performance of classic semantic segmentation networks such as PSPNet, UNet and DeepLab on the marine dataset according to the above method. They used the MasTr1325 dataset for training and then used the most challenging dataset MODD2 for evaluation. From the results, it can be seen that the semantic segmentation network used for ground obstacle detection has poor performance in water obstacle detection. The main reason is that the water surface is dynamic, while the ground is static. Moreover, there are reflections and reflections in the water, which increase the detection difficulty. In view of the above situation, Bovcon et al. studied a new water obstacle detection network WaSR. The detection performance of WaSR is very outstanding, and the evaluation accuracy on MODD2 reaches 93.7%, which is the best semantic segmentation network in terms of detection performance. However, WaSR uses the fusion of IMU and camera, which has high detection cost, large model parameter quantity and slow inference speed, which is difficult to meet the real-time requirement of the unmanned vehicle. SUMMARY

[0004] The purpose of the present application is to provide a lightweight semantic segmentation method applied to water obstacle detection, which can guarantee the real-time detection while having high detection accuracy and improve the water obstacle detection capability of the unmanned vehicle.

[0005] To solve the above technical problems, the technical scheme of the present application is: a lightweight semantic segmentation method applied to underwater obstacle detection, comprising the following steps:

[0006] S1, construct a semantic segmentation network overall framework model, the overall network framework adopts the pyramid pooling architecture of PSPNet, and utilizes multiple pool kernels to extract feature information in the image in parallel;

[0007] S2, replace the backbone feature extraction network of the model in the overall network framework with the bottleneck structure in Mobilenetv2, to reduce the parameter quantity of the model and improve the real-time performance of the unmanned ship detection;

[0008] S3, introduce dilated convolution in the bottleneck structure, which is used to increase the receptive field of the convolution kernel and thus strengthen feature extraction;

[0009] S4, auxiliary branch training is adopted to avoid local optimization during model training;

[0010] S5, the focalloss loss function is used to increase the loss value of the difficult-to-detect part of the image, so that the model focuses on the difficult-to-detect part of the image;

[0011] S6, the transformer attention mechanism is used to increase the connection between each part of the image, and the predicted image is output.

[0012] The PSPNet network architecture in S1 reduces the number of feature layers in the model, resulting in fewer trainable parameters.

[0013] The bottleneck structure in S2 adopts the DW convolution method, which reduces the computational load and the parameter quantity of the model.

[0014] The dilated convolution in S3 increases the holes in the convolution kernel, so that the same size of the convolution kernel has a larger mapping area in the image.

[0015] The auxiliary branch in S4 introduces an auxiliary loss.

[0016] The focalloss loss function used in S5 is as follows:

[0017] FL(pt)=-αt(1-pt) γ log(pt)

[0018] In the formula, pt is the prediction probability, αt is the weight factor, and γ is the focus parameter.

[0019] The focal loss loss function is improved by introducing an auxiliary loss, further increasing the loss value of the difficult-to-detect target, improving the attention of the model to the difficult-to-detect target, and the improved focal loss function is as follows:

[0020] FL sum (p t )=-λ m α t (1-p tm ) γ log(p tm )-λ b α t (1-p tb ) γ log(p tb )

[0021] In the formula, m is a main branch, b is an auxiliary branch, λ m is a weight parameter of the main branch, λ b is a weight parameter of the auxiliary branch, p tm is a main branch prediction probability, and p tb is an auxiliary branch prediction probability.

[0022] The dynamic interference of the water surface in S6 is improved by using the Transformer block as the attention mechanism module, and the multi-head in the Transformer block is used to calculate the connection between a certain image area and the surrounding image area through Q, K and V parameters, so that the prediction accuracy of the image is improved.

[0023] Also provided is a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of the preceding embodiments when executing the computer program.

[0024] Also provided is a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method according to any one of the preceding embodiments.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] First, the water surface anti-interference problem is solved. Because the water surface is different from the ground, there are a large number of interference of reflection and refraction, which affect the accuracy of model detection, in order to improve the anti-interference ability of the model to the reflection and refraction, the present application adopts the way of increasing the receptive field, improves the extraction ability of the model to the feature information in the image. In order to increase the receptive field without producing too much calculation, the present application adopts the way of adding dilated convolution in the DW convolution of bottleneck structure, adds the hole in the convolution kernel, so that the mapping area of the convolution kernel in the feature map is larger, thereby increasing the receptive field.

[0027] Second, the problem of difficult to identify negative samples is solved. In the present application, the water obstacles are set as positive samples, and the water surface, sky and the like are set as negative samples. Because the water surface will reflect the color of the sky, combined with fog and strong light, it is difficult to distinguish the negative samples. In order to solve this problem, the present application adopts focalloss loss function, which can improve the attention of the model to the difficult to detect samples, thereby improving the detection accuracy of the negative samples. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flowchart of the embodiment of the present application is shown in the figure;

[0029] Figure 2 The Unet network model diagram in the embodiment of the present application is shown in the figure;

[0030] Figure 3 The PSPNet network model diagram in the embodiment of the present application is shown in the figure;

[0031] Figure 4 The architecture diagram of bottleneck structure in the embodiment of the present application is shown in the figure;

[0032] Figure 5 The dilated convolution diagram in the embodiment of the present application is shown in the figure;

[0033] Figure 6 The model structure diagram after introducing auxiliary branch in the embodiment of the present application is shown in the figure;

[0034] Figure 7 The complete semantic segmentation model diagram in the embodiment of the present application is shown in the figure;

[0035] Figure 8 The data set used in the embodiment of the present application is shown in the figure;

[0036] Figure 9 The detection accuracy (F-measure) comparison table in the embodiment of the present application is shown in the figure;

[0037] Figure 10 The model size and inference speed comparison table in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0039] The present application is intended to use NVIDIA-Telsa-K80 as a hardware experimental platform, use Linux operating system, and use pytorch neural network framework for training.

[0040] At the same time, the most accurate marine data set MaSTr1325 data set is used as the training data set, the MaSTr1325 data set contains 1325 artificial annotation data sets, involving various water surface environments. And the most challenging marine data set MODD2 is used as the evaluation data set, MODD2 has 11711 images, containing 28 various water surface scenes, the detection accuracy and generalization ability of the network model can be more accurately evaluated by using MODD2 data set. The MaSTr1325 data set and the MODD2 data set are shown in Figure 2 .

[0041] The most widely used semantic segmentation architecture for water surface obstacle detection is the UNet network architecture, UNet uses encoder and decoder architecture, as shown in Figure 3 .

[0042] If the Unet architecture shown in Figure 3 is used, it will cause the number of feature layers in the model to be larger and the amount of calculation to increase. Therefore, the present application decides to use the PSPNet network architecture, PSPNet uses a multi-pyramid pooling model to reduce the number of feature layers in the model, and the amount of trainable parameters is less, and PSPNet is shown in the figure.

[0043] As can be seen from Figure 4 , the model only has one splicing operation, while the Unet model has five splicing operations, therefore, the number of feature layers of the Unet model is more than that of the PSPNet, and the amount of calculation is larger, the FeatureMap in the PSPNet source network adopts resnet, the present application attempts to replace resnet with bottleneck in mobilenetv2, bottleneck adopts DW convolution method, which has smaller amount of calculation than CNN convolution method, therefore, using bottleneck can further reduce the amount of model parameters, and the architecture after using bottleneck structure is shown in Figure 5 .

[0044] The increase of the receptive field can increase the extraction of feature information by the model, and in the case of limiting the size of the model, the use of the dilated convolution method to increase the receptive field can extract more feature information from the feature layer without increasing the calculation amount, and the application determines to use the dilated convolution method to increase the receptive field, as shown in Figure 6 . Figure 6 In the drawings, the left drawing is a standard convolution kernel, and the right drawing is a convolution kernel using dilated convolution, and it can be seen that the dilated convolution increases the mapping area of the same size of the convolution kernel in the image by increasing the hole of the convolution.

[0045] In order to prevent the model from falling into a local optimal solution during training, the application determines to use an auxiliary branch to help the model jump out of the local optimum during training. The auxiliary loss is as shown in Figure 7 , and the use of the auxiliary branch can also introduce the auxiliary loss.

[0046] The PSPNet uses a cross-entropy loss function, and the cross-entropy loss function cannot make the model pay more attention to the difficult-to-detect places in the image, and there are a large number of reflections and refractions in the water surface environment, which are almost indistinguishable from the ground, which makes the cross-entropy loss function difficult to produce good results in water obstacle detection. Therefore, the application determines to use a focal loss loss function, as shown in the following formula:

[0047] FL(pt)=-αt(1-pt) γ log(pt)

[0048] pt represents the predicted probability

[0049] For difficult-to-detect targets, pt is small, which will cause the overall loss value to become large, therefore, the focal loss can pay more attention to the difficult-to-detect image information.

[0050] Due to the introduction of the auxiliary loss, the focal loss loss function can be improved, and the loss value of the difficult-to-detect target is further increased, and the attention of the model to the difficult-to-detect target is improved, and the improved focal loss function is as follows:

[0051] FL sum (p t )=-λ m α t (1-p tm ) γ log(p tm )-λ b α t (1-p tb ) γ log(p tb )

[0052] m represents the main branch

[0053] b - auxiliary branch

[0054] λ m - weight parameter of the main branch

[0055] λ b - weight parameter of the auxiliary branch

[0056] For the dynamic interference of the water surface, the application decides to use the transformer attention mechanism to increase the connection of each part in the image, so that the model also has strong anti-interference ability in the dynamic water surface. The application improves the Transfomer block as an attention mechanism module. The multi-head in the Transformer block can calculate the connection between a certain image area and the surrounding image area through Q, K and V parameters, so that the prediction of the entire image is more accurate. The detailed diagram of the Attention attention module used is as shown in Figure 7 .

[0057] In order to improve the detection efficiency, the application changes the input channel number of the feature map to 192, the encoder block to 6, and the number of heads to 3, and finally obtains a feature map of 28x28x192. The overall semantic segmentation model after introducing the attention block is as shown in Figure 8 , and the application names the segmentation model as LWDNet.

[0058] The lightweight semantic segmentation algorithm designed by the application uses F-measure as an evaluation index on the MODD2 dataset:

[0059] F-measure = 2TP / (2TP+FP+FN)

[0060] TP - the number of correctly detected obstacles

[0061] FP - the number of water surfaces misdetected as obstacles

[0062] FN - the number of water surfaces misdetected as sky

[0063] The application compares the prediction results with the most advanced network, and the results are as shown in Figure 9 . It can be seen from Figure 9 that the LWDNet designed by the application is only lower than the WaSR network in F-measure, and is better than other networks.

[0064] At the same time, the model size and inference speed of LWDNet are compared as shown in Figure 10 . From Figure 10As can be seen, LWDNet is still in the forefront in inference speed, far more than WaSR, and from the comparison of the computing platform, the computing speed of GTX1080Ti is higher than that of TeslaK80, so it can be seen that LWDNet is very excellent in detection accuracy and detection rate, and is a very effective underwater obstacle detection network.

[0065] Those skilled in the art can easily understand that the above description is only preferred embodiments of the present application, and is not used to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A lightweight semantic segmentation method for underwater obstacle detection, characterized in that, Includes the following steps: S1. Construct an overall framework model for the semantic segmentation network. The overall framework model adopts the pyramid pooling architecture of PSPNet and uses multiple pooling kernels to extract feature information from the image in parallel. S2. Replace the backbone feature extraction network of the model in the overall network framework with the bottleneck structure in Mobilenetv2 to reduce the number of model parameters and improve the real-time performance of unmanned surface vessel detection. S3. Introduce dilated convolution in the bottleneck structure. The dilated convolution is used to increase the receptive field of the convolution kernel, thereby enhancing feature extraction. S4. Use auxiliary branch training to avoid local optimization during model training; S5. The focal loss function is used to increase the loss value of hard-to-detect parts of the image, so that the model focuses on these hard-to-detect parts. An auxiliary loss function is introduced to improve the focal loss function, further increasing the loss value of hard-to-detect targets and improving the model's focus on these targets. The improved focal loss function is as follows: In the formula, m is the main branch, b is the auxiliary branch, and λ is the auxiliary branch. m The weight parameter for the main branch, λ b p is the weight parameter for the auxiliary branch. tm It is the main branch prediction probability, p tb It is an auxiliary branch that predicts probabilities; S6. Employ the transformer attention mechanism to increase the connections between different parts of the image and output the predicted image.

2. The lightweight semantic segmentation method for underwater obstacle detection according to claim 1, characterized in that, The PSPNet network architecture in S1 reduces the number of feature layers in the model, resulting in fewer trainable parameters.

3. The lightweight semantic segmentation method for underwater obstacle detection according to claim 1, characterized in that, The bottleneck structure in S2 uses DW convolution, which reduces the amount of computation and the number of model parameters.

4. The lightweight semantic segmentation method for underwater obstacle detection according to claim 1, characterized in that, In S3, dilated convolution adds holes to the convolution kernel, making the mapping area of ​​the same-sized convolution kernel in the image larger.

5. A lightweight semantic segmentation method for underwater obstacle detection according to claim 1, characterized in that, The auxiliary branch in S4 introduces an auxiliary loss.

6. A lightweight semantic segmentation method for underwater obstacle detection according to claim 5, characterized in that, The focal loss function used in S5 is as follows: In the formula, pt is the prediction probability, αt is the weighting factor, and γ is the focusing parameter.

7. A lightweight semantic segmentation method for underwater obstacle detection according to claim 1, characterized in that, In S6, to address dynamic disturbances on the water surface, the Transformer block is improved as an attention mechanism module, and multi-channel attention mechanisms are implemented within the Transformer block. The head calculates the relationship between a certain image region and surrounding image regions using Q, K, and V parameters, thereby improving the accuracy of image prediction.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements as claimed in claim 1.

7. The steps of any of the methods described.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements as described in claim 1.

7. The steps of any of the methods described.

Images