A submarine target detection method based on a double-branch lightweight backbone

Abstract

A kind of seabed target detection method based on double-branch lightweight backbone belongs to the field of side-scan sonar image seabed target detection. We design a double-branch lightweight backbone model DBnet, which uses two lightweight backbone networks with superior performance, PP-LCNet and GhostNet, to extract features from input images in the feature extraction stage. The simplified Neck structure can reduce the number of parameters while still better fusing the features extracted by the double backbone at each level, enhancing the target detection capability of different scales. In the inference stage, we use the SAHI method. According to the actual underwater operation of AUV, the whole waterfall image is cut into several slices with a certain overlap rate and fixed size, which are then input into the network. This effectively solves the problems of difficult deployment of target detection models for unmanned platforms such as AUV, low detection efficiency and low detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of side-scan sonar image seabed target detection research, and particularly relates to a seabed target detection model lightening method. BACKGROUND

[0002] The detection and recognition of seabed targets play an extremely important role in underwater search and rescue, marine engineering construction, marine topography and geomorphology measurement, marine resource survey and other fields. However, due to the influence of complex marine environment, imaging conditions and measurement methods, the precision and efficiency are relatively low, which is difficult to meet the demand, and has become a hot and difficult point of current research. Side-scan sonar can detect a large range of seabed area in a short time, and due to its acoustic detection means, it is not affected by underwater visibility and other factors and has a relatively low cost, so it is widely used in seabed target detection and plays an important role in searching for crashed aircrafts, ships and missing personnel, and positioning seabed pipelines. However, the current recognition of seabed targets in side-scan sonar images is mainly artificial, which relies too much on subjective experience and is low in efficiency, seriously affecting its wide application in seabed target detection, and it is difficult to be carried on the unmanned equipment such as AUV with limited computing power and high requirements for detection efficiency and real-time performance. Therefore, how to improve the detection precision and efficiency is an important work in side-scan sonar measurement.

[0003] With the rapid development of computer vision field and the emergence of convolutional neural network, the method based on deep learning has been widely applied in the field of seabed target detection in side-scan sonar image. The existing target detection model is generally divided into two-stage detection model and one-stage detection model. However, because the two-stage model needs to generate the region proposal network of candidate target frame, and then classify and regress the boundary frame of these candidate frames, the target detection is realized through two processes, which has the problems of complex structure, large amount of calculation, long calculation time, etc., so it is rarely used in practical engineering application. The core advantage of single-stage target detection model is first reflected in speed, through simplifying the detection process and reducing the consumption of computing resources, realizing efficient real-time detection, especially suitable for application scenarios with high real-time requirements. Secondly, its structure is relatively simple, easy to realize and deploy, at the same time, reduces the hyperparameters that need to be adjusted, and reduces the complexity of model optimization. In addition, the single-stage model also shows strong adaptability and flexibility, which can be easily adjusted to adapt to different task requirements, and is easy to integrate with other image processing systems or platforms.

[0004] Although existing research has improved the detection efficiency and accuracy of the target detection model to some extent, it is still difficult to meet the actual needs in engineering deployment. Taking AUV equipped with side scan sonar as an example, in order to meet the requirements of detection efficiency, mapping quality and endurance of the carrier, etc., the side scan sonar is usually used to measure at an economic speed of 2-3 knots when detecting the seabed, which requires the target detection model to have high precision and improve the detection efficiency to meet the real-time demand of detection. And due to the limitations of acoustic detection principle, working mode and other factors, compared with remote sensing target detection, it is difficult to obtain different modal data of the same area, that is, the non-repeatability of data acquisition and the singleness of source, which leads to the inability to fuse multi-modal information during feature extraction and learning, restricting the improvement of the accuracy of the target detection model. Moreover, the image measured by the side scan sonar is usually large in size and unbalanced in length and width, and if the whole image is input into the network, the image will be compressed and it will be difficult to detect small targets in the image, but small targets on the seabed often have important significance for emergency search and rescue. SUMMARY

[0005] The present application mainly aims at the side scan sonar image target detection model, which has large parameters and is difficult to deploy on small AUVs and other mobile platforms, and the detection accuracy and detection strategy are difficult to meet the engineering deployment requirements. A seabed target detection model named DBnet is proposed. First, in order to solve the problem of large model parameters and low detection efficiency, we use a lightweight network backbone and adopt a simplified feature fusion structure, which greatly reduces the model parameters and computational complexity. Secondly, in order to alleviate the problem of accuracy decline caused by model lightweight and the serious limitation of model performance caused by single source of detection data, we design a double-branch structure, that is, we use double backbone networks for feature extraction to effectively obtain more rich semantic information. Finally, due to the large size of the original waterfall image of the side scan sonar, the input network will be reshaped, which will make it difficult to detect some small-scale targets. We introduce a sliding window slicing to segment the original image and input it into the network. The window size is set according to the detection efficiency of the model, and all the slices and the global detection results are fused to effectively improve the detection ability of small-scale targets. The proposed algorithm can be deployed on AUVs and other underwater mobile platforms to realize real-time intelligent detection of seabed targets.

[0006] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0007] A seabed target detection method based on a double-branch lightweight backbone, specifically comprising the following steps:

[0008] Step 1: data set preprocessing

[0009] Collect and label the side-scan sonar image data. For classes with a small number of samples in the data set, use data augmentation methods to generate more sample images. Finally, divide the data set into training set, validation set and test set according to the proportion.

[0010] Second step, network model construction and training

[0011] The network model includes an input layer, a Backbone, a Neck, and an output layer. The training set and the validation set are used to train and verify the network model, and a trained network model is obtained.

[0012] (2.1) Input layer

[0013] The input layer is the first layer of the network model that receives image data, responsible for scaling the original image to a fixed size required by the network and inputting it into the network, and then performing pixel value normalization processing to provide original data for subsequent feature extraction and processing.

[0014] Before the image data is input, the SAHI algorithm (Slicing Aided Hyper Inference) is used to cut the side-scan sonar original data waterfall chart into multiple slices, and independent target detection is performed on these slices.

[0015] (2.2) Backbone

[0016] The Backbone extracts image features through a series of convolution layers and activation functions, gradually reducing the spatial dimension of the image while increasing the number of channels. A double-branch lightweight backbone network is constructed using PP-LCNet and GhostNet, which parallelly extracts high-level features of the input image through multiple layers of convolutional neural network layers.

[0017] Further, the GhostNet is used as the first feature extraction backbone network, which is constructed based on the Ghost module and consists of 5 layers of GhostConv, 4 layers of C3Ghost, and 1 SPPF module.

[0018] Further, the PP-LCNet is used as the second feature extraction backbone network, which is based on the DepthSepConv feature extraction backbone and consists of 1 layer of standard convolution, 6 layers of 3x3 DepthSepConv, and 3 layers of 5x5 DepthSepConv.

[0019] (2.3) Neck

[0020] The Neck part is located between the Backbone and the output layer, fuses the high-level features of the double-branch lightweight backbone network to the image, and fuses the high-resolution but less information feature map output by the same backbone network with the low-resolution but information-rich feature map, fuses feature maps of different scales through the Neck structure, and enhances the detection ability of the network model to targets of different sizes.

[0021] Further, the 5th layer, the 7th layer and the 10th layer in each double-branch lightweight backbone network are extracted as inputs of the Neck part. In the Neck part, only the feature maps extracted from the layers need to be fused.

[0022] (2.4) Output layer

[0023] The output layer is responsible for generating the final target detection result, and predicts the target position, category and confidence in the image according to the fused feature map.

[0024] Step 3, model evaluation

[0025] By putting the test set into the trained network model, detection is carried out to obtain the target information in the side scan sonar image, and the model performance is evaluated through indicators such as precision P (precision), recall R (recall) and mean average precision (mean average precision).

[0026] The beneficial effects of the present application are: in view of the problems of low detection efficiency and large parameter quantity of the existing target detection model, which is difficult to deploy to AUV and other small unmanned platforms, a double-branch lightweight backbone model DBnet is designed, two lightweight backbone networks with superior performance, PP-LCNet and GhostNet, are used in the feature extraction stage to extract the features of the input image, and through the simplified Neck structure, the features extracted by the double backbone are well fused while reducing the parameter quantity, and the target detection ability of different scales is enhanced. In the inference stage, due to the large size of the original side scan sonar image, direct input into the network will compress the real size of the target image after resampling, therefore, the SAHI method is adopted, according to the actual underwater operation of AUV, the whole waterfall image is cut into several fixed-size slices with a certain overlap rate and then input into the network, effectively solving the problems of difficult deployment of target detection model for AUV and other unmanned platforms, low detection efficiency and low detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 DBnet network structure diagram;

[0028] Figure 2 SAHI principle diagram;

[0029] Figure 3 Figure 3 is a contrastive diagram of mAP@0.5 curves of DBnet and YOLOv8;

[0030] Figure 4 Figure 4 is a prediction effect diagram of the model, (a) is a prediction effect diagram before improvement, and (b) is a prediction effect diagram after improvement. DETAILED DESCRIPTION

[0031] In order to make the method problems solved by the present application, the method schemes adopted and the method effects achieved more clear, the present application will be further described in detail below in combination with the drawings and experiments. It can be understood that the specific experiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the present application are shown in the drawings, not all.

[0032] The model of the present application is built based on the YOLOv8 algorithm framework, and the DBnet network structure is as shown in Figure 1 In the present application, we first use PP-LCNet and GhostNet as the double backbone network of the model, which enables the model to fuse the feature information extracted from different backbone networks in the case of only one modal data. Different backbone networks can extract complementary feature information, and by fusing these features, more comprehensive target information can be captured, thereby improving the detection accuracy of the model. Single source data is easily affected by noise, occlusion, etc., and the fusion of such similar multi-modal data can enhance the robustness of the model through the redundant information between the features extracted from different backbone networks. In addition, by extracting features through double backbone networks, the model can learn more general feature representations, thereby performing better in different background environments. Specifically, we use GhostNet as the first feature extraction backbone, which is built based on Ghost module and consists of 5 layers of GhostConv, 4 layers of C3Ghost and SPPF module. Then PP-LCNet is used as the second feature extraction backbone, which is a feature extraction backbone based on DepthSepConv and consists of six consecutive layers of 3x3 DepthSepConv and three layers of 5x5 DepthSepConv.

[0033] In the Neck part, multiple up-sampling steps of the original structure tend to bring noise amplification and other adverse effects. Therefore, we avoid using up-sampling to combine small-scale high-level convolutional features and large-scale low-level convolutional features in the standard YOLOv8 model. Instead, we utilize the sufficiently rich feature maps in the dual stem feature extraction for fusion, thereby optimizing the Neck part. This method reduces the computational load of the model, simplifies the complexity of the model, and helps to lightweight the model. After feature extraction, we extract the 5th, 7th, and 10th layers in each stem network, respectively, as the input of the neck part. In the Neck part, we only need to fuse the feature maps extracted from each layer. It is worth mentioning that we not only simplify the structure, but also use C3Ghost instead of C2f module in the original YOLOv8 network, further reducing the parameters and computational load of the model.

[0034] The output layer functions to complete the output of the target detection result.

[0035] We collected two SSS datasets, SSUTD (Side-Scan Sonar Undersea Target Dataset) and Sonar Common Target Detection Dataset (SCTD), for the training and testing of the detector. The SSUTD dataset is mainly obtained by using mainstream side-scan sonar instruments and equipment by various hydrographic survey departments and companies, with a small amount of data collected from the network, totaling 1584 images. The SCTD dataset mainly contains side-scan sonar images of aircraft wreckage, sunken ships, etc.

[0036] Step 1: Data set preprocessing

[0037] Collect and label side-scan sonar image data. Since the target detection model has data-driven characteristics, the performance of deep learning algorithms is easily affected by the quantity and distribution of the training data set. For underwater target recognition model training, there are two problems: insufficient image quantity and unbalanced quantity in different targets will limit the performance of deep learning algorithms. In order to enrich the sample quantity in the data set and improve the model generalization, data augmentation technology is introduced to generate more SSS images for the relatively small number of drowning targets. Common data augmentation algorithms include Gaussian noise, brightness change, image translation, rotation, flipping, scaling, and shearing.

[0038] The dataset SSUTD is composed of 980 shipwreck images, 36 drowning person images and 568 airplane wreckage images, the dataset SCTD is composed of 266 shipwreck images, 34 drowning person images and 57 airplane wreckage images, and the dataset A is based on the dataset SSUTD and each drowning person image is data-augmented. According to the ratio of 8:1:1, the dataset A is divided into three subsets, i.e., a training set, a validation set and a test set. The detailed division is shown in Table 1.

[0039] Table 1: Dataset division

[0040]

[0041] Second step, model training

[0042] The training process of the model is performed according to the technical solution

[0043] The specific principle of the prediction inference is shown in Figure 2 First, the original image I (blue box) is cut into x MxN slices (red box) with a certain degree of overlap, denoted as P1, P2...P X Then, the size of each slice is adjusted while maintaining the aspect ratio. After that, the content contained in each slice is predicted. As the size of the slice decreases, the detection performance of the model for larger targets will decrease. Therefore, in order to more accurately detect larger targets, the full-inference (FI) result of the original image is used, and the NMS (Non-Maximum Suppression) algorithm combines the prediction results of the slice and the FI result back to the original size. In the NMS process, the boxes with an Intersection over Union (IoU) higher than a pre-set matching threshold Tm are matched, and the detection with a detection probability lower than Td is deleted for each matching. This parallel processing method can significantly improve the detection efficiency, especially in processing side-scan sonar images or in high computational resource requirement scenarios. Moreover, by performing inference on smaller slices, the SAHI algorithm can reduce the consumption of memory and computational resources. This is particularly important for devices with limited computational resources or real-time detection systems.

[0044] Third step, model evaluation

[0045] In order to comprehensively and objectively evaluate the prediction effect of different models, the present application evaluates the performance of the optimized model by the following coefficients: average precision AP (average precision), mean AP (mean AP).

[0046] Precision and recall: In the classification task of predicting whether an image contains a package, the four elements in the explainable precision and recall are as follows: TP (true positive): the prediction result of the positive sample is correctly marked as a positive sample; TN (true negative): the prediction result of the negative sample is correctly marked as a negative sample; FP (false positive): the prediction result of the positive sample is incorrectly marked as a negative sample; FN (false negative): the prediction result of the negative sample is incorrectly marked as a positive sample, and the calculation relationship is:

[0047]

[0048] Average precision: The geometric meaning of average precision AP is the area corresponding to the P-R curve as shown in equation (3), wherein the interpolation summation method can be used to approximate the integral.

[0049]

[0050] Average precision mean: The mAP type used by the present application is: mAP@0.5 and mAP@0.5:0.95. mAP@0.5 (average precision coefficient of total category) represents the precision and average of all images of each category when the IoU is set to 0.5; mAP@0.5:0.95 represents that the IoU threshold of mAP is in the range of 0.5-0.95, and the calculation method is as follows:

[0051]

[0052] Wherein, N represents the number of target categories;

[0053] In addition, the following parameters can reflect the complexity and lightweight effect of the model, and FPS is an important indicator for measuring the speed of an algorithm. It is the ratio of frame number to time consumed.

[0054] Params and FLOPs represent the calculation space complexity and calculation complexity of the model, respectively.

[0055] FLOPs (floating point of operations) is the number of floating point operations. For a convolution layer, the calculation formula of FLOPs is as follows:

[0056] FLOPs = 2 x H x W x (C in x K 2 + 1) x C out (5)

[0057] Wherein, H and W represent the height and width of the convolution layer output, respectively, and Cin C represents the number of input channels, K represents the number of output channels, and N represents the size of the convolution kernel. out C represents the number of input channels, K represents the number of output channels, and N represents the size of the convolution kernel.

[0058] To verify the influence of the selection of different backbone networks on the performance of the model and the generalization of the model on different data sets, we conducted multiple sets of ablation experiments. First, the selection of the backbone network, the double backbone detector model proposed by the model, makes up for the single detection means for submarine obstacles, and the lack of multi-source and multi-modal data fusion. At the same time, in order to meet the lightweight implementation of the model, we selected the current mainstream lightweight backbone network for combination experiment, and the experimental results are shown in Table 2.

[0059] Table 2 Experimental results of different data sets. Bold font is the method proposed in the application.

[0060] Table 2 Prediction evaluation results of different models.

[0061]

[0062] From the table data, it can be seen that DBnet has obvious improvement in mAP value compared with most single backbone networks. Although the parameter amount and GFLOPs of the method proposed in the application are not the smallest in the double backbone network, the comprehensive detection performance index mAP is good, and the lightweight degree of the model is also better than other networks, which can meet the real-time target detection and engineering deployment requirements.

[0063] The analysis of the experimental results mainly includes two aspects: one is the analysis of the evaluation index of the original model and the improved model, and the other is the analysis of the detection effect of the improved model.

[0064] Figure 3 is the mAP@0.5 curve comparison diagram of the improved model and the baseline model. It can be seen that the improved curve is higher than the baseline model. Not only the accuracy is improved, but also the convergence speed is faster and more stable. From the statistical data, it can be found that the improved model has good prediction performance

[0065] Figure 4 shows the prediction results of the YOLOv8 model before and after improvement. When the improved model performs target detection, the model can accurately identify the sunken ship in the image, and avoid the misdetection of the terrain undulations in the background as a sunken ship in (a). It can be seen that using the application can effectively improve the accuracy of target detection, especially for large-size images such as the waterfall chart of the side-scan sonar original data, avoiding the situation of small target missing detection and large target misdetection caused by image compression when inputting the network.

[0066] Finally, it should be noted that the above experiments are only used to illustrate the method of the present application, and are not intended to limit the present application; although the present application has been described in detail, those skilled in the art should understand that the modification of the above-mentioned method, or the equivalent replacement of part or all of the method features, does not make the essence of the corresponding method deviate from the scope of the method of the present application.

Claims

1. A method for seabed target detection based on a dual-branch lightweight backbone, characterized in that, Specifically comprising the following steps: First, data set preprocessing Collect and label side scan sonar image data; For classes with a small number of samples in the data set, use data augmentation methods to generate more sample images; Finally, divide the data set into training set, validation set and test set according to the proportion; Second, network model construction and training The network model includes an input layer, a Backbone, a Neck, and an output layer; The training set and the validation set are used to train and verify the network model, and a trained network model is obtained; (2.1) Input layer The input layer is the first layer of the network model receiving image data, responsible for scaling the original image to the fixed size required by the network and inputting it into the network, and then performing pixel value normalization processing to provide original data for subsequent feature extraction and processing; Before the image data is input, the SAHI algorithm is used to cut the side scan sonar waterfall chart into multiple slices, and the target detection is independently performed on these slices; (2.2) Backbone The Backbone extracts image features through a series of convolution layers and activation functions, gradually reduces the spatial dimension of the image, and increases the number of channels; A double-branch lightweight backbone network is formed by using PP-LCNet and GhostNet, which extracts high-level features of the image from the input image through multiple layers of convolutional neural network layers in parallel; (2.3) Neck The Neck part is located between the Backbone and the output layer, which fuses the high-level features extracted by the double-branch lightweight backbone network, and fuses the high-resolution but less information feature map and the low-resolution but rich information feature map output by the same backbone network. Through the Neck structure, the feature maps of different scales are fused to enhance the detection ability of the network model for targets of different sizes; (2.4) Output layer The output layer is responsible for generating the final target detection result, which predicts the target position, class and confidence in the image according to the fused feature map; Third, model evaluation By putting the test set into the trained network model, the target information in the side scan sonar image is obtained, and the model performance is evaluated through the precision P, recall R and average precision mean indicators.

2. The method of claim 1, wherein the method comprises: The GhostNet is used as the first feature extraction backbone, and the GhostNet is constructed based on the Ghost module, which is composed of 5 layers of GhostConv, 4 layers of C3Ghost and 1 SPPF module.

3. The method of claim 1, wherein the method comprises: The PP-LCNet is used as the second feature extraction backbone, and the PP-LCNet is a feature extraction backbone based on depth separable convolution, which is composed of 1 layer of standard convolution, 6 layers of 3x3 depth separable convolution and 3 layers of 5x5 depth separable convolution.

4. The method of claim 1, wherein the method is characterized by, The 5th layer, the 7th layer and the 10th layer in each double-branch lightweight backbone network are extracted as the input of the Neck part; In the Neck part, only the fused feature maps extracted from each layer are needed.

Images