Marine target detection method based on multi-scale lightweight deep network

Abstract

The invention discloses a marine target detection method based on a multi-scale lightweight deep network. The method comprises the following steps: (1) constructing a data set image; (2) dividing data; step (3), inputting the training image into a feature extraction module, and coding the input image; step (4), selecting the feature map after position coding through a weight selection function to obtain FSelect; step (5), the FSelect is sent to a subsequent LSKBlock structure layer, and a feature map FLSKBlock after feature extraction of different scales is obtained; step (6), inputting the FLSKBlock into a feature fusion module; the FAIFI is output; step (7), carrying out further fusion processing on the FAIFI through a CCFM module; step (8), splicing the fused feature maps to obtain a final feature map; and (9) decoding and outputting the obtained feature map to generate a bounding box. According to the invention, through combination of multi-scale feature extraction and lightweight network design, efficient and accurate sea target detection is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detecting marine targets based on a multi-scale lightweight deep network. Background Art

[0002] In the marine environment, target detection is of great significance for applications such as marine navigation safety, marine resource exploration, and marine search and rescue. However, due to the complexity of the marine environment, such as weather changes, sea surface reflections, and light changes, marine target detection faces many challenges. Traditional target detection methods often fail to achieve ideal results in the face of complex marine environments.

[0003] In recent years, with the rapid development of computer vision technology, deep learning-based object detection methods have made significant progress in various application scenarios. Deep learning methods learn features from a large amount of labeled data by training convolutional neural networks (CNNs), achieving step-by-step extraction from low-level to high-level features, thereby showing excellent performance in object detection tasks. However, deep learning models usually have high computational complexity and storage requirements, which may become a bottleneck in resource-constrained marine applications.

[0004] Researchers have made some progress in marine target detection, but existing technologies still have certain limitations. Researchers have proposed some methods based on multi-scale feature extraction and lightweight network design. Although they can fuse feature information of different scales, due to information loss or noise amplification in the feature fusion process, especially when facing distant or low-contrast targets, the detection accuracy may decrease.

[0005] For marine target detection, researchers have proposed some methods based on multi-scale feature extraction and lightweight network design. Multi-scale feature extraction methods can effectively handle targets of different sizes and improve the accuracy and robustness of detection. Lightweight network design improves the operating efficiency of the model by reducing model parameters and calculation amount, and adapts to the application requirements of limited computing resources in marine environments.

[0006] However, due to the complexity and variability of the marine environment, such as different weather conditions, sea surface reflections, illumination changes, etc., existing multi-scale and lightweight detection methods are insufficient in generalization ability. Especially under different sea conditions and different target characteristics, the detection effect of the model fluctuates greatly, lacks sufficient robustness, and cannot meet the stability requirements of real-time detection and long-term operation.

[0007] At present, common multi-scale feature extraction methods include pyramid structure, feature pyramid network (FPN), etc. These methods can fuse feature information of different scales while maintaining high-resolution features to improve detection effect. On the other hand, lightweight network design mainly uses deep separable convolution, pruning technology, quantization technology and other means to significantly reduce the number of model parameters and calculations while maintaining model performance.

[0008] Although existing research has made some progress in the field of marine target detection, facing the complexity and real-time requirements of the marine environment, there is still a need to further optimize and improve the detection algorithm. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a marine target detection method based on a multi-scale lightweight deep network, which realizes efficient and accurate marine target detection by combining the extraction methods of features at different scales and a lighter network model design.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A method for detecting marine targets based on a multi-scale lightweight deep network comprises the following steps:

[0012] Step (1): annotate the acquired marine target images and construct an image dataset;

[0013] Step (2): Divide the data into two parts: training set D and training set D. train With the test set D test ;

[0014] For example, the training set D train The test set D accounts for 70% of the total data. test 30% of all data;

[0015] Step (3): By inputting the training image into the feature extraction module, the input image is positionally encoded to obtain F Patchembed ;

[0016] Step (4): Select the position encoding feature map F through the weight selection function Patchembed , and get F Select ;

[0017] Step (5): F Select Send it to the LSKBlock structure layer and combine it with the position encoding feature map F Patchembed The feature map F after feature extraction of different scales is obtained by using the feature selection function LSKBlock ;

[0018] Step (6): F LSKBlock Input into the feature fusion module and output F AIFI ;

[0019] Step (7): F AIFI Through the CCFM module, the fused feature map F is obtained. CCFM ;

[0020] Step (8): F CCFM Splice to get the final feature map;

[0021] Step (9): Decode the obtained feature map and generate a bounding box.

[0022] In step (1), the constructed data set is used in the training process of the deep learning network model. i Represented as the i-th marine target image, the input image size is [W, H], the channel is C, and the input image size is adjusted to 640×640.

[0023] In the step (3), a position encoding operation is first performed to divide the input image into blocks of fixed size, and the position information of each block in the original image is supplemented to encode the input image I i Position encoding can be expressed as: F Patchembed =f patchembed (I i ), where f patchembed () represents the position encoding operation, which divides the image into blocks, generates position information, and combines the global position and local position in the original image.

[0024] In the step (4), the weight selection function selects the feature map after position encoding as follows:

[0025] F Select =

[0026] Sigmoid(mean(Conv1f patchembed (I i )),avg(Conv2f patchembed (I i ))), where the weight selection function is composed of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolution Conv1 and Conv2;

[0027] The convolution operation Conv extracts features at different levels by performing sliding window processing on the input feature map through the convolution kernel. The activation function Sigmoid maps the results of convolution and pooling to probability values ​​of weight selection, that is, each pixel in the output feature will be assigned a weight value for subsequent feature map fusion or selection. The average pooling operation mean and the maximum pooling operation avg divide the feature map into windows of fixed size and then take the average or maximum value in each window.

[0028] In the step (5), feature extraction is performed through the LSKBlock structure layer, F LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

[0029] The inputs of step (4) and step (3) are input into step (5). The F of step (4) Select is a selection weight. Here f patchembed (I i ) The same method as in step (3) is used. patchembed (I i );by changing F Patchembed Combined with the selection weight obtained in step (4), the feature selection process is completed.

[0030] The step (6) is specifically as follows:

[0031] First, it passes through the AIFI module, which consists of shape change and position encoding. First, the result F of the previous step is input. LSKBlock Position encoding is performed to supplement the position information in the original image. The encoded features undergo shape changes. The shape changes are readjusted by dimension expansion, transposition and other operations on the input tensor, so that the features at different stages can maintain the same shape. The overall AIFI module operation can be expressed as: F AIFI =TransShape(PositonEmbedding(F LSKBlock )), where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

[0032] The step (7) is specifically:

[0033] First, the convolution operation is used to further extract and filter the input feature map. Then, the residual module composed of parallel convolution blocks is used to alleviate the gradient vanishing problem in the deep network. It can be expressed as: F CCFM =RepBlock(Conv(F AIFI)), where RepBlock is the parallel combined residual module.

[0034] In the step (8):

[0035] X feats , X shappes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and feature map shape, where GetEncoder includes flattening and permutation operations, and then concatenates the features to get a large feature map;

[0036] In the step (9):

[0037] The decoded output generates a bounding box which can be represented as: Head i =Decoder(X feats , X shapes ),i<500;

[0038] It can be expressed as Decoder = Socres(Bboxs(X feats , X shapes )), and i is the i-th generated anchor point and valid mask, and the loss is calculated for the generated bounding box and label image, and the loss uses CIOU_Loss as the loss function.

[0039] The loss is calculated as follows:

[0040] p=f d (Head, label), v = min(f s (Head,label)),, CIOU=1-CIoU, where p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the directional differences between the bounding boxes, and α is the influence of v.

[0041] Beneficial effects of the present invention:

[0042] (1) The present invention proposes a lightweight feature extraction strategy and adopts a smaller number of large convolution kernels, thereby significantly reducing the model parameter scale and computational overhead, and realizing efficient marine target detection. The lightweight design strategy of the present invention enables the algorithm to still run efficiently in a marine environment with limited computing resources.

[0043] (2) In view of the small size of marine targets, the present invention adopts the idea of ​​parallel divide and conquer in the feature extraction process, and captures multi-scale feature maps by using feature extraction modules of different sizes in parallel, and performs the feature fusion operation proposed by the present invention on the feature maps, thereby significantly improving the detection accuracy. The multi-scale feature extraction method can capture target features of different sizes and distances, thereby achieving higher detection accuracy and robustness when detecting various marine targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the network model framework of the present invention based on a multi-scale lightweight deep network.

[0045] Figure 2 This is a structural diagram of the large convolution kernel backbone network module of the feature extraction part of the present invention.

[0046] Figure 3 This is a comparison chart of indicators between the present invention and the existing algorithms after testing on the marine target data set.

[0047] Figure 4 This is a comparison chart of some visualization results of the present invention and existing algorithms. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0049] Reference Figure 1 and Figure 2 , the marine target detection method based on multi-scale lightweight deep network mainly includes three stages. Feature extraction stage, feature fusion stage, and output detection stage; in the feature extraction stage, a lightweight deep network is adopted to realize multi-scale feature extraction through large convolution kernels and parallel feature extraction modules, effectively capturing the multi-scale features of marine targets while reducing computational complexity. In the feature fusion stage, feature information of different scales is integrated through an adaptive feature fusion mechanism, which enhances the detection capability of tiny targets and maintains high accuracy under complex backgrounds. In the output detection stage, the advantages of lightweight and multi-scale processing are combined to ensure efficient output of high-precision detection results under limited computing resources, significantly reduce processing delays, and adapt to actual marine application needs.

[0050] 1. In the feature extraction stage, multi-scale deep features are extracted from the input marine images to effectively capture target objects of different sizes in the image as well as local and global information of the scene, making the model more robust to target objects in different marine environments.

[0051] Step 1: First, a large number of marine target images are obtained through drones and other equipment, the image targets are annotated, and input into the network model for training. The image before the model input is adjusted to make the image size 640×640, the input size is [W,H], and the channel is C;

[0052] Step 2: By transforming the training image I i The input feature extraction module first performs Patternbed operation on the input image I i Position encoding can be expressed as: F Patchembed =f patchembed (I i ).

[0053] Step 3: Construct a feature map selection method to select the feature map after position encoding by designing a weight selection function, where the weight selection can be expressed as F Select =Sigmoid(mean(Conv1f patchembed (I i )),avg(Conv2f patchembed (I i ))), where the weight selection function is composed of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolutions Conv1 and Conv2.

[0054] Step 4: embed the feature map selection method into the LSKBlock structure, and perform feature extraction through the LSKBlock structure layer. LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

[0055] 2. In the feature fusion stage, the multi-scale features obtained in the feature extraction stage are integrated and enhanced to further refine the features, thereby enhancing the feature expression ability of the model and facilitating the recognition of the target object.

[0056] Step 1: Perform feature fusion operation on the feature maps of different scales output by the LSKBlock structure layer. First, the input result F LSKBlock Input to the subsequent AIFI feature fusion module, F AIFI =TransShape(PositonEmbedding(F LSKBlock )), the AIFI module consists of shape change and position encoding, where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

[0057] Step 2: The feature fusion results of the intermediate state are further fused through the CCFM module, which can be expressed as: CCFM =RepBlock(Conv(F AIFI )), where RepBlock is the parallel combined residual convolution.

[0058] Step 3: Concatenate the feature maps fused by the AIFI and CCFM modules to obtain the final feature map, which can be expressed as: feats , X shapes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and feature map shape, where GetEncoder includes flattening and permutation operations, and then concatenates the features to obtain a large feature map.

[0059] 3. In the output detection stage, the final target positioning and classification are performed based on the fused feature map, and detection results such as the target’s bounding box, category, and confidence score are generated.

[0060] Step 1: Decode the feature map to generate a bounding box, which can be expressed as: Head i =Decoder(C feats , X shapes ), i<500, which can be expressed as Decoder = Socres (Bboxs (X feats , X shapes )), and i is the i-th generated anchor point and valid mask.

[0061] Step 2: Calculate the loss of the generated bounding box and the label image. The loss uses CIOU_Loss as the loss function. The loss calculation method is as follows: p = f d (Head i ,label),,v=min(f s (Head i ,label)),, CIOU=1-CIoU,,p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the directional differences between the bounding boxes, and α is the influence of v.

[0062] like Figure 1As shown: the overall framework of the invention is mainly divided into three parts, the feature extraction part, the feature fusion part and the output detection part. The multi-scale feature extraction and lightweight design strategy are completed through parallel LSKBlock, and the AIFI and CCFM modules are used to effectively fuse the image features while ensuring the detection accuracy. The output detection module is used to locate the target and category.

[0063] like Figure 2 As shown in the figure, the deep features of the image are extracted at different scales through multi-scale large convolution kernels. The module captures target objects of various sizes in the image at different levels. At the same time, the lightweight design reduces the computational overhead of the model.

[0064] like Figure 3 As shown: The present invention is tested and compared with several existing algorithms on the marine target detection dataset. The figure shows the performance of different algorithms in terms of accuracy, recall, mAP50 and mAP50-95.

[0065] like Figure 4 As shown: a comparison of some visualization results of the present invention and several existing algorithms in the marine target detection task. The figure lists the comparison pictures of the detection results of different algorithms, which intuitively shows that the present invention has fewer missed detections and false detections under complex marine backgrounds, and can accurately identify and locate target objects.

[0066] The effectiveness of the present invention is verified by simulation experiments as follows.

[0067] Simulation content:

[0068] (1) On the same training set and test images, six representative marine target detection methods are selected in the form of comparative experiments to compare with the simulation results of the present invention to verify the effectiveness of the present invention. Six representative detection methods are the method proposed by Wang et al., using YOLOv7 to achieve the task of marine target detection. For specific reference, please refer to "Wang CY, Bochkovskiy A, Liao HY M.YOLOv7:Trainable bag-of-freebies sets newstate-of-the-art for real-time object detectors[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2023:7464-7475.", the method proposed by Wang et al., using YOLOv9 to achieve the task of marine target detection. For specific reference, please refer to "Wang CY, YehI H, Liao HY M.Yolov9:Learning what you want to learn using programmable gradient information[J].arXiv preprint arXiv:2402.13616,2024.", the method proposed by Cheng et al., referred to as ODConvNeXt, for specific reference, please refer to "Cheng S, Zhu Y, Wu S.Deep learning based efficient ship detection from drone-captured images for maritime surveillance[J].Ocean Engineering, 2023, 285:115440. ", the method proposed by Zhao et al., referred to as RT-DETR, specific reference "Zhao Y, Lv W, Xu S, et al. Detrs beat yolos on real-time object detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024:16965-16974.", and the use of YOLOv5 model and YOLOv8 model to implement PCB defect detection method.

[0069] (2) In order to further compare the target detection performance of the models, the test index comparison results of the four models under the same dataset were calculated.

[0070] The specific simulation conditions are detailed in the description of each experiment.

[0071] Experiment 1: The training and testing of the present invention and the other three methods are compared on the data set. The comparison results are as follows: Figure 4 As shown: Figure 4 The method listed in this application is the result of the test by this method. Figure 4 The YOLOv5 column is the detection result of the YOLOv5 model method. Figure 4 The YOLOv9 column is the detection result of the YOLOv9 model method. Figure 4 The RT-DETR column is the test result of the RT-DETR model method. Figure 4 It can be seen from the simulation results that when comparing the results detected by different methods, most methods can produce relatively satisfactory results. In comparison, the present invention has higher detection accuracy when detecting and identifying targets at sea. Referring to the first row, when the background is dark, the method of this paper has no missed detection, while other methods have different degrees of missed detection. Referring to the second and third rows of images, the detection of the present invention is more comprehensive, and there is no misidentification and missed detection of targets of different categories, but other algorithms have different degrees of misdetection or missed detection. Referring to the fourth row of images, the present algorithm still maintains a good detection effect for small target detection when the sea wave interference is large.

[0072] Experiment 2: For marine target detection, the accuracy of target detection is the key to evaluating the model. We compared different detection accuracy indicators of different methods, such as Figure 3 As shown, there are four indicators, including accuracy, recall, mAP50 and mAP50-95.

[0073] from Figure 3 and Figure 4 It can be seen from the simulation results that the present invention is significantly superior to other methods in all aspects: 1) it shows satisfactory performance in terms of accuracy in detecting smaller targets at sea, and 2) there are fewer cases of missed defects.

[0074] The results of the above two experiments show that compared with the existing marine target detection method, the present invention has higher detection accuracy and can obtain richer image features from the image, so that it can better detect small targets at sea with fewer false detections and missed detections.

[0075] The present invention utilizes multi-scale feature fusion and lightweight convolution operations to reduce the computational complexity of the model while ensuring detection accuracy, thereby improving the real-time and applicability of the model. This method is not only suitable for marine target detection, but can also be extended to target detection tasks in other complex environments.

[0076] This paper focuses on the marine target detection method based on multi-scale lightweight deep network. Through the convolution of different sizes, the multi-scale information of the image is extracted, and the multi-scale features of the image are processed through the fusion module to obtain more comprehensive feature information of the detected target, thereby effectively enhancing the network's ability to represent key features, improving the recognition detection rate of small-sized targets in marine targets, and realizing a detection method with high detection accuracy.

Claims

1. A method for detecting marine targets based on a multi-scale lightweight deep network, characterized in that: The steps include: Step (1): annotate the acquired marine target images and construct an image dataset; Step (2): Divide the data into two parts: training set D and training set D. train With the test set D test ; Step (3): By inputting the training image into the feature extraction module, the input image is positionally encoded to obtain F Patchembed ; Step (4): Select the position encoding feature map F through the weight selection function Patchembed , and get F Select ; Step (5): F Select Send it to the LSKBlock structure layer, combine the position encoding feature map with F Patchembed The feature selection function obtains the feature map F after extracting features of different scales LSKBlock ; Step (6): F LSKBlock Input into the feature fusion module and output F AIFI ; Step (7): F AIFI Through the CCFM module, the fused feature map F is obtained. CCFM ; Step (8): F CCFM Splice to get the final feature map; Step (9): Decode the obtained feature map and generate a bounding box.

2. According to claim 1, a method for detecting marine targets based on a multi-scale lightweight deep network is characterized in that: In the step (1), the constructed dataset image is used in the training process of the deep learning network model. i Represented as the i-th marine target image, the input image size is [W, H], the channel is C, and the input image size is adjusted to 640×640.

3. According to claim 1, a method for detecting marine targets based on a multi-scale lightweight deep network is characterized in that: In the step (3), a position encoding operation is first performed to divide the input image into blocks of fixed size, and the position information of each block in the original image is supplemented to encode the input image I i Position encoding can be expressed as: F Patchembed =f patchembed (I i ), where f patchembed () represents the position encoding operation, which divides the image into blocks, generates position information, and combines the global position and local position in the original image.

4. According to claim 3, a method for detecting marine targets based on a multi-scale lightweight deep network is characterized in that: In step (4), the weight selection function is used to select the feature map after position encoding: F Select =Sigmoid(mean(Conv1f patchembed (I i )),avg(Conv2f patchembed (I i ))), where the weight selection function is composed of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolution Conv1 and Conv2; The convolution operation Conv extracts features at different levels by performing sliding window processing on the input feature map through the convolution kernel. The activation function Sigmoid maps the results of convolution and pooling to probability values ​​of weight selection, that is, each pixel in the output feature will be assigned a weight value for subsequent feature map fusion or selection. The average pooling operation mean and the maximum pooling operation avg divide the feature map into windows of fixed size and then take the average or maximum value in each window.

5. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 4, characterized in that: In the step (5), feature extraction is performed through the LSKBlock structure layer, F LSKBlock =F Patchembed ·F Select , F LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

6. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 5, characterized in that: The step (6) is specifically as follows: First, it passes through the AIFI module, which consists of shape change and position encoding. First, the result F of the previous step is input. LSKBlock Position encoding is performed to supplement the position information in the original image. The encoded features undergo shape changes. The shape changes are readjusted by dimension expansion, transposition and other operations on the input tensor, so that the features at different stages maintain the same shape. The overall AIFI module operation is expressed as: F AIFI =TransShape(PositonEmbedding(F LSKBlock )), where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

7. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 6, characterized in that: The step (7) is specifically: First, the convolution operation is used to further extract and filter the input feature map. Then, the residual module composed of parallel convolution blocks is used to alleviate the gradient vanishing problem in the deep network. It can be expressed as: CCFM =RepBlock(Conv(F AIFI )), where RepBlock is the parallel combined residual module.

8. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, characterized in that: In the step (8): X feats , X shapes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and feature map shape, where GetEncoder includes flattening and permutation operations, and then concatenates the features to obtain a large feature map.

9. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, characterized in that: In the step (9): The decoded output generates a bounding box which can be represented as: Head i =Decoder(X feats , X shapes ),i<500; It can be expressed as Decoder = Socres(Bboxs(X feats , X shapes )), and i is the i-th generated anchor point and valid mask, and the loss is calculated for the generated bounding box and label image, and the loss uses CIOU_Loss as the loss function.

10. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 9, characterized in that: The loss is calculated as follows: p=f d (Head, label), v = min(f s (Head,label)),, CIoU = 1-CIoU, p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the directional differences between the bounding boxes, and α is the influence of v.

Images