A sea target detection method based on a multi-scale lightweight deep network

Abstract

A kind of offshore target detection method based on multi-scale lightweight deep network, comprising the following steps;Step (1): constructing data set image;Step (2): data is divided;Step (3): by inputting training image into feature extraction module, the input image is encoded;Step (4): by weight selection function, the feature map after position coding is selected, and F Select ; Step (5): F Select It is sent to subsequent LSKBlock structure layer, and the feature map after feature extraction of different scales F LSKBlock ; Step (6): F LSKBlock It is input into feature fusion module;Output F AIFI ; Step (7): F AIFI Further fusion processing is carried out by CCFM module;Step (8): the feature map after fusion is spliced to obtain the last feature map;Step (9): the feature map obtained is decoded and output to generate boundary box.The present application realizes efficient and accurate offshore target detection by combining multi-scale feature extraction and lightweight network design.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detecting marine targets based on a multi-scale lightweight deep network. Background Art

[0002] In maritime environments, object detection is crucial for applications such as navigation safety, marine resource exploration, and maritime search and rescue. However, due to the complexity of the maritime environment, such as weather changes, sea surface reflections, and varying illumination, maritime object detection faces numerous challenges. Traditional object detection methods often struggle to achieve optimal results in complex maritime environments.

[0003] With the rapid development of computer vision technology in recent years, deep learning-based object detection methods have made significant progress in various application scenarios. Deep learning methods train convolutional neural networks (CNNs) to learn features from large amounts of annotated data, enabling a step-by-step extraction of low-level to high-level features, resulting in excellent performance in object detection tasks. However, deep learning models typically have high computational complexity and storage requirements, which can become a bottleneck in resource-constrained maritime applications.

[0004] Researchers have made some progress in maritime target detection, but existing technologies still have limitations. Researchers have proposed methods based on multi-scale feature extraction and lightweight network design. While these methods can fuse feature information from different scales, information loss or noise amplification during the feature fusion process can lead to decreased detection accuracy, especially for distant or low-contrast targets.

[0005] For maritime object detection, researchers have proposed methods based on multi-scale feature extraction and lightweight network design. Multi-scale feature extraction methods can effectively handle objects of varying sizes, improving detection accuracy and robustness. Lightweight network design reduces model parameters and computational complexity, improving model efficiency and adapting to applications with limited computing resources in maritime environments.

[0006] However, due to the complexity and variability of the maritime environment, such as varying weather conditions, sea surface reflections, and illumination variations, existing multi-scale and lightweight detection methods lack generalization capabilities. In particular, under varying sea conditions and target characteristics, the model's detection performance fluctuates significantly, lacking sufficient robustness to meet the requirements of real-time detection and long-term stability.

[0007] Currently, common multi-scale feature extraction methods include pyramid structures and feature pyramid networks (FPNs). These methods can fuse feature information at different scales while maintaining high-resolution features, improving detection effectiveness. Meanwhile, lightweight network design, primarily through the use of depthwise separable convolutions, pruning techniques, and quantization, significantly reduces the number of model parameters and computational complexity while maintaining performance.

[0008] Although existing research has made some progress in the field of marine target detection, facing the complexity and real-time requirements of the marine environment, there is still a need to further optimize and improve the detection algorithm. Summary of the Invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a marine target detection method based on a multi-scale lightweight deep network, which achieves efficient and accurate marine target detection by combining the extraction methods of features at different scales and a lighter network model design.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A method for detecting marine targets based on a multi-scale lightweight deep network comprises the following steps:

[0012] Step (1): Annotate the acquired marine target images and construct an image dataset;

[0013] Step (2): Divide the data into two parts: training set D and training set D. train With the test set D test ;

[0014] For example, the training set D train Accounting for 70% of all data, the test set D test 30% of all data;

[0015] Step (3): By inputting the training image into the feature extraction module, the input image is position-encoded to obtain F Patchembed ;

[0016] Step (4): Select the position encoding feature map F through the weight selection function Patchembed , and get F Select ;

[0017] Step (5): F Select Send it to the LSKBlock structure layer and combine it with the position encoding feature map F Patchembed The feature map F after feature extraction of different scales is obtained by using the feature selection function LSKBlock ;

[0018] Step (6): F LSKBlock Input into the feature fusion module and output F AIFI ;

[0019] Step (7): F AIFI Through the CCFM module, the fused feature map F is obtained. CCFM ;

[0020] Step (8): F CCFM Splice to get the final feature map;

[0021] Step (9): Decode the obtained feature map and generate a bounding box.

[0022] In step (1), the constructed data set is used in the training process of the deep learning network model. i Represented as the i-th marine target image, the input image size is [W, H], the channel is C, and the input image size is adjusted to 640×640.

[0023] In the step (3), a position encoding operation is first performed to divide the input image into blocks of fixed size, and the position information of each block in the original image is supplemented to encode the input image I. i Position encoding can be expressed as: F Patchembed =f patchembed (I i ), where f patchembed () represents the position encoding operation, which divides the image into blocks, generates position information, and combines the global position and local position in the original image.

[0024] In step (4), the weight selection function selects the feature map after position encoding as follows:

[0025] F Select =

[0026] Sigmoid(mean(Conv1f patchembed (I i )),avg(Conv2f patchembed (I i ))), where the weight selection function is composed of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolution Conv1 and Conv2;

[0027] The convolution operation Conv extracts features at different levels by performing sliding window processing on the input feature map through the convolution kernel. The activation function Sigmoid maps the results of convolution and pooling to probability values ​​of weight selection, that is, each pixel in the output feature will be assigned a weight value for subsequent feature map fusion or selection. The average pooling operation mean and the maximum pooling operation avg divide the feature map into fixed-size windows and then take the average or maximum value in each window.

[0028] In the step (5), feature extraction is performed through the LSKBlock structure layer, F LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

[0029] The inputs of step (4) and step (3) are input into step (5), and the F of step (4) Select is a selection weight. Here f patchembed (I i ) Followed the steps (3) f patchembed (I i ); by F Patchembed Combined with the selection weight obtained in step (4), the feature selection process is completed.

[0030] The step (6) is specifically as follows:

[0031] First, it passes through the AIFI module, which consists of shape change and position encoding. First, the result F of the previous step is input. LSKBlock Position encoding is performed to supplement the position information in the original image. The encoded features undergo shape changes. The shape changes are readjusted by dimension expansion, transposition, and other operations on the input tensor, so that the features at different stages can maintain the same shape. The overall AIFI module operation can be expressed as: F AIFI =TransShape(PositonEmbedding(F LSKBlock )), where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

[0032] The step (7) is specifically as follows:

[0033] First, the convolution operation is used to further extract and filter the input feature map. Then, the residual module composed of parallel convolution blocks is used to alleviate the gradient disappearance problem in the deep network. It can be expressed as: CCFM =RepBlock(Conv(F AIFI)), where RepBlock is the parallel combined residual module.

[0034] In the step (8):

[0035] X feats , X shappes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and the shape of the feature map. GetEncoder includes flattening and permutation operations, and then concatenates the features to obtain a large feature map.

[0036] In the step (9):

[0037] The bounding box generated by the decoded output can be represented as: Head i =Decoder(X feats , X shapes ),i<500;

[0038] It can be expressed as Decoder = Socres(Bboxs(X feats , X shapes )), and i is the i-th generated anchor point and valid mask, and the loss is calculated for the generated bounding box and label image, and the loss uses CIOU_Loss as the loss function.

[0039] The loss is calculated as follows:

[0040] p=f d (Head, label), v = min(f s (Head,label)), CIOU=1-CIoU, where p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the direction difference between the bounding boxes, and α is the influence of v.

[0041] Beneficial effects of the present invention:

[0042] (1) This paper proposes a lightweight feature extraction strategy that uses fewer large convolution kernels, significantly reducing the model parameter size and computational overhead, thereby achieving efficient marine target detection. This lightweight design strategy enables the algorithm to operate efficiently in a marine environment with limited computing resources.

[0043] (2) In view of the small size of maritime targets, the present invention adopts a parallel divide-and-conquer approach in the feature extraction process. By using feature extraction modules of different sizes in parallel to capture multi-scale feature maps, the feature maps are then fused through the feature fusion operation proposed by the present invention, significantly improving detection accuracy. The multi-scale feature extraction method can capture target features of different sizes and distances, thereby achieving higher detection accuracy and robustness when detecting various maritime targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a network model framework based on a multi-scale lightweight deep network of the present invention.

[0045] Figure 2 This is the structural diagram of the large convolution kernel backbone network module of the feature extraction part of the present invention.

[0046] Figure 3 This is a comparison chart of indicators between the present invention and existing algorithms after testing on a marine target dataset.

[0047] Figure 4 This is a comparison chart of some visualization results of the present invention and existing algorithms. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below with reference to the accompanying drawings.

[0049] Reference Figure 1 and Figure 2 The maritime target detection method based on a multi-scale lightweight deep network mainly includes three stages. Feature extraction stage, feature fusion stage, and output detection stage; in the feature extraction stage, a lightweight deep network is adopted to realize multi-scale feature extraction through large convolution kernels and parallel feature extraction modules, effectively capturing the multi-scale features of maritime targets while reducing computational complexity. In the feature fusion stage, feature information of different scales is integrated through an adaptive feature fusion mechanism, which enhances the detection capability of small targets and maintains high accuracy in complex backgrounds. In the output detection stage, the advantages of lightweight and multi-scale processing are combined to ensure efficient output of high-precision detection results under limited computing resources, significantly reduce processing delays, and adapt to actual maritime application needs.

[0050] 1. Feature extraction stage: Multi-scale deep features are extracted from the input marine imagery to effectively capture target objects of different sizes in the image as well as local and global information of the scene. This makes the model more robust in detecting target objects in different marine environments.

[0051] Step 1: First, use drones and other equipment to acquire a large number of marine target images, annotate the image targets, and input them into the network model for training. The images before input are resized to 640×640, with the input size set to [W, H] and the number of channels set to C.

[0052] Step 2: By transforming the training image I i Input feature extraction module, firstly, the input image I is processed by Patternbed operation i Position encoding can be expressed as: F Patchembed =f patchembed (I i ).

[0053] Step 3: Construct a feature map selection method and select the feature map after position encoding by designing a weight selection function, where the weight selection can be expressed as F Select =Sigmoid(mean(Conv1f patchembed (I i )),avg(Conv2f patchembed (I i ))), where the weight selection function is composed of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolution Conv1 and Conv2.

[0054] Step 4: embed the feature map selection method into the LSKBlock structure, and perform feature extraction through the LSKBlock structure layer. LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

[0055] 2. In the feature fusion stage, the multi-scale features obtained in the feature extraction stage are integrated and enhanced to further refine the features, thereby enhancing the feature expression ability of the model and facilitating the recognition of the target object.

[0056] Step 1: The feature fusion operation is performed by taking the different scale feature maps output by the LSKBlock structure layer. First, the input result F LSKBlock Input to the subsequent AIFI feature fusion module, F AIFI =TransShape(PositonEmbedding(F LSKBlock ), the AIFI module consists of shape change and position encoding, where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

[0057] Step 2: The feature fusion results of the intermediate state are further fused through the CCFM module, which is expressed as: F CCFM =RepBlock(Conv(F AIFI )), where RepBlock is the parallel combination of residual convolution.

[0058] Step 3: The feature maps fused by AIFI and CCFM modules are spliced ​​to obtain the final feature map, which can be expressed as: X feats , X shapes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and feature map shape, where GetEncoder includes flattening and permutation operations, and then concatenates the features to obtain a large feature map.

[0059] 3. In the output detection stage, the final target positioning and classification are performed based on the fused feature map, and detection results such as the target's bounding box, category, and confidence score are generated.

[0060] Step 1: Decode the feature map to generate a bounding box, which can be expressed as: Head i =Decoder(C feats , X shapes ), i<500, which can be expressed as Decoder=Socres(Bboxs(X feats , X shapes )), and i is the i-th generated anchor point and valid mask.

[0061] Step 2: Calculate the loss between the generated bounding box and the label image. The loss uses CIOU_Loss as the loss function. The loss calculation method is as follows: p = f d (Head i ,label),v=min(f s (Head i ,label)), CIOU=1-CIoU, where p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the direction difference between the bounding boxes, and α is the influence of v.

[0062] like Figure 1As shown in the figure, the overall framework of the invention is mainly divided into three parts: feature extraction, feature fusion and output detection. The multi-scale feature extraction and lightweight design strategy are completed through parallel LSKBlock, and the AIFI and CCFM modules are used to effectively fuse image features while ensuring detection accuracy. The output detection module locates the target and category.

[0063] like Figure 2 As shown in the figure, the deep features of the image are extracted at different scales through multi-scale large convolution kernels. The module captures target objects of various sizes in the image at different levels. At the same time, the lightweight design reduces the computational overhead of the model.

[0064] like Figure 3 Figure 2: The proposed method is compared with several existing algorithms on a marine target detection dataset. The figure shows the performance of different algorithms in terms of accuracy, recall, mAP50, and mAP50-95.

[0065] like Figure 4 Figure 2 shows a comparison of the results of the present invention and several existing algorithms for marine target detection. The figure compares the detection results of different algorithms, visually demonstrating that the present invention reduces missed detections and false detections in complex marine environments, accurately identifying and locating target objects.

[0066] The effectiveness of the present invention is verified by simulation experiments below.

[0067] Simulation content:

[0068] (1) On the same training set and test images, six representative marine target detection methods are selected and compared with the simulation results of the present invention in the form of comparative experiments to verify the effectiveness of the present invention. Six representative detection methods are the method proposed by Wang et al., using YOLOv7 to achieve marine target detection tasks, with specific reference to "Wang CY, Bochkovskiy A, Liao HY M.YOLOv7:Trainable bag-of-freebies sets newstate-of-the-art for real-time object detectors[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2023:7464-7475.", the method proposed by Wang et al., using YOLOv9 to achieve marine target detection tasks, with specific reference to "Wang CY, YehI H, Liao HY M.Yolov9:Learning what you want to learn using programmable gradient information[J].arXiv preprint arXiv:2402.13616,2024.", the method proposed by Cheng et al., referred to as ODConvNeXt, with specific reference to "Cheng S, Zhu Y, Wu S.Deep learning based efficient ship detection from Drone-captured images for maritime surveillance[J].Ocean Engineering, 2023, 285:115440. ", the method proposed by Zhao et al., referred to as RT-DETR, specific reference "Zhao Y, Lv W, Xu S, et al. Detrs beat yolos on real-time object detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024:16965-16974.", and the use of YOLOv5 model and YOLOv8 model to implement PCB defect detection method.

[0069] (2) In order to further compare the target detection performance of the model, the test index comparison results of the four models under the same data set are calculated

[0070] The specific simulation conditions are described in each experiment.

[0071] In experiment one, the training and testing of the present application and other three methods are compared on the data set, and the comparison results are as shown in Figure 4 Figure 4 The results of the method of the present application are shown in the column, Figure 4 The detection results of the YOLOv5 model method are shown in the column, Figure 4 The detection results of the YOLOv9 model method are shown in the column, Figure 4 The detection results of the RT-DETR model method are shown in the column. From Figure 4 The simulation results can be seen: when comparing the results detected by different methods, most methods can produce satisfactory results. Compared with other methods, the present application has higher detection accuracy in detecting and identifying marine targets. Referring to the first row, in the case of a deeper background, the method of the present application has no missed detection, while other methods have different degrees of missed detection. Referring to the second and third rows of images, the detection of the present application is more comprehensive, and there is no error recognition and missed detection for different types of targets, but other algorithms have different degrees of error recognition or missed detection. Referring to the fourth row of images, the present algorithm still has good detection effect for small targets in the case of large sea wave interference.

[0072] In experiment two, the accuracy of the target detection is the key to evaluate the model for marine target detection. The detection accuracy of different methods is compared, as shown in Figure 3 The four indexes include accuracy, recall rate, mAP50 and mAP50-95.

[0073] From Figure 3 and Figure 4 The simulation results can be seen: the present application is obviously better than other methods in various aspects: 1) it has satisfactory performance in detecting small marine targets, and 2) it has fewer defects and missed detection.

[0074] The results of the above two experiments show that the present application has higher detection accuracy compared with the existing marine target detection method, can obtain more rich image features from the image, and can better detect small marine targets with less error recognition and missed detection.

[0075] ​This method utilizes multi-scale feature fusion and lightweight convolution operations to reduce the computational complexity of the model while ensuring detection accuracy, improving its real-time performance and applicability. This method is not only suitable for marine target detection but can also be applied to target detection tasks in other complex environments.

[0076] This paper focuses on a maritime target detection method based on a multi-scale lightweight deep network. By extracting multi-scale image information through convolutions of varying sizes and processing multi-scale features through a fusion module, the method acquires more comprehensive feature information about the detected target, effectively enhancing the network's ability to represent key features and improving the recognition rate of small-sized targets at sea, achieving a highly accurate detection method.

Claims

1. A method for detecting marine targets based on a multi-scale lightweight deep network, characterized in that: The following steps are included: Step (1): Annotate the acquired marine target images and construct an image dataset; Step (2): Divide the data into two parts: training set D and training set D. train With the test set D test ; Step (3): By inputting the training image into the feature extraction module, the input image is position-encoded to obtain F Patchembed ; Step (4): Select the position encoding feature map F through the weight selection function Patchembed , and get F Select ; Step (5): F Select Send it to the LSKBlock structure layer and combine it with the position encoding feature map F Patchembed And the feature selection function F Select Get the feature map F after extracting features at different scales LSKBlock ; Step (6): F LSKBlock Input into the feature fusion module and output F AIFI ; Step (7): F AIFI Through the CCFM module, the fused feature map F is obtained. CCFM ; Step (8): F CCFM Splice to get the final feature map; Step (9): Decode the obtained feature map and generate a bounding box; In step (4), the weight selection function is used to select the feature map after position encoding as follows: F Select =Sigmoid(mean(Conv1(f patchembed (I i ))),avg(Conv2 (f patchembed (I i )))), The weight selection function consists of the activation function Sigmoid, the average pooling operation mean, the maximum pooling operation avg, and the convolution Conv1 and Conv2; The convolution operation Conv extracts features at different levels by performing sliding window processing on the input feature map using the convolution kernel. The activation function Sigmoid maps the results of convolution and pooling to probability values ​​of weight selection, that is, each pixel in the output feature will be assigned a weight value for subsequent feature map fusion or selection. The average pooling operation mean and the maximum pooling operation avg divide the feature map into fixed-size windows and then take the average or maximum value in each window. The step (6) is specifically as follows: First, it passes through the AIFI module, which consists of shape change and position encoding. First, the result F of the previous step is input. LSKBlock Position encoding is performed to supplement the position information in the original image. The encoded features undergo shape changes. The shape change is achieved by expanding the dimension of the input tensor and readjusting the transposition operation so that the features at different stages maintain the same shape. The overall AIFI module operation is expressed as: F AIFI =TransShape(PositonEmbedding(F LSKBlock )), where TransShape is the shape change of the tensor and PositonEmbedding is the position encoding.

2. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: In the step (1), the constructed dataset image is used in the training process of the deep learning network model. i Represented as the i-th marine target image, the input image size is [W, H], the channel is C, and the input image size is adjusted to 640×640.

3. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: In the step (3), a position encoding operation is first performed to divide the input image into blocks of fixed size, and the position information of each block in the original image is supplemented to encode the input image I. i Position encoding can be expressed as: F Patchembed =f patchembed (I i ), where f patchembed () represents the position encoding operation, which divides the image into blocks, generates position information, and combines the global position and local position in the original image.

4. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: In the step (5), feature extraction is performed through the LSKBlock structure layer, F LSKBlock =F Patchembed ·F Select , combining the position encoding feature map with the feature selection function to obtain the feature map after feature extraction at different scales.

5. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: The step (7) is specifically as follows: First, the convolution operation is used to further extract and filter the input feature map. Then, the residual module composed of parallel convolution blocks is used to alleviate the gradient disappearance problem in the deep network. It can be expressed as: CCFM =RepBlock(Conv(F AIFI )), where RepBlock is the parallel combined residual module.

6. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: In the step (8): X feats , X shapes =GetEncoder(F CCFM ), where X feats , X shapes Represents the concatenated feature map and feature map shape, where GetEncoder includes flattening and permutation operations, and then concatenates the features to obtain a large feature map.

7. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 1, wherein: In the step (9): The decoded output generates a bounding box which can be represented as: Head j =Decoder(X feats , X shapes ),j<500; It can be expressed as Decoder = Socres(Bboxs(X feats , X shapes )), and j is the jth generated anchor point and valid mask, and the loss is calculated for the generated bounding box and label image, and the loss uses CIOU_Loss as the loss function.

8. The method for detecting marine targets based on a multi-scale lightweight deep network according to claim 7, wherein: The loss is calculated as follows: p=f d (Head,label),v=min(f s (Head,label)), CIOU=1-CIoU; p is the diagonal distance between the predicted bounding box and the true bounding box, c is the diagonal distance of the minimum bounding rectangle that can contain the predicted and true bounding boxes, v is the term used to correct the direction difference between the bounding boxes, and α is the influence of v.

Images