A deep learning-based offshore small target detection method

Abstract

This invention discloses a deep learning-based method for detecting small targets at sea, comprising the following steps: Step S1: Acquire video images of the actual sea area and perform frame segmentation processing on the video images to obtain an image dataset containing target features; Step S2: Annotate the image dataset to obtain a feature image test set and a feature image training set; Step S3: Based on the YOLOv5 network framework, construct a YOLO-sea network model with an attention mechanism module SCAM++ and an enhanced bidirectional feature fusion structure PANet; Step S4: Use DIoU as the loss function of YOLO-sea, and train the YOLO-sea network model using the image training set to obtain the optimized YOLO-sea model; Step S5: Use the optimized YOLO-sea model to perform target detection on the image test set. This method solves the problems of low accuracy and poor real-time performance in detecting extremely small targets near the horizon, which are concentrated together and affected by interference from the coastline and waves.

Description

Technical Field

[0001] This invention relates to the field of marine target detection technology, and in particular to a deep learning-based method for detecting small marine targets. Background Technology

[0002] The design of intelligent ship target detectors should meet the following two conditions. First, high-precision detection of obstacles on the sea surface. Second, the detection speed of the algorithm should meet the requirements of real-time and low latency. Common CNN-based target detection algorithms can be divided into two categories. The first category is a two-stage method based on region proposal generation, such as R-CNN[1], Faster R-CNN[2], etc. These methods generate region proposals in the first stage and classify and regress the contents of the region of interest in the second stage, losing the spatial information of local targets in the whole image, and the detection speed cannot reach real-time. The second category is a single-stage target detector, such as YOLO[3], RetinaNet[4] and SSD[5]. This type of algorithm does not directly generate the region of interest, but regards the target detection task as a regression task of the whole image. Therefore, the speed of single-stage detectors can mostly meet the requirements of real-time, but there are also problems of low object localization accuracy and poor recall. YOLOv5 can infer multiple objects at once, and the detection speed is extremely fast[6]. In addition, the low detection accuracy of YOLOv3[7] and YOLOv4[8] was further improved by adaptive design of anchor frames and optimization of network module structure.

[0003] In real maritime navigation scenarios, the marine optical images collected by shipborne cameras are characterized by a large number of small-scale targets. Due to the existence of perspective distortion in photogrammetry, distant objects appear smaller, and accurate detection of small targets at sea is a known problem [9]–

[11] . On a high-resolution image with a resolution of 1920×1080, the bounding boxes of objects near the ship are more than 120 times smaller than the obstacle targets near the horizon. It is worth noting that the extremely small targets near the horizon are concentrated together and are affected by the coast and waves, which greatly increases the difficulty of detecting small targets at sea. Since the existing algorithm models cannot meet the requirements of accurate and real-time obstacle detection in the autonomous driving scenario at sea, it is necessary to design a detection method that can efficiently handle the detection of targets at sea.

[0004] The references for the above technologies are as follows:

[0005] [1]R.Girshick,J.Donahue,T.Darrell,and J.Malik,“Rich FeatureHierarchies for Accurate Object Detection and Semantic Segmentation,”in2014IEEE Conference on Computer Vision and Pattern Recognition(CVPR),Columbus,OH,USA,Jun.2014,pp.580–587.doi:10.1109 / CVPR.2014.81.

[0006] [2]S.Ren,K.He,R.Girshick,and J.Sun,“Faster R-CNN:Towards Real-TimeObject Detection with Region Proposal Networks,”IEEE Transactions on PatternAnalysis and Machine Intelligence,vol.39,no.6,pp.1137–1149,2017.

[0007] [3]J.Redmon,S.Divvala,R.Girshick,and A.Farhadi,“You Only Look Once:Unified,Real-Time Object Detection,”in 2016IEEE Conference on Computer Visionand Pattern Recognition(CVPR),Las Vegas,Nevada,USA,2016,doi:10.1109 / CVPR.2016.91.

[0008] [4]T.-Y.Lin,P.Goyal,R.Girshick,K.He,and P.Dollar,“Focal Loss forDense Object Detection,”IEEE Transactions on Pattern Analysis and MachineIntelligence,vol.42,no.2,pp.318–327,Feb.2020,doi:10.1109 / TPAMI.2018.2858826.

[0009] [5] W. Liu et al., "SSD: Single Shot MultiBox Detector," in Proceedings of the European conference on computer vision (ECCV), 2016. doi:10.1007 / 978-3-319-46448-0_2.

[0010] [6]Ultralytics.YOLOv5.Available online: https: / / github.com / ultralytics / yolov5(accessed on 1November 2022).

[0011] [7] J.Redmon and A.Farhadi, "YOLOv3: An Incremental Improvement," 2018, arXiv:1804.02767.[Online],Available:http: / / arxiv.org / abs / 1804.02767.

[0012] [8] A. Bochkovskiy, C.-Y. Wang, and H.-YMLiao, "YOLOv4: Optimal Speed ​​and Accuracy of Object Detection," arXiv:2004.10934.[Online].Available:http: / / arxiv.org / abs / 2004.10934.

[0013] [9] Feng Hui, Jiang Chengxin, Ding Yihang, et al. Real-time small-scale target detection algorithm in real sea area [J / OL]. Journal of Huazhong University of Science and Technology (Natural Science Edition): 1-7 [2023-03-16]. https: / / doi.org / 10.13245 / j.hust.230404.

[0014]

[10] Zhao Wenqiang, Sun Wei. A method for detection and recognition of marine targets based on S4-YOLO [J]. Optics and Optoelectronics Technology, 2020, 18(4):38-46.

[0015]

[11] B.Iancu, V.Soloviev, L.Zelioli, and J.Lilius, "ABOships—An Inshoreand Offshore Maritime Vessel Detection Dataset with Precise Annotations," Remote Sensing, vol.13, no.5, p.988, Mar.2021, doi:10.3390 / rs13050988. Summary of the Invention

[0016] This invention provides a deep learning-based method for detecting small targets at sea, overcoming the technical problems of low detection accuracy and poor real-time performance caused by the concentration of extremely small targets near the horizon and interference from the coast and waves.

[0017] To achieve the above objectives, the technical solution of the present invention is: a deep learning-based method for detecting small targets at sea, comprising the following steps:

[0018] Step S1: Acquire video images of the actual sea area and perform frame segmentation processing on the video images to obtain an image dataset containing target features; the target features include contour, texture, and color;

[0019] Step S2: Label the image dataset to obtain a feature image test set and a feature image training set;

[0020] Step S3: Based on the YOLOv5 network framework, construct the YOLO-sea network model with the addition of the attention mechanism module SCAM++ and the enhanced bidirectional feature fusion structure PANet;

[0021] Step S4: Use DIoU as the loss function for YOLO-sea, train the YOLO-sea network model using the image training set, and obtain the optimized YOLO-sea model;

[0022] Step S5: Use the YOLO-sea optimization model to perform object detection on the image test set.

[0023] Furthermore, the structure of the YOLO-sea network model described in step S3 includes processing the image test set obtained in step S2 through a backbone network system, a feature fusion network system, and a detection head network system in sequence.

[0024] The backbone network system includes an initial layer convolutional Conv module, an intermediate layer network module, an attention mechanism module SCAM++, and an SPPF module;

[0025] The initial layer convolutional Conv module extracts features from the training images in the image training set to obtain an initial feature map. The target features include the outline, texture, and color of the target.

[0026] The intermediate layer network module includes a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a fourth feature extraction unit connected in sequence, and the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, and the fourth feature extraction unit are all composed of a deformable convolutional DCN v2 module and a first C3 module;

[0027] The deformable convolutional DCN v2 module is used to assign a set weight coefficient to the offset of each sampling point on the feature map input to the deformable convolutional DCN v2 module; the offset of the sampling point includes the size of the target feature, the angle transformation, and the proportion of the target feature relative to the feature map;

[0028] The C3 module is used to perform residual feature learning and feature fusion on the feature map output by the deformable convolutional DCN v2 module.

[0029] The first feature extraction unit extracts and fuses features from the initial feature map to obtain a low-level feature map; and then transmits the low-level feature map to the second feature extraction unit.

[0030] The second feature extraction unit extracts and fuses features from the low-level feature map to obtain a first mid-level feature map; and transmits the first mid-level feature map to the third feature extraction unit;

[0031] The third feature extraction unit extracts and fuses features from the first mid-layer feature map to obtain a second mid-layer feature map; and transmits the second mid-layer feature map to the fourth feature extraction unit;

[0032] The fourth feature extraction unit extracts and fuses features from the second middle-layer feature map to obtain a high-level feature map; and transmits the high-level feature map to the attention mechanism module SCAM++.

[0033] The layer attention mechanism module SCAM++ performs weighted processing on the high-level feature map to obtain an optimized feature map, and then transmits the optimized feature map to the SPPF module;

[0034] The SPPF module performs feature fusion on the optimized feature map to obtain feature maps at different scales.

[0035] Furthermore, the feature fusion network system includes a first feature fusion unit, a second feature fusion unit, a third feature fusion unit, a fourth feature fusion unit, a fifth feature fusion unit, and a sixth feature fusion unit connected in sequence.

[0036] The first feature fusion unit, the second feature fusion unit, and the third feature fusion unit each include a first convolutional Conv module, an upsample module, a first Concat module, and a second C3 module connected in sequence.

[0037] The fourth feature fusion unit, the fifth feature fusion unit and the sixth feature fusion unit each include a second convolutional Conv module, a second Concat module and a third C3 module connected in sequence.

[0038] The first and second convolutional Conv modules are used to reduce the size of the input feature map and perform feature extraction; the size of the feature map includes the feature map width W, the feature map height H, and the resolution.

[0039] The Upsample module is used to enlarge the size of the feature map while keeping the number of feature map channels N constant.

[0040] The first Concat module and the second Concat module are used to increase the number of channels N of the feature map while keeping the feature map size unchanged; and to optimize the feature map by combining the strong semantic information of the high-level feature map and the localization information of the low-level feature map.

[0041] The strong semantic information refers to the coarse-grained information of the image, and the positioning information includes the texture, color, edge, and corner information of the target features;

[0042] The output of the first feature extraction unit and the output of the Upsample module of the third feature fusion unit are connected to the second C3 module of the third feature fusion unit after channel splicing through the first Concat module of the third feature fusion unit; and the second C3 module of the third feature fusion unit serves as the input module of the extremely small scale target detection head and is connected to the detection head network system.

[0043] The output of the second feature extraction unit is connected to the first convolutional Conv module of the third feature fusion unit, and the output of the first convolutional Conv module of the third feature fusion unit is connected to the output of the Upsample module of the second feature fusion unit. After channel splicing by the first Concat module of the second feature fusion unit, the output is input to the second C3 module of the second feature fusion unit.

[0044] The output of the first convolutional Conv module of the third feature fusion unit and the output of the second convolutional Conv module of the fourth feature fusion unit are connected to the third C3 module of the fourth feature fusion unit after channel splicing through the second Concat module of the fourth feature fusion unit; the third C3 module of the fourth feature fusion unit serves as the input module of the small-scale target detection head and is connected to the detection head network system.

[0045] The output of the third feature extraction unit is connected to the first convolutional Conv module of the second feature fusion unit, and the output of the third feature extraction unit is connected to the output of the Upsample module of the first feature fusion unit. After channel splicing by the first Concat module of the first feature fusion unit, the output is input to the second C3 module of the first feature fusion unit.

[0046] The output of the first convolutional Conv module of the second feature fusion unit and the output of the second convolutional Conv module of the fifth feature fusion unit are connected to the third C3 module of the fifth feature fusion unit after channel splicing through the second Concat module of the fifth feature fusion unit; the third C3 module of the fifth feature fusion unit serves as the input module of the mesoscale target detection head and is connected to the detection head network system.

[0047] The output of the first convolutional Conv module of the first feature fusion unit and the output of the second convolutional Conv module of the sixth feature fusion unit are connected to the third C3 module of the sixth feature fusion unit after channel splicing through the first Concat module of the first feature fusion unit; the third C3 module of the sixth feature fusion unit serves as the input module of the large-scale target detection head and is connected to the detection head network system.

[0048] The detection head network system uses the NMS (Non-Maximum Suppression) algorithm to merge the output feature maps of the second C3 module of the third feature fusion unit, the third C3 module of the fourth feature fusion unit, the third C3 module of the fifth feature fusion unit, and the third C3 module of the sixth feature fusion unit to obtain the final detection image of small maritime targets.

[0049] Furthermore, each of the C3 modules includes a first branch and a second branch;

[0050] The first branch includes a third convolutional Conv module and several bottlenecks connected in series; the second branch includes a fourth convolutional Conv module; the outputs of the two branches are concatenated by a third Concat module and then connected in series with a fifth convolutional Conv module.

[0051] Furthermore, the attention mechanism module SCAM++ mentioned in step S3 includes a channel attention module and a spatial attention module;

[0052] The channel attention module includes a first global pooling layer, a first average pooling layer, a fully connected layer, and a Silu activation function layer;

[0053] The first global pooling layer and the first average pooling layer respectively obtain the feature map output by the eighth layer C3 module, and perform adaptive average pooling and adaptive max pooling operations to obtain the global feature vector;

[0054] The features of the global feature vector include color, texture, and contour in the dimensional direction;

[0055] The formulas for performing max pooling operations on the first global pooling layer and the first average pooling layer are expressed as follows:

[0056]

[0057] F B =max(a(x,y))

[0058] In the formula: F A The global feature vector is obtained by performing average pooling on the feature map; F B It is the global feature vector obtained by adaptive max pooling of the feature map; H is the height of the feature map; W is the width of the feature map; a(x,y) is the feature value of each pixel position in the feature map;

[0059] The two global feature vectors obtained after pooling are passed to the fully connected layer of the multilayer perceptron (MLP) for convolutional classification to obtain optimized feature vectors.

[0060] The optimized feature vector uses the sigmoid activation function to generate channel attention weights Mc, expressed by the formula:

[0061] M C =sigmoid(MLP(F A )+MLP(F B ))

[0062] The Silu activation function is used to improve the weight allocation of the channel attention weight Mc, obtaining the optimized feature vector with the optimal weight values. The Silu activation function is expressed as follows:

[0063] Silu(x) = x × Sigmoid(x)

[0064] The channel attention weights M obtained after improving weight allocation c It can be represented as:

[0065] MC =Silu(MLP(F A )+MLP(F B ))

[0066] The spatial attention module includes a second global pooling layer, a second average pooling layer, a convolutional layer, and a sigmoid activation function layer;

[0067] The feature map output by the first C3 module of the fourth feature extraction unit is compressed in the channel dimension by the second global pooling layer and the second average pooling layer respectively to obtain a multi-dimensional feature map.

[0068] After concatenating the two multidimensional feature maps by channels, the convolutional layer is used to perform dimensionality reduction processing on the multidimensional feature maps to obtain dimensionality-reduced feature maps.

[0069] The reduced-dimensional feature map generates spatial attention weights Ms using a sigmoid activation function, expressed by the following formula:

[0070] M S =sigmoid(conv((F A ), (F B )))

[0071] The channel attention module and the spatial attention module are summed to obtain the attention mechanism module SCAM++, expressed by the formula:

[0072]

[0073] Furthermore, in step S4, DIoU is used as the loss function to calculate the target box regression, and the DIoU calculates the distance loss between the predicted object box and the ground truth box. The calculation formula is as follows:

[0074]

[0075] Among them, b and b gt These represent the center points of the predicted bounding box and the ground truth bounding box, respectively; ρ represents the Euclidean distance; ρ(b,b) gt ) represents the Euclidean distance between the center point of the predicted bounding box and the ground truth bounding box; c represents the diagonal distance between the minimum outer rectangles of the predicted bounding box and the ground truth bounding box.

[0076] Beneficial Effects: This invention provides a deep learning-based method for detecting small targets at sea. It employs extended deformable convolution to enhance modeling capabilities and uses a parallel spatial-channel attention mechanism to increase attention to areas where small targets cluster. Secondly, to perform feature fusion, an enhanced bidirectional feature fusion module is designed, and skip connections and concat channel concatenation are introduced. This combines the localization information of low-level feature maps with the strong semantic information of high-level feature maps. At the same time, a new minimal target detection layer is added to improve the localization accuracy of distant targets near the horizon. Finally, by optimizing the target loss function, the regression ability of the model to detect target boxes is improved, thereby comprehensively improving the detection accuracy of small targets at sea while ensuring real-time performance. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart of a deep learning-based method for detecting small targets at sea according to the present invention.

[0079] Figure 2 This is a schematic diagram of the overall architecture of YOLO-sea, a deep learning-based method for detecting small targets at sea according to the present invention.

[0080] Figure 3 This is a schematic diagram of the attention mechanism module SCAM++ of a deep learning-based method for detecting small targets at sea according to the present invention.

[0081] Figure 4 This is a schematic diagram of PANet, an enhanced bidirectional feature fusion module for a deep learning-based method for detecting small targets at sea according to the present invention.

[0082] Figure 5 A schematic diagram of the C3 module of a deep learning-based method for detecting small targets at sea according to the present invention;

[0083] Figure 6 This is a schematic diagram of the SPPF module of a deep learning-based method for detecting small targets at sea according to the present invention.

[0084] Figure 7 This is a schematic diagram of the Concat module of a deep learning-based method for detecting small targets at sea according to the present invention.

[0085] Figure 8This is a schematic diagram of the Upsample module in the deep learning-based method for detecting small targets at sea according to the present invention.

[0086] Figure 9 This is a schematic diagram of PANet, an enhanced bidirectional feature fusion module for a deep learning-based method for detecting small targets at sea according to the present invention.

[0087] Figure 10 This is a schematic diagram of the loss function in a deep learning-based method for detecting small targets at sea proposed in this invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] This embodiment provides a deep learning-based method for detecting small targets at sea, such as... Figure 1 As shown, it includes the following steps:

[0090] Step S1: Acquire video images of the actual sea area and perform frame segmentation processing on the video images to obtain an image dataset containing target features; the target features include contour, texture, and color; the target features also include target category information, which is the feature information of different targets in the image;

[0091] Step S2: Manually proofread and annotate the image dataset using Labelimg annotation software to obtain a feature image test set and a feature image training set;

[0092] Step S3: Based on the YOLOv5 network framework, construct the YOLO-sea network model with the addition of the attention mechanism module SCAM++ and the enhanced bidirectional feature fusion structure PANet;

[0093] Step S4: Use DIoU as the loss function for YOLO-sea, train the YOLO-sea network model using the image training set, and obtain the optimized YOLO-sea model;

[0094] Step S5: Use the YOLO-sea optimization model to perform object detection on the image test set.

[0095] This invention designs a deep learning-based method for detecting small targets at sea. First, a single-stage target detector based on anchor boxes typically consists of a backbone network, a feature fusion part (usually called the Neck), and a detection head for target classification and localization. Using a YOLOv5-based network framework, the backbone network is optimized using deformable convolutional network DCN v2, and an attention mechanism module SCAM++ is introduced to make the network pay more attention to small target regions. Second, in the feature fusion part, an enhanced bidirectional feature fusion structure PANet is designed, which not only introduces strong semantic information into the low-level feature maps but also introduces localization information into the high-level feature maps. Finally, skip connections and multi-layer feature fusion further enhance the fusion efficiency between feature maps of different scales. Finally, a very small target detection layer is added to predict small targets at sea on the low-level feature maps, improving the localization accuracy of very small targets at long distances at sea. The YOLO-sea optimized model, by fully considering the characteristics and location distribution of maritime targets, is suitable for small target detection in autonomous maritime operations. At the same time, the YOLO-sea optimized model can accurately locate small targets at long distances at sea and supports real-time operation, achieving a good trade-off between accuracy and real-time performance, thus providing a solid foundation for the realization of intelligent navigation function modules for smart ships.

[0096] In a specific embodiment, such as Figure 2 As shown, the structure of the YOLO-sea network model described in step S3 includes processing the image test set obtained in step S2 through the backbone network system, the feature fusion network system, and the detection head network system 18 in sequence.

[0097] The backbone network system includes an initial layer convolutional Conv module, an intermediate layer network module 11, an attention mechanism module SCAM++, and an SPPF module;

[0098] The initial layer convolutional Conv module extracts features from the training images in the image training set to obtain an initial feature map, the features including the contour, texture and color of the target;

[0099] The intermediate layer network module 11 includes a first feature extraction unit 101, a second feature extraction unit 102, a third feature extraction unit 103, and a fourth feature extraction unit 104 connected in sequence. The first feature extraction unit 101, the second feature extraction unit 102, the third feature extraction unit 103, and the fourth feature extraction unit 104 are all composed of a deformable convolutional DCNv2 module and a first C3 module.

[0100] The deformable convolutional DCN v2 module is used to assign a set weight coefficient to the offset of each sampling point on the feature map input to the deformable convolutional DCN v2 module; the offset of the sampling point includes the size of the target feature, the angle transformation, and the proportion of the target feature relative to the feature map;

[0101] The C3 module is used to perform residual feature learning and feature fusion on the feature map output by the deformable convolutional DCN v2 module.

[0102] The first feature extraction unit 101 performs feature extraction and fusion on the initial feature map to obtain a low-level feature map; and transmits the low-level feature map to the second feature extraction unit 102;

[0103] The second feature extraction unit 102 performs feature extraction and fusion on the low-level feature map to obtain a first mid-level feature map; and transmits the first mid-level feature map to the third feature extraction unit 103;

[0104] The third feature extraction unit 103 extracts and fuses features from the first mid-layer feature map to obtain a second mid-layer feature map; and transmits the second mid-layer feature map to the fourth feature extraction unit 104;

[0105] The fourth feature extraction unit 104 performs feature extraction and fusion on the second middle-layer feature map to obtain a high-level feature map; and transmits the high-level feature map to the attention mechanism module SCAM++;

[0106] The layer attention mechanism module SCAM++ performs weighted processing on the high-level feature map to obtain an optimized feature map, and then transmits the optimized feature map to the SPPF module;

[0107] The SPPF module performs feature fusion on the optimized feature map to obtain feature maps at different scales.

[0108] A deformable convolutional model is introduced using a YOLOv5-based network framework. First, deformable convolutions are used to optimize the backbone network. Existing traditional convolutions use regular windows for feature extraction, which are difficult to adapt to geometric transformations of targets. DCNv1, however, uses learnable offsets to describe target features. Through transformations in size, scale, and angle, it can better adapt to the geometric transformations of maritime targets, such as… Figure 4As shown, the Deformable Convolutional DCNv2 module, based on DCNv1, assigns weight coefficients to the offset of each sampling point to evaluate whether the introduced region is a region of interest, further improving the accuracy of information extraction. Furthermore, the Deformable Convolutional DCNv2 module expands upon this by incorporating more deformable convolutions, enhancing the network's ability to extract features from various geometric deformations. By introducing the Deformable Convolutional DCNv2 module into the YOLOv5 backbone network, where circles represent sampling points of the convolution, deformable convolutions can cover more effective regions, enabling the extraction of better target features from marine images.

[0109] like Figure 9 As shown, the feature fusion network system further includes a first feature fusion unit 12, a second feature fusion unit 13, a third feature fusion unit 14, a fourth feature fusion unit 15, a fifth feature fusion unit 16, and a sixth feature fusion unit 17 connected in sequence.

[0110] The first feature fusion unit 12, the second feature fusion unit 13, and the third feature fusion unit 14 each include a first convolutional Conv module, an upsample module, a first Concat module, and a second C3 module connected in sequence.

[0111] The fourth feature fusion unit 15, the fifth feature fusion unit 16 and the sixth feature fusion unit 17 each include a second convolutional Conv module, a second Concat module and a third C3 module connected in sequence.

[0112] The first and second convolutional Conv modules are used to reduce the size of the input feature map and perform feature extraction; the size of the feature map includes the feature map width W, the feature map height H, and the resolution.

[0113] The Upsample module is used to enlarge the size of the feature map while keeping the number of feature map channels N constant.

[0114] The first Concat module and the second Concat module are used to increase the number of channels N of the feature map while keeping the feature map size unchanged; and to optimize the feature map by combining the strong semantic information of the high-level feature map and the localization information of the low-level feature map.

[0115] The strong semantic information refers to the coarse-grained information of the image, and the positioning information includes the texture, color, edge, and corner information of the target features;

[0116] The output of the first feature extraction unit 101 and the output of the Upsample module of the third feature fusion unit 14 are connected to the second C3 module of the third feature fusion unit 14 after channel splicing through the first Concat module of the third feature fusion unit 14; and the second C3 module of the third feature fusion unit 14 serves as the input module of the extremely small scale target detection head and is connected to the detection head network system.

[0117] The output of the second feature extraction unit 102 is connected to the first convolution Conv module of the third feature fusion unit 14, and the output of the first convolution Conv module of the third feature fusion unit 14 is connected to the output of the Upsample module of the second feature fusion unit 13. After channel splicing by the first Concat module of the second feature fusion unit 13, the output is input to the second C3 module of the second feature fusion unit 13.

[0118] The output of the first convolutional Conv module of the third feature fusion unit 14 and the output of the second convolutional Conv module of the fourth feature fusion unit 15 are concatenated by the second Concat module of the fourth feature fusion unit 15 and then input to the third C3 module of the fourth feature fusion unit 15; the third C3 module of the fourth feature fusion unit 15 serves as the input module of the small-scale target detection head and is connected to the detection head network system.

[0119] The output of the third feature extraction unit 103 is connected to the first convolution Conv module of the second feature fusion unit 13, and the output of the third feature extraction unit 103 is connected to the output of the Upsample module of the first feature fusion unit 12. After channel splicing by the first Concat module of the first feature fusion unit 12, the output is input to the C3 module of the first feature fusion unit 12.

[0120] The output of the first convolutional Conv module of the second feature fusion unit 13 and the output of the second convolutional Conv module of the fifth feature fusion unit 16 are concatenated by the second Concat module of the fifth feature fusion unit 16 and then input to the third C3 module of the fifth feature fusion unit 16; the third C3 module of the fifth feature fusion unit 16 serves as the input module of the mesoscale target detection head and is connected to the detection head network system.

[0121] The output of the first convolutional Conv module of the first feature fusion unit 12 and the output of the second convolutional Conv module of the sixth feature fusion unit 17 are concatenated by the first Concat module of the first feature fusion unit 12 and then input to the third C3 module of the sixth feature fusion unit 17. The third C3 module of the sixth feature fusion unit 17 serves as the input module of the large-scale target detection head and is connected to the detection head network system. The detection head network system includes third convolutional Conv modules that are respectively connected to the output of the second C3 module of the third feature fusion unit 14, the third C3 module of the fourth feature fusion unit 15, the third C3 module of the fifth feature fusion unit 16, and the third C3 module of the sixth feature fusion unit 17.

[0122] The detection head network system 18 uses the NMS (Non-Maximum Suppression) algorithm to merge the output feature maps of the second C3 module of the third feature fusion unit 14, the third C3 module of the fourth feature fusion unit 15, the third C3 module of the fifth feature fusion unit 16, and the third C3 module of the sixth feature fusion unit 17 to obtain the final detection image of small targets at sea.

[0123] The enhanced bidirectional feature fusion structure PANet abandons the idea of ​​weighted fusion, combining bidirectional skip connections and Concat module feature fusion. It does not compress the channel dimension, but instead preserves as many features as possible at the cost of pre-defined memory resources. Figure 7As shown, feature fusion in each Concat module can fully utilize feature channel resources, thereby achieving better detection performance. Furthermore, a small target detection head is added to the original three detection heads of the YOLOv5 network, which can effectively mitigate the negative impact of scale variance. The added tiny target detection head targets low-level feature maps. Low-level feature maps have large and dense receptive fields, containing richer edge and texture features of the target. This allows for more accurate localization of small targets on low-level feature maps, thereby improving the detection capability of tiny targets near the sea-line area. Each convolutional Conv module encapsulates three functions: 2D convolution operation Conv2D, batch normalization (BNU), and the activation function SiLU. The padding of Conv2D in each Conv module is automatically calculated, and the stride is used to determine the feature map reduction factor. In this example, the stride of all convolutional Conv modules in the network is 2, and the kernel size is 3. Therefore, each Conv module halves the width and height of the feature map. The Conv module's function is to extract features and organize the feature map. Batch Normalization is a batch normalization layer that normalizes each batch of data. SiLU is the activation function used to add non-linear data to the feature map.

[0124] like Figure 6 As shown, the SPPF module is a spatial pyramid pooling module. It uses multiple max-pooling layers (MaxPool2d) in series and replaces the convolution operations of 9×9 and 13×13 convolution kernels with 5×5 kernels, thus improving the network's computational speed while maintaining the same receptive field. Each Concat module is a channel concatenation module, which increases the number of channels N while keeping the feature map width H and height W constant. Figure 8 As shown, the UpSample module functions to enlarge the size of the feature map while keeping the number of channels N constant.

[0125] In a specific embodiment, such as Figure 5 As shown, the function of each C3 module is to learn residual features, and each C3 module includes a first branch and a second branch; the first branch includes a third convolutional Conv module and several bottleneck layers connected in series; the second branch includes a fourth convolutional Conv module; the outputs of the two branches are concatenated by the third Concat module and then connected to a fifth convolutional Conv module; the functions of each convolutional Conv module and each Concat module are the same.

[0126] In a specific embodiment, such as Figure 3As shown, the attention mechanism module SCAM++ mentioned in step S3 includes a channel attention module and a spatial attention module;

[0127] The channel attention module includes a first global pooling layer, a first average pooling layer, a fully connected layer, and a Silu activation function layer;

[0128] The first global pooling layer and the first average pooling layer respectively obtain the feature map output by the eighth layer C3 module, and perform adaptive average pooling and adaptive max pooling operations to obtain the global feature vector;

[0129] The features of the global feature vector include color, texture, and contour in the dimensional direction;

[0130] The formulas for performing max pooling operations on the first global pooling layer and the first average pooling layer are expressed as follows:

[0131]

[0132] F B =max(a(x,y))

[0133] In the formula: F A The global feature vector is obtained by performing average pooling on the feature map; F B It is the global feature vector obtained by adaptive max pooling of the feature map; H is the height of the feature map; W is the width of the feature map; a(x,y) is the feature value of each pixel position in the feature map;

[0134] The two global feature vectors obtained after pooling are passed to the fully connected layer of the multilayer perceptron (MLP) for convolutional classification to obtain optimized feature vectors.

[0135] The optimized feature vector uses the sigmoid activation function to generate channel attention weights Mc, expressed by the formula:

[0136] M C =sigmoid(MLP(F A )+MLP(F B ))

[0137] The Silu activation function is used to improve the weight allocation of the channel attention weight Mc, obtaining the optimized feature vector with the optimal weight values. The Silu activation function is expressed as follows:

[0138] Silu(x) = x × Sigmoid(x)

[0139] The channel attention weights M obtained after improving weight allocation c It can be represented as:

[0140] MC =Silu(MLP(F A )+MLP(F B ))

[0141] The spatial attention module includes a second global pooling layer, a second average pooling layer, a convolutional layer, and a sigmoid activation function layer;

[0142] The feature map output by the first C3 module of the fourth feature extraction unit is compressed in the channel dimension by the second global pooling layer and the second average pooling layer respectively to obtain a multi-dimensional feature map.

[0143] After concatenating the two multidimensional feature maps by channels, the convolutional layer is used to perform dimensionality reduction processing on the multidimensional feature maps to obtain dimensionality-reduced feature maps.

[0144] The reduced-dimensional feature map generates spatial attention weights Ms using a sigmoid activation function, expressed by the following formula:

[0145] M S =sigmoid(conv((F A ), (F B )))

[0146] The channel attention module and the spatial attention module are summed to obtain the attention mechanism module SCAM++, expressed by the formula:

[0147]

[0148] In a specific embodiment, such as Figure 10 As shown, in step S4, DIoU is used as the loss function to calculate the target box regression, and DIoU calculates the distance loss between the predicted object box and the ground truth box. The calculation formula is as follows:

[0149]

[0150] Among them, b and b gt These represent the center points of the predicted bounding box and the ground truth bounding box, respectively; ρ represents the Euclidean distance; ρ(b,b) gt ) represents the Euclidean distance between the center point of the predicted bounding box and the ground truth bounding box; c represents the diagonal distance between the minimum outer rectangles of the predicted bounding box and the ground truth bounding box. Using the DIoU loss function to obtain the distance loss between the predicted bounding box and the ground truth bounding box is more consistent with the mechanism of target bounding box regression, taking into account factors such as the distance between the target and the anchor box, and the overlap rate, making the target bounding box regression more stable.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based method for detecting small targets at sea, characterized in that, Includes the following steps: Step S1: Acquire video images of the actual sea area and perform frame segmentation processing on the video images to obtain an image dataset containing target features; the target features include contour, texture, and color; Step S2: Label the image dataset to obtain a feature image test set and a feature image training set; Step S3: Based on the YOLOv5 network architecture, construct a YOLO-sea network model with the addition of the attention mechanism module SCAM++ and the enhanced bidirectional feature fusion structure PANet; the structure of the YOLO-sea network model includes processing the image test set obtained in step S2 through the backbone network system, the feature fusion network system, and the detection head network system in sequence. The backbone network system includes an initial layer convolutional Conv module, an intermediate layer network module, an attention mechanism module SCAM++, and an SPPF module; The initial layer convolutional Conv module extracts features from the training images in the image training set to obtain an initial feature map. The target features include the outline, texture, and color of the target. The intermediate layer network module includes a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a fourth feature extraction unit connected in sequence, and the first feature extraction unit, the second feature extraction unit, the third feature extraction unit, and the fourth feature extraction unit are all composed of a deformable convolutional DCN v2 module and a first C3 module; The deformable convolutional DCN v2 module is used to assign a set weight coefficient to the offset of each sampling point on the feature map input to the deformable convolutional DCN v2 module; the offset of the sampling point includes the size of the target feature, the angle transformation, and the proportion of the target feature relative to the feature map; The C3 module is used to perform residual feature learning and feature fusion on the feature map output by the deformable convolutional DCN v2 module. The first feature extraction unit extracts and fuses features from the initial feature map to obtain a low-level feature map; and then transmits the low-level feature map to the second feature extraction unit. The second feature extraction unit extracts and fuses features from the low-level feature map to obtain a first mid-level feature map; and transmits the first mid-level feature map to the third feature extraction unit; The third feature extraction unit extracts and fuses features from the first mid-layer feature map to obtain a second mid-layer feature map; and transmits the second mid-layer feature map to the fourth feature extraction unit; The fourth feature extraction unit extracts and fuses features from the second middle-layer feature map to obtain a high-level feature map; and transmits the high-level feature map to the attention mechanism module SCAM++. The layer attention mechanism module SCAM++ performs weighted processing on the high-level feature map to obtain an optimized feature map, and then transmits the optimized feature map to the SPPF module; The SPPF module performs feature fusion on the optimized feature map to obtain feature maps at different scales. Step S4: Use DIoU as the loss function for YOLO-sea, train the YOLO-sea network model using the image training set, and obtain the optimized YOLO-sea model; Step S5: Use the YOLO-sea optimization model to perform object detection on the image test set.

2. The method for detecting small maritime targets based on deep learning according to claim 1, characterized in that, The feature fusion network system includes a first feature fusion unit, a second feature fusion unit, a third feature fusion unit, a fourth feature fusion unit, a fifth feature fusion unit, and a sixth feature fusion unit connected in sequence. The first feature fusion unit, the second feature fusion unit, and the third feature fusion unit each include a first convolutional Conv module, an upsample module, a first Concat module, and a second C3 module connected in sequence. The fourth feature fusion unit, the fifth feature fusion unit and the sixth feature fusion unit each include a second convolutional Conv module, a second Concat module and a third C3 module connected in sequence. The first and second convolutional Conv modules are used to reduce the size of the input feature map and perform feature extraction; the size of the feature map includes the feature map width W, the feature map height H, and the resolution. The Upsample module is used to enlarge the size of the feature map while keeping the number of feature map channels N constant. The first Concat module and the second Concat module are used to increase the number of channels N of the feature map while keeping the feature map size unchanged; and to optimize the feature map by combining the strong semantic information of the high-level feature map and the localization information of the low-level feature map. The strong semantic information refers to the coarse-grained information of the image, and the positioning information includes the texture, color, edge, and corner information of the target features; The output of the first feature extraction unit and the output of the Upsample module of the third feature fusion unit are connected to the second C3 module of the third feature fusion unit after channel splicing through the first Concat module of the third feature fusion unit; and the second C3 module of the third feature fusion unit serves as the input module of the extremely small scale target detection head and is connected to the detection head network system. The output of the second feature extraction unit is connected to the first convolutional Conv module of the third feature fusion unit, and the output of the first convolutional Conv module of the third feature fusion unit is connected to the output of the Upsample module of the second feature fusion unit. After channel splicing by the first Concat module of the second feature fusion unit, the output is input to the second C3 module of the second feature fusion unit. The output of the first convolutional Conv module of the third feature fusion unit and the output of the second convolutional Conv module of the fourth feature fusion unit are connected to the third C3 module of the fourth feature fusion unit after channel splicing through the second Concat module of the fourth feature fusion unit; the third C3 module of the fourth feature fusion unit serves as the input module of the small-scale target detection head and is connected to the detection head network system. The output of the third feature extraction unit is connected to the first convolutional Conv module of the second feature fusion unit, and the output of the third feature extraction unit is connected to the output of the Upsample module of the first feature fusion unit. After channel splicing by the first Concat module of the first feature fusion unit, the output is input to the second C3 module of the first feature fusion unit. The output of the first convolutional Conv module of the second feature fusion unit and the output of the second convolutional Conv module of the fifth feature fusion unit are connected to the third C3 module of the fifth feature fusion unit after channel splicing through the second Concat module of the fifth feature fusion unit; the third C3 module of the fifth feature fusion unit serves as the input module of the mesoscale target detection head and is connected to the detection head network system. The output of the first convolutional Conv module of the first feature fusion unit and the output of the second convolutional Conv module of the sixth feature fusion unit are connected to the third C3 module of the sixth feature fusion unit after channel splicing through the first Concat module of the first feature fusion unit; the third C3 module of the sixth feature fusion unit serves as the input module of the large-scale target detection head and is connected to the detection head network system. The detection head network system uses the NMS (Non-Maximum Suppression) algorithm to merge the output feature maps of the second C3 module of the third feature fusion unit, the third C3 module of the fourth feature fusion unit, the third C3 module of the fifth feature fusion unit, and the third C3 module of the sixth feature fusion unit to obtain the final detection image of small maritime targets.

3. The method for detecting small maritime targets based on deep learning according to claim 2, characterized in that, Each of the C3 modules includes a first branch and a second branch; The first branch includes a third convolutional Conv module and several bottlenecks connected in series; the second branch includes a fourth convolutional Conv module; the outputs of the two branches are concatenated by a third Concat module and then connected in series with a fifth convolutional Conv module.

4. The method for detecting small maritime targets based on deep learning according to claim 3, characterized in that, The attention mechanism module SCAM++ mentioned in step S3 includes a channel attention module and a spatial attention module; The channel attention module includes a first global pooling layer, a first average pooling layer, a fully connected layer, and a Silu activation function layer; The first global pooling layer and the first average pooling layer respectively obtain the feature map output by the eighth layer C3 module, and perform adaptive average pooling and adaptive max pooling operations to obtain the global feature vector; The features of the global feature vector include color, texture, and contour in the dimensional direction. ； The formulas for performing max pooling operations on the first global pooling layer and the first average pooling layer are expressed as follows: In the formula: F A It is the global feature vector obtained by performing average pooling on the feature map; F B It is a global feature vector obtained by adaptive max pooling of the feature map; H The height of the feature map; W y is the width of the feature map; a(x, y) is the feature value at each pixel position in the feature map; The two global feature vectors obtained after pooling are passed to the multilayer perceptron. The fully connected layer is used to perform convolutional classification to obtain optimized feature vectors; The optimized feature vector uses the sigmoid activation function to generate channel attention weights. Mc The formula is expressed as Channel attention weights are applied using the activation function Silu. Mc The weight allocation is improved to obtain the optimized feature vector with the optimal weight values. The activation function Silu is expressed as follows: Channel attention weights obtained after improved weight allocation M c It can be represented as: The spatial attention module includes a second global pooling layer, a second average pooling layer, a convolutional layer, and a sigmoid activation function layer; The feature map output by the first C3 module of the fourth feature extraction unit is compressed in the channel dimension by the second global pooling layer and the second average pooling layer respectively to obtain a multi-dimensional feature map. After concatenating the two multidimensional feature maps by channels, the convolutional layer is used to perform dimensionality reduction processing on the multidimensional feature maps to obtain dimensionality-reduced feature maps. The dimensionality-reduced feature map generates spatial attention weights using the sigmoid activation function. Ms The formula is expressed as The channel attention module and the spatial attention module are summed to obtain the attention mechanism module SCAM++, expressed by the formula: 。 5. The deep learning-based method for detecting small maritime targets according to claim 4, characterized in that, In step S4, DIoU is used as the loss function to calculate the target bounding box regression. DIoU calculates the distance loss between the predicted object bounding box and the ground truth bounding box. The calculation formula is as follows: in, b and b gt These represent the center points of the predicted bounding box and the ground truth bounding box, respectively. ρ Indicates Euclidean distance; ρ ( b , b gt () represents the Euclidean distance between the center point of the predicted bounding box and the actual object bounding box; c This represents the diagonal distance between the smallest outer rectangle of the predicted bounding box and the ground truth bounding box.

Images