An ert-detr model for water surface small target detection

Abstract

The application relates to an ERT-DETR model for water surface small target detection, which comprises the following steps: input of a dynamic feature pyramid network is the features of the last four stages of a backbone network; an AIFI module processes high-level semantic features by using a self-attention mechanism, and key features are extracted from the deepest layer of the backbone network; a RepBi-PAN network fuses the key features with low-level features extracted from the backbone network to obtain fused features; an SPD-Conv module performs multi-scale fusion convolution on the fused features to obtain convolution features; a depth separable convolution module performs depth convolution on the features after being segmented along the channel direction to obtain depth-convoluted features; a point-by-point convolution module extracts key features after integrating and refining the depth-convoluted features; and a channel attention module is used for strengthening the channel features of the key features to obtain a weighted feature map, so that the problem that it is difficult to simultaneously consider real-time performance and end-to-end performance when detecting water floating small targets in a complex water surface environment is solved.

Description

Technical Field

[0001] This invention relates to the field of small target detection, and more particularly to an ERT-DETR model for detecting small targets on the water surface. Background Technology

[0002] 85% of marine debris is plastic, and this amount is increasing by 23-37 million tons annually. If this type of floating debris, including plastic waste, is not monitored and managed, it will not only affect the safety and efficiency of maritime transport, but also, once it degrades and enters aquatic ecosystems, have a severe impact on human health, the global economy, biodiversity, and climate change. Detection of floating targets is a crucial step in the monitoring and management of floating debris, and it is of great significance in marine management, environmental protection, resource development, and rescue operations.

[0003] However, detecting small floating objects on water has always been a challenging research topic in the field of object detection. Firstly, small objects typically occupy a limited number of pixels in an image, leading to low feature resolution, information loss, and a lack of feature representation ability. Secondly, compared to ordinary small objects, detecting small floating objects on water presents two main challenges: Firstly, floating objects in water environments exhibit significant variations in shape, color, and texture, often appearing distorted and blurred. Learning appropriate features from these distorted characteristics is inherently difficult, especially since targets in complex water environments are also affected by factors such as lighting changes, water ripples, and water reflection. Secondly, large-scale datasets for detecting small floating objects on water are scarce and often widely distributed but difficult to obtain. This lack of datasets results in insufficient generalization ability of trained models in practical applications, making it difficult to handle complex situations in water environments. Therefore, algorithms and data for detecting small floating objects in complex environments are relatively insufficient, causing it to lag far behind object detection technologies in other application scenarios.

[0004] Currently, there are many improved methods for small object detection based on deep learning, mainly including optimizing AnchorBox, optimizing the backbone network, introducing attention mechanisms and other networks into the model, fusing features from feature maps of different scales, and optimizing IoU and loss functions. Surface object detection is used to locate floating objects on the water surface and determine the category of various targets (ships, buoys, etc.), which is of great significance for USV environmental awareness and safe navigation.

[0005] Traditional surface target detection methods mainly include background subtraction, frame difference, and Hough transform. These methods require hand-crafted models to extract specific features. For example, Szpak et al. used real-time approximations based on background subtraction and level set curve evolution to calibrate moving ships in the ocean and track ships moving in dynamic backgrounds. Wu et al. proposed an adaptive Gaussian mixture model to simulate complex backgrounds, improving the accuracy of buoy detection. These methods can achieve high detection accuracy in static background scenes. However, they are easily affected by background factors such as sunlight changes, rain, and surface fluctuations, leading to poor performance. Xu et al. constructed a 3D feature detector based on the multiple polarization characteristics of radar signals, improving the accuracy of small surface target detection. Although traditional algorithms have made great progress, they require hand-designed features and do not allow end-to-end detection. Furthermore, because hand-designed features are not robust enough to different inputs, they cannot be applied to complex surface target detection tasks. However, for small floating targets, more detailed feature extraction and higher resolution are usually required, which increases the computational cost of the model and thus affects real-time performance. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art by providing an ERT-DETR model for detecting small targets on the water surface. This model is of great significance for improving the detection capability of small targets in complex environments and can be extended to fields such as marine management, environmental monitoring, and ship tracking.

[0007] According to a first aspect of the present invention, an ERT-DETR model for detecting small targets on the water surface is provided, comprising: a backbone network, a dynamic feature pyramid network, and a micro attention module;

[0008] The input to the dynamic feature pyramid network is the features of the last four stages of the backbone network;

[0009] The dynamic feature pyramid network includes an AIFI module, a RepBi-PAN network, and a spatial depth transformation convolution module. The AIFI module uses a self-attention mechanism to process high-level semantic features and extracts key features from the deepest layer of the backbone network. The RepBi-PAN network fuses the key features with the low-level features extracted from the backbone network to obtain fused features. The SPD-Conv module performs multi-scale fusion convolution on the fused features to obtain convolutional features.

[0010] The micro-attention module includes: a depthwise separable convolution module, a stationary convolution module, and a channel attention module; the depthwise separable convolution module segments the features along the channel direction and then performs depthwise convolution on each feature to obtain depthwise convolutional features; the pointwise convolution module integrates the depthwise convolutional features and extracts key features; the channel attention module is used to enhance the key features to obtain a weighted feature map.

[0011] Based on the above technical solution, the present invention can also be improved as follows.

[0012] Optionally, the SPD-Conv module includes: a spatial-to-depth layer and a non-stretch convolutional layer;

[0013] The space-to-depth layer is used to transform the feature maps within the entire CNN into intermediate feature maps. The transformation process includes: downsampling the feature maps to obtain individual sub-feature maps, and connecting the individual sub-feature maps along the channel dimension to obtain the intermediate feature maps.

[0014] Optionally, the process by which the depthwise separable convolution module obtains the depthwise convolutional features includes:

[0015] The input feature X is divided into four equal parts along the channel direction to obtain features X corresponding to different directions. i =Split(X), where i represents dividing X into the i-th equal parts along the channel, and Split represents the splitting operation;

[0016] Different depthwise convolution kernels of different feature map branches are used to process the segmented feature X. i =Split(X) performs depthwise convolution operations separately: X i =DWConv2d(X) i ), where DWconv2d represents the depthwise convolution operation, X1 uses a 3×3 depthwise convolution; X2 and X3 use 1×11 and 11×1 depthwise convolutions respectively; while X4 remains unchanged as the identity branch.

[0017] Optionally, the process by which the pointwise convolution module obtains the key features includes:

[0018] The features obtained after depthwise convolution are concatenated, and then processed by activation function, normalization layer and residual connection;

[0019] Pointwise convolution is used to integrate channel information to extract the key features.

[0020] Optionally, the process by which the channel attention module obtains the weighted feature map from the key features includes:

[0021] The global features of the key features are obtained by using an adaptive average pooling layer, and the global features are channel compressed and recalibrated by a two-layer fully connected network. The weight of each channel is obtained by using the Sigmoid function to obtain the output features.

[0022] The weighted feature map is obtained by performing element-wise multiplication between the output features processed by the exponential function and the original input features of the micro-attention module.

[0023] Optionally, the loss function of the ERT-DETR model is an IoU-based bounding box regression loss function that applies the Inner-IoU loss.

[0024] Optionally, the Inner-IoU loss is:

[0025]

[0026] in, b l b r b t b t inter and union are intermediate variables. Let x be the coordinates of the ground truth bounding box and the center point inside the ground truth bounding box. c ,y c ) represents the center point of the anchor frame and the inner anchor frame, w gt and h gt , where are the width and height of the ground truth bounding box, and w and h are the width and height of the anchor box, respectively; the variable ratio is the scale factor; IoU inner This is the Inner-IoU loss.

[0027] Optionally, the IoU-based bounding box regression loss function includes: Iou, SIou, and CIou;

[0028] L Inner-IoU =1-IoU inner ;

[0029] L Inner-SIoU =L SIoU +IoU-IoU inner ;

[0030] L Inner-CIoU =L CIoU +IoU-IoU inner ;

[0031] Among them, IoU inner For Inner-IoU loss, IoU, L SIoU and L CIoULet L represent the loss functions for Iou, SIou, and CIou. Inner-IoU L Inner-SIoU and L Inner-CIoU They represent IoU respectively inner Loss functions applied to Iou, CIou, and SIou, IoU inner This is the Inner-IoU loss.

[0032] The present invention provides an ERT-DETR model for detecting small targets on the water surface, the beneficial effects of which include:

[0033] 1. To address the challenge of detecting small floating targets on water, this invention proposes an ERT-DETR model. This model, while possessing both real-time performance and end-to-end capabilities, can also improve the detection accuracy and efficiency of small floating targets in complex environments.

[0034] 2. In order to improve the feature representation capability of the network model, this embodiment of the invention proposes a novel neck network DFPNet, which uses the AIFI module to extract the deepest features of the backbone network and introduces the SPDConv module to enhance the local detail representation of the feature map.

[0035] 3. To enable the model to focus on learning the salient features of small floating targets, this embodiment of the invention introduces a novel attention module and uses the Inner-IoU auxiliary bounding box loss function to accelerate the convergence speed of the model. Attached Figure Description

[0036] Figure 1 This is a structural diagram of an embodiment of the ERT-DETR model for detecting small targets on the water surface provided by the present invention;

[0037] Figure 2 A structural diagram of an embodiment of a dynamic feature pyramid network provided by the present invention;

[0038] Figure 3 A structural diagram of an embodiment of the SPD-Conv module provided by the present invention;

[0039] Figure 4 A structural diagram of an embodiment of a microscopic attention module provided by the present invention;

[0040] Figure 5 A schematic diagram illustrating the Inner-IoU calculation process provided by this invention;

[0041] Figure 6(a) is a schematic diagram of the quantity distribution of different categories of targets in the water-floating small target detection dataset provided by the present invention;

[0042] Figure 6(b) is a schematic diagram of the distribution of the target center point in the data set for detecting small floating targets provided by the present invention.

[0043] Figure 6(c) is a schematic diagram of the distribution of object size in the water-floating small target detection dataset provided by the present invention;

[0044] Figure 7 A comparative diagram of Inner-CIoU and Inner-SIoU under different ratios provided in this embodiment of the invention;

[0045] Figure 8 This is a schematic diagram of the loss curve of the ablation experiment provided in an embodiment of the present invention;

[0046] Figure 9(a) is a schematic diagram of an embodiment of input image;

[0047] Figure 9(b) shows the attention effect of the input image after detection by the RT-DETR model;

[0048] Figure 9(c) shows the attention effect of the input image detected by the ERT-DETR model provided in this embodiment of the invention;

[0049] Figure 10(a) is a schematic diagram of another embodiment of the input image;

[0050] Figure 10(b) shows the detection results of the input image after passing through the RT-DETR model;

[0051] Figure 10(c) shows the detection effect of the input image using the ERT-DETR model provided in this embodiment of the invention. Detailed Implementation

[0052] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0053] Figure 1 The structure of an embodiment of the ERT-DETR model for detecting small targets on the water surface provided by the present invention is as follows: Figure 1 As shown, the ERT-DETR (Enhancing Real-Time Detection Transformer) model includes: a backbone network, a Dynamic Feature Pyramid Network (DFPNet), and a Micro-Attention Module (MAM).

[0054] The input to the dynamic feature pyramid network is the features of the last four stages of the backbone network; the backbone network is a backbone network based on the DETR model.

[0055] The dynamic feature pyramid network includes: the AIFI module, the RepBi-PAN network, and the Spatial Depth Transformation Convolution module. The AIFI module uses a self-attention mechanism to process high-level semantic features and extracts key features from the deepest layer of the backbone network. The RepBi-PAN network fuses the key features with the low-level features extracted from the backbone network to obtain fused features. The SPD-Conv module (Spatially Pooled Dilated Convolution) performs multi-scale fusion convolution on the fused features to obtain convolutional features.

[0056] The micro-attention module includes: a depthwise separable convolution module, a stationary convolution module, and a channel attention module. The depthwise separable convolution module segments the features along the channel direction and then performs depthwise convolution on each feature to obtain the depthwise convolutional features. The pointwise convolution module integrates the depthwise convolutional features and extracts the key features. The channel attention (SE, Squeeze and Excitation) module is used to enhance the key features to obtain a weighted feature map.

[0057] This invention provides an ERT-DETR model for detecting small objects on the water surface. First, it utilizes features from the last four stages of the backbone network {C2, C3, C4, C5} as input to the neck network. An efficient hybrid encoder converts multi-scale features into image feature sequences through Intra-Scale Feature Interaction (AIFI). DFPNet incorporates the idea of ​​a bidirectional feature pyramid network, strengthening connections between features through an enhanced path aggregation network, effectively shortening the information transmission path between high-level and low-level features. Then, SPD-Conv reduces information loss and improves the accuracy of feature extraction. A micro-attention module enhances the capture of small-sized features. Finally, Inner-IoU uses auxiliary bounding boxes to calculate IoU loss to accelerate sample convergence and improve bounding box localization accuracy. The decoder generates bounding boxes and confidence scores by iteratively optimizing the target query. Comparative and ablation experiments were conducted using the FloW-Img floating object dataset as the experimental subject. The results show that the ERT-DETR model achieves an AP of 92.6% and FPS of 118.84, which are 4.3% and 19.8% higher than the benchmark RT-DETR, respectively. The ERT-DETR model outperforms the state-of-the-art YOLO and DETR series detectors in both accuracy and real-time performance for small target detection in dynamic water surface environments. This model is of great significance for improving small target detection capabilities in complex environments and can be extended to fields such as marine management, environmental monitoring, and vessel tracking.

[0058] Example 1

[0059] Embodiment 1 provided by this invention is an embodiment of the ERT-DETR model for detecting small targets on the water surface provided by this invention, combined with... Figure 1 It can be seen that the ERT-DETR model implementation includes: a backbone network, a dynamic feature pyramid network, and a micro attention module.

[0060] The input to the dynamic feature pyramid network is the features of the last four stages of the backbone network.

[0061] Dynamic Feature Pyramid Network aims to improve the detection performance of small targets, such as Figure 2 The diagram shown is a structural schematic of an embodiment of a dynamic feature pyramid network provided by the present invention. Figure 1 and Figure 2 It is known that it includes: AIFI module, RepBi-PAN network and spatial depth transformation convolution module.

[0062] The AIFI module utilizes a self-attention mechanism to process high-level semantic features, extracting key features from the deepest layers of the backbone network, which helps to capture the connections between conceptual entities in an image.

[0063] The RepBi-PAN network fuses key features with low-level features extracted from the backbone network to obtain fused features, thereby strengthening the connections between features and effectively shortening the information transmission path.

[0064] The SPD-Conv (Spatially Pooled Dilated Convolution) module performs multi-scale fusion convolution on the fused features to obtain convolutional features, which improves the accuracy of information capture and target localization for small targets.

[0065] like Figure 3 The diagram shown is a structural diagram of an embodiment of the SPD-Conv module provided by the present invention. In one possible embodiment, it is combined with... Figure 3 As can be seen, the SPD-Conv module includes: spatial-to-depth (SPD) layers and non-stretch convolution (Conv) layers.

[0066] The spatial-to-depth layer is used to transform the feature maps within the entire CNN into intermediate feature maps. The transformation process includes: downsampling the feature maps to obtain individual sub-feature maps, and connecting the individual sub-feature maps along the channel dimension to obtain intermediate feature maps.

[0067] The Spatial-to-Depth (SPD) layer extends the original image transformation technique to downsampling feature maps within and throughout the CNN. Given any (original) feature map S(H,W,C), where H and W are the height and width of the feature map, respectively, and C is the number of channels, after downsampling, these sub-feature maps are concatenated along the channel dimension to obtain feature map S'. SPD transforms feature map S(H,W,C) into an intermediate feature map.

[0068] To improve the detection performance of small targets in complex backgrounds and enhance the capture and representation of these details, a micro-attention module (MAM) is proposed. The MAM structure diagram is shown below. Figure 4 As shown, MAM can leverage its depthwise separable convolutions to enhance the capture of small-sized features, then further integrate channel information through pointwise convolutions to extract key features, and finally enhance the channel features of the input feature map through the SE module.

[0069] Combination Figure 4 It can be seen that the micro-attention module includes: a depthwise separable convolution module, a stationary convolution module, and a channel attention module. The depthwise separable convolution module segments the features along the channel direction and then performs depthwise convolution to obtain the depthwise convolution features. The pointwise convolution module integrates the depthwise convolution features and extracts the key features. The channel attention (SE, Squeeze and Excitation) module is used to enhance the key features to obtain a weighted feature map.

[0070] In one possible embodiment, the process by which the depthwise separable convolution module obtains the depthwise convolutional features includes:

[0071] Suppose the input features of MAM are X∈R B×C×H×W Where B is the batch size, the input feature X is divided into four equal parts along the channel direction to obtain features X corresponding to different directions. i =Split(X), where i represents dividing X into the i-th equal parts along the channel, and Split represents the splitting operation.

[0072] The segmented features are processed through an Inception branch structure, which incorporates a multi-scale depthwise separable convolution (DWConv) module. This module applies depthwise convolution (DWConv) operations to the segmented feature maps, using different sized depthwise convolution kernels for different feature map branches, and applying these operations to the segmented feature X. i =Split(X) performs depthwise convolution operations separately: X i =DWConv2d(X) i), where DWconv2d represents the depthwise convolution operation, X1 uses a 3×3 depthwise convolution; X2 and X3 use 1×11 and 11×1 depthwise convolutions respectively; while X4 remains unchanged as an identity branch to preserve the integrity of the original features.

[0073] In one possible implementation, the process by which the pointwise convolution module obtains key features includes:

[0074] The features obtained after depthwise convolution are concatenated, then processed by activation function, normalization layer, and residual connection: X′=σ(Concat(X i ))+X, where σ(·) is a nonlinear activation function.

[0075] Pointwise convolution is used to integrate channel information to extract key features: X″=σ(Conv2d 1×1 (X′))+X.

[0076] In one possible embodiment, the process by which the channel attention module obtains a weighted feature map of key features includes:

[0077] The global features of key features are obtained by using an adaptive average pooling layer. The global features are then compressed and recalibrated by a two-layer fully connected network. The weight of each channel is obtained by using the Sigmoid function, resulting in the output features.

[0078] The output features processed by the exponential function are multiplied element-wise with the original input features of the micro-attention module to obtain a weighted feature map.

[0079] Specifically, the channel attention module then uses an adaptive average pooling layer (AvgPool) to reduce the spatial dimension of X” to 1×1; then, through a two-layer fully connected network (FC), the first layer reduces the feature C′ to C′ / / reduction, and the second layer increases the feature from C′ / / reduction back to C′.

[0080] SE=f(σ(f(AvgPool(X″)))).

[0081] Where f(·) is a fully connected network. The SE module uses an adaptive average pooling layer to obtain global features, then performs channel compression and recalibration through two fully connected layers, and finally uses the sigmoid function to obtain the weights of each channel. The output of the SE module is then:

[0082] SE out =Sigmoid(SE).

[0083] The output of the SE module is processed by an exponential function, and then multiplied element-wise with the original input feature X to obtain the final weighted feature map.

[0084] Y = X exp(SE) out ).

[0085] Here, ⊙ represents the Hadamard product (element-wise multiplication), and exp(·) is an exponential function used to convert the output of Sigmoid(·) into multiplication weights.

[0086] The IoU loss function has wide applications in computer vision tasks. In bounding box regression, it not only evaluates the quality of the regression state but also accelerates convergence through gradient propagation by calculating the regression loss. In one possible implementation, the ERT-DETR model uses the Inner-IoU loss applied to the IoU-based bounding box regression loss function. By choosing Inner-IoU as the loss function, auxiliary bounding boxes are used to accelerate sample convergence for small object detection. Figure 5 The diagram shown illustrates the calculation process of Inner-IoU provided by this invention. Figure 5 As shown, the GT box and anchor box are represented by B. gt And B. The Inner-IoU loss is:

[0087]

[0088] in, b l b r b t b t inter and union are intermediate variables. Let x be the coordinates of the ground truth bounding box and the center point inside the ground truth bounding box. c ,y c ) represents the center point of the anchor frame and the inner anchor frame, w gt and h gt , where are the width and height of the ground truth bounding box, and w and h are the width and height of the anchor box, respectively; the variable ratio is the scale factor, typically ranging from [0.5, 1.5]; IoU inner The Inner-IoU loss is represented by the width and height of the InnerGT box. and The width and height of the Inner anchor frame are represented by w. inner and h inner .

[0089] Inner-IoU loss inherits some characteristics from IoU loss while also possessing its own unique features. Like IoU loss, the value range of Inner-IoU loss is [0,1]. Because the auxiliary bounding box differs from the actual bounding box only in scale, and the trend of IoU value changes during regression is consistent with the trend of IoU value change of the actual bounding box, it can reflect the quality of the regression results of the actual bounding box.

[0090] In one possible implementation, the IoU-based bounding box regression loss function includes: Iou (Intersection of Union), SIou (SCYLLA-Intersection of Union), and CIou (Complete-Intersection of Union).

[0091] L Inner-IoU =1-IoU inner .

[0092] L Inner-SIoU =L SIoU +IoU-IoU inner .

[0093] L Inner-CIoU =L CIoU +IoU-IoU inner .

[0094] Among them, IoU inner For Inner-IoU loss, IoU, L SIoU and L CIoU Let L represent the loss functions for Iou, SIou, and CIou. Inner-IoU L Inner-SIoU and L Inner-CIoU They represent IoU respectively inner Loss functions applied to Iou, CIou, and SIou, IoU inner This is the Inner-IoU loss.

[0095] Example 2

[0096] Embodiment 2 of this invention is an application embodiment of the ERT-DETR model for detecting small objects on the water surface provided by this invention. In this application embodiment, experiments were conducted on the FloW-Img dataset to verify the actual performance of ERT-DETR. FloW-Img is mainly composed of video clips recorded by unmanned vessels, and 2000 images containing small floating targets were selected as the dataset. The training set, validation set, and test set were divided in a 6:2:2 ratio. Figure 6 shows the visualization results of the analysis of the small floating target detection dataset. Figure 6(a) shows the distribution of the number of targets of different categories in the dataset. The horizontal and vertical axes of Figures 6(b) and 6(c) are normalized results, where 6(b) represents the distribution of the target center point position. The darker the color, the more concentrated the center point of the target box, which is beneficial to improving the model's detection effect on occluded targets. Figure 6(c) shows the distribution of object size. It can be seen from this figure that small objects account for a high proportion.

[0097] The detector was trained using the AdamW optimizer with learning_rate = 0.0001, weight_decay = 0.0001, warmup_epochs = 3.0, warmup_momentum = 0.8, and warmup_bias_lr = 0.1. Data augmentation included random {HSV augmentation, translation, scaling, and flipping} operations, with the following parameter settings: hsv_h = 0.015, hsv_s = 0.7, hsv_v = 0.4, translate = 0.1, scale = 0.5, and fliplr = 0.5. Both the end-to-end and real-time end-to-end algorithms used the following scaling factor: [depth, width, max_channels] = [1.00, 1.00, 1024]. The early stopping epochs (Epoch = 200, patience = 50) where the end-to-end algorithm showed no significant performance improvement were used, while the real-time end-to-end algorithm used Epoch = 300 and patience = 100. Inner-IoU for both the comparative and ablation experiments was calculated using Inner-CIoU, with a ratio of 1.15. The decoder training strategy and hyperparameters largely followed RT-DETR.

[0098] In this embodiment of the invention, RT-DETR is used as the detector. The Inner-CIoU method and the Inner-SIoU method are compared. When ratio = 1, it degenerates into the IoU method. To demonstrate the superiority of the ERT-DETR model for detecting small objects on the water surface provided by this invention, it was trained on the training set for 200 epochs, and comparative experiments were conducted on the test set. The experimental results are shown in Fig. 6. It can be seen that after using the Inner-IoU method, both the detection effect and real-time performance are improved, with AP and mAP increasing by more than 0.9% and FPS increasing by more than 5. The best performing method is the Inner-CIoU method with ratio = 0.80, with FPS / AP / mAP improvements of 8.85% / 1.3% / 1.8%, respectively.

[0099] In the FloW-Img dataset, small targets (area < 32×32) account for 56.81%, potentially indicating a large number of low IoU samples. Experimental results show that the Inner-CIoU method outperforms eInner-SIoU overall. Therefore, this embodiment uses Inner-CIoU with a ratio > 1 for experiments, and the results are shown in Table 1. When the ratio = 1.15, the Inner-CIoU method performs best, outperforming both Inner-CIoU and Inner-SIoU when the ratio < 1, achieving FPS / AP / mAP of 109.11 / 90.9% / 49.9%, respectively, representing improvements of 11.58% / 2.4% / 2.3% compared to the CIoU method.

[0100] Table 1. Performance of various CIoU losses (ratio>1, between 1.1 and 1.2)

[0101]

[0102] Figure 7 This is a comparative diagram of Inner-CIoU and Inner-SIoU under different ratios (ratio < 1, between 0.7 and 0.8) provided for embodiments of the present invention. Figure 7 As can be seen, by setting ratio < 1, an auxiliary bounding box smaller than the actual bounding box is generated, which still achieves a certain performance improvement in high IoU samples, indicating that the method has strong generalization ability. As shown in Table 1, when ratio > 1, generating a larger auxiliary bounding box accelerates the convergence of low IoU samples, enabling more real-time and accurate detection and tracking of floating objects on the water surface.

[0103] The ERT-DETR model was compared with the state-of-the-art real-time end-to-end object detector RT-DETR, as well as real-time and end-to-end object detectors.

[0104] The performance of ERTSWSO-DETR was compared with that of a real-time detector while adhering to the end-to-end setup of RT-DETR. This embodiment of the invention compares ERT-DETR with YOLOv5, YOLOv6v3.0 (hereinafter referred to as YOLOv6), and YOLOv8 in Table 2. Compared to YOLOv5-L / YOLOv6-L / YOLOv8-L, ERTSWSO-DETR improves AP by 4.4% / 7.0% / 2.8%, FPS by 103.2% / 114.3% / 55.7%, and reduces the number of parameters by 25.5% / 64.2% / 9.5%. Compared to YOLOv6-L and YOLOv8-L, ERTSWSO-DETR reduces computation by 65.4% / 17.8%. Compared to YOLOv5-X / YOLOv6-X / YOLOv8-X, ERTSWSO-DETR improves AP by 3.5% / 15.2% / 2.4%, FPS by 238.7% / 174.5% / 104.4%, reduces the number of parameters by 59.1% / 77.0% / 41.5%, and reduces computation by 44.9% / 350.3% / 90.0%.

[0105] Compared to end-to-end detectors, for a fair comparison, the embodiments of this invention are only compared with end-to-end detectors using the same backbone. As shown in Table 2, the ERTSWSO-DETR outperforms state-of-the-art end-to-end detectors. Compared to the DINO-Deformable-DETR-R50, the ERTSWSO-DETR-R50 significantly improves AP by 4.8% (92.2% vs 87.4%) and speed by 11 times (118.84 FPS vs 10.75 FPS).

[0106] As shown in Table 2, under the same backbone, ERTSWSO-DETR outperforms RT-DETR, the first known real-time end-to-end detector, in both real-time performance and accuracy. Compared to RT-DETR, ERTSWSO-DETR significantly improves AP by 4.3% (92.6% vs 88.3%) and speed by 19.8% (118.84 FPS vs 99.18 FPS).

[0107] Table 2: Comparison of results with the SOTA detector (The input size of the real-time and real-time end-to-end detectors is 640, and the input size of the end-to-end detectors is (800, 1333)).

[0108]

[0109]

[0110] Experimental results show that the ERT-DETR model provided by this invention achieves an AP of 92.6% and FPS of 118.84, outperforming the RT-DETR and YOLO series detectors in both speed and accuracy. Furthermore, the ERT-DETR-R50 proposed in this embodiment achieves an AP of 92.2% and FPS of 113.26, outperforming state-of-the-art end-to-end detectors with the same backbone in both speed and accuracy.

[0111] To verify the effectiveness of the proposed innovation, this invention designed seven sets of ablation experiments for comparison with a baseline, aiming to evaluate the improvement in detector performance. This invention visualizes the loss curves of the experimental process, such as... Figure 8 As shown, when only the Inner-CIoU improvement method is retained, the number of epochs required to reduce the loss to 2 / 5 of the initial value is only 1 / 7 of the baseline. The model basically converges after 163 epochs, while the baseline requires 200 epochs. With the improvement strategy proposed in this embodiment, the convergence speed of the detector in the experimental group is significantly faster than the baseline, and the final loss value is reduced to about 1 / 4 of the initial value, while the baseline is only 1 / 3. It is worth noting that the speed of loss reduction directly reflects the training efficiency and performance of the model.

[0112] The quantitative evaluation results of the ablation experiments are shown in Table 3. Compared with the baseline, the improved method proposed in this embodiment of the invention achieves a significant performance improvement. After introducing the MAM and Inner-CIoU methods of this embodiment into RT-DETR, the number of parameters and computational cost remain almost unchanged, but the FPS increases by 16.2% / 10.0% and the AP increases by 3.6% / 2.6%, respectively. After introducing the DFPNet method of this embodiment of the invention, although the number of parameters and computational cost increases, the FPS / AP increases by 13.5% / 3.6%, making the increase in the number of parameters and computational cost worthwhile. Through continuous improvement, the detector of this embodiment of the invention finally achieves an FPS of 118.84 and an AP of 92.6, which are improvements of 19.8% / 4.3% compared with the baseline, respectively. The ablation experiment results show that in complex river environments, the improved strategy of this embodiment of the invention can detect small floating targets on the water surface more accurately and in real time.

[0113] Table 3. Main results of the ablation experiment

[0114]

[0115] Finally, to clearly demonstrate the improved model performance, the embodiments of this invention are compared with the benchmark RT-DETR method in terms of both attention performance and actual detection performance. Figures 9(a), (b), and (c) show the input image, the attention performance of the input image detected by the RT-DETR model, and the attention performance of the input image detected by the ERT-DETR model provided in this embodiment, respectively. The spherical yellow blocks in the figures represent the areas of attention. As can be seen from Figures 9(a), (b), and (c), the ERT-DETR detector provided in this embodiment can still better focus on the target area even in environments with low light, multiple targets occluding each other, and targets at long distances with few features. As can be seen from Figures 10(a), (b), and (c), compared with the benchmark algorithm, ERT-DETR achieves a significant improvement in detection performance in complex environments, with fewer false positives and false negatives. Therefore, the performance of the ERT-DETR detector proposed in this embodiment surpasses the current state-of-the-art detectors.

[0116] This invention provides an ERT-DETR model for detecting small targets on the water surface. This ERT-DETR model strikes a balance between performance and speed, employing a lightweight model structure or optimizing the model inference process to improve speed. Its beneficial effects include:

[0117] 1. To address the challenge of detecting small floating targets on water, this invention proposes an ERT-DETR model. This model, while possessing both real-time performance and end-to-end capabilities, can also improve the detection accuracy and efficiency of small floating targets in complex environments.

[0118] 2. In order to improve the feature representation capability of the network model, this embodiment of the invention proposes a novel neck network DFPNet, which uses the AIFI module to extract the deepest features of the backbone network and introduces the SPDConv module to enhance the local detail representation of the feature map.

[0119] 3. To enable the model to focus on learning the salient features of small floating targets, this embodiment of the invention introduces a novel attention module and uses the Inner-IoU auxiliary bounding box loss function to accelerate the convergence speed of the model.

[0120] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0126] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An ERT-DETR model for water surface small target detection, characterized in that, The ERT-DETR model comprises: a backbone network, a dynamic feature pyramid network, and a micro-attention module; An input of the dynamic feature pyramid network is features of the last four stages of the backbone network; The dynamic feature pyramid network comprises: an AIFI module, a RepBi-PAN network, and an SPD-Conv module; the AIFI module processes high-level semantic features by using a self-attention mechanism to extract key features from the deepest layer of the backbone network; the RepBi-PAN network fuses the key features with low-level features extracted from the backbone network to obtain fused features; and the SPD-Conv module performs multi-scale fusion convolution on the fused features to obtain convolution features; The micro-attention module comprises: a depth separable convolution module, a point-wise convolution module, and a channel attention module; the depth separable convolution module performs depth convolution on features after being divided along the channel direction to obtain depth-convoluted features; the point-wise convolution module extracts key features by integrating channels of each depth-convoluted feature; and the channel attention module is used to strengthen channel features of the key features to obtain a weighted feature map; The process of the channel attention module for obtaining the weighted feature map from the key features comprises: An adaptive average pooling layer is used to obtain global features of the key features, two fully connected networks are used to compress and recalibrate channels of the global features, a Sigmoid function is used to obtain weights of each channel, and an output feature is obtained; The output feature processed by an exponential function is multiplied by the original input feature of the micro-attention module element by element to obtain the weighted feature map.

2. The ERT-DETR model of claim 1, wherein, The SPD-Conv module comprises: a space-to-depth layer and a non-step convolution layer; The space-to-depth layer is used to transform a feature map inside the entire CNN into an intermediate feature map, and the transformation process comprises: down-sampling the feature map to obtain sub-feature maps, and connecting the sub-feature maps along the channel dimension to obtain the intermediate feature map.

3. The ERT-DETR model of claim 1, wherein, The process of the depth separable convolution module for obtaining the depth-convoluted features comprises: The input feature X is divided into four equal parts along the channel direction to obtain features corresponding to different directions. , This indicates that X is divided into the first part along the channel. Divide equally; Split represents the splitting operation. different sizes of deep convolution kernels are used for different feature map branches, and deep convolution operations are performed on the segmented features respectively, wherein DWconv2d represents a deep convolution operation, using a deep convolution of 3x3; and using deep convolutions of 1x11 and 11x1, respectively; and remains unchanged as an identity branch.​ 4. The ERT-DETR model of claim 1, wherein, The process of the point-wise convolution module for obtaining the key features comprises: Features obtained after depth convolution are spliced, and then processed by an activation function, a normalization layer, and residual connection; Point-wise convolution is used to integrate channel information to extract the key features.

5. The ERT-DETR model of claim 1, wherein, The loss function of the ERT-DETR model is to apply an Inner-IoU loss to an IoU-based bounding box regression loss function.

6. The ERT-DETR model of claim 5, wherein, The Inner-IoU loss is: ； where, , , , , , , , , and are intermediate variables, are the center point coordinates inside the GT box and box, are the center points of the anchor box and the inner anchor box, and are the width and height of the GT box, respectively, and h are the width and height of the anchor box, respectively; and the variable ratio is a scale factor; is the Inner-IoU loss.

7. The ERT-DETR model of claim 5, wherein, The IoU-based bounding box regression loss function comprises: Iou, SIou, and CIou; ； ； ； wherein IoU, and denote the loss functions for Iou, CIou and SIou, respectively, and denote the loss functions for Iou, CIou and SIou, respectively, denote the loss functions for Iou, CIou and SIou, respectively, is the Inner-IoU loss.

Images