A target detection method from the perspective of an unmanned aerial vehicle (UAV)

Abstract

This invention discloses a target detection method from the perspective of a UAV. By reducing the number of layers in the backbone network and increasing the size of the feature map to be detected, a feature layer containing rich semantic information of small targets is added. This reduces the network depth while allowing the model to focus more on the detection of small targets. The RepNCSPELAN4 structure, based on the GELAN design concept, replaces the C2f structure in the original model. This structure allows it to support multiple types of computational blocks, making it better adaptable to different computational and hardware requirements and effectively improving computational efficiency and performance. A deformable attention mechanism (DAT) is introduced in the Spatial Pyramid Pooling layer (SPPF), enabling the network to dynamically select sampling points and allowing the model to focus more on the most important regions in the current task. This invention improves the performance of small target detection without increasing the complexity of the algorithm, thus achieving a better balance between model complexity and accuracy, resulting in more accurate detection results and fewer model parameters.

Description

Technical Field

[0001] This application belongs to the field of target detection technology, and in particular relates to a target detection method from the perspective of an unmanned aerial vehicle (UAV). Background Technology

[0002] With the development of drone technology, target detection using deep learning algorithms has been widely applied in related fields. However, target detection from a drone's perspective faces challenges such as densely distributed targets, small pixel size, and imbalanced sample sizes across different categories in the dataset. Furthermore, the hardware limitations of drones restrict model size, resulting in detection results that are difficult to meet the needs of practical applications.

[0003] Currently, deep learning-based object detection algorithms are mainly divided into two categories: two-stage region proposal algorithms, such as R-CNN, Fast R-CNN, and Faster R-CNN; and one-stage regression algorithms, such as SSD, the YOLO series, and RetinaNet. While two-stage algorithms offer high detection accuracy, they are slow and unsuitable for detection tasks from a UAV perspective. Single-stage algorithms, on the other hand, are fast but have relatively low accuracy, particularly in small object detection, where further optimization is needed.

[0004] The YOLO series of models performs excellently in various detection tasks, especially in target detection from the perspective of drones. The YOLOv8 target detection model has five versions, from smallest to largest: v8n, v8s, v8m, v8l, and v8x. It mainly consists of three parts: the backbone, the neck, and the head. The backbone is mainly responsible for feature extraction, including the Conv convolutional module, the C2f module, and the SPPF module, which evolved from the SPP module in the YOLOv5 architecture. The neck is mainly responsible for feature fusion, using the C2f module and a feature pyramid structure combining PAN and FPN. The head is gradually developed and integrated from the YOLO-Head of the YOLO series, including three detection heads with feature maps of different sizes for detecting targets of different sizes. However, since the basic YOLOv8 model is designed based on general natural scenes, directly applying the original model to targets captured by drones leads to a significant loss of feature information during the sampling process, and the detection effect for small targets is poor, making it difficult to achieve ideal results. Summary of the Invention

[0005] The purpose of this application is to provide a target detection method from the perspective of a UAV, which solves the difficulties of target detection from the perspective of a UAV, such as dense distribution of the target to be detected, small pixel size, and unbalanced number of samples of each category in the corresponding dataset. At the same time, due to the hardware limitations of the UAV itself, it is necessary to ensure that the network model can maintain high detection accuracy with a small number of parameters.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] A target detection method from the perspective of an unmanned aerial vehicle (UAV) includes:

[0008] Build and train an improved YOLOv8 object detection network model, including an improved backbone network, neck network, and detection head;

[0009] The image to be identified is input into the improved backbone network to extract multi-scale features. The improved backbone network removes the 7th and 8th layers of the original backbone network and introduces a deformable attention mechanism in the spatial pyramid pooling layer.

[0010] The extracted multi-scale features are processed through a neck network and a detection head to obtain the recognition results.

[0011] Furthermore, the multi-scale features extracted by the improved backbone network are the output features of the second layer, the fourth layer, and the spatial pyramid pooling layer after introducing a deformable attention mechanism.

[0012] Furthermore, the multi-scale features extracted by the improved backbone network have sizes of 160×160×128 pixels, 80×80×256 pixels, and 40×40×512 pixels, respectively.

[0013] Furthermore, the C2f modules in the backbone network and neck network are replaced with RepNCSPELAN4 modules.

[0014] Furthermore, during training, the improved YOLOv8 object detection network model uses ShapeIoU as the boundary regression loss function, and replaces the Cross-Union Ratio (IoU) in the ShapeIoU loss function with Inner-IoU loss.

[0015] This application proposes a target detection method from the perspective of a UAV. Considering the hardware limitations of the UAV itself, it uses the YOLOv8s network structure, which is smaller in scale and has higher detection accuracy, as a benchmark. By reducing the number of layers in the backbone network and increasing the size of the feature map to be detected, a feature layer containing rich semantic information of small targets is added. This reduces the network depth while allowing the model to focus more on the detection of small targets. The RepNCSPELAN4 structure with the GELAN design idea replaces the C2f structure in the original model. This structure allows it to support multiple types of computational blocks, making it more adaptable to different computational and hardware requirements and effectively improving computational efficiency and performance. A deformable attention mechanism (DAT) is introduced in the Spatial Pyramid Pooling layer (SPPF). SPPF_DAT is proposed to enable the network to dynamically select sampling points, allowing the model to focus more on the most important regions in the current task. The ShapeIoU function is improved by using the idea of ​​inner-IoU loss function. The inner-ShapeIoU loss function replaces the original loss function. By focusing on the shape and size of the bounding box itself, the loss is calculated, improving the algorithm's localization and classification performance of small targets in complex environments. This application improves the performance of small target detection without increasing the complexity of the algorithm, thus achieving a better balance between model complexity and accuracy; the detection results are more accurate, the number of model parameters is less, and there are fewer missed detections and false detections. Attached Figure Description

[0016] Figure 1 This is a flowchart of the target detection method from the perspective of an unmanned aerial vehicle (UAV) in this application.

[0017] Figure 2 This is a diagram of the YOLOv8 network structure.

[0018] Figure 3 This is a structural diagram of the improved YOLOv8 target detection network model in the embodiments of this application.

[0019] Figure 4 This is a comparison chart of the performance metrics of the object detection network model proposed in this application and YOLOv8. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] Example 1, as Figure 1 As shown, a target detection method from the perspective of an unmanned aerial vehicle (UAV) is provided, including:

[0022] Step S1: Construct and train an improved YOLOv8 object detection network model, including an improved backbone network, neck network, and detection head.

[0023] The object detection network model constructed in this application is an improvement upon YOLOv8. The structure of YOLOv8 is shown below. Figure 2 The system comprises a backbone network, a neck network, and a head. The backbone network has 10 layers, numbered 0 to 9. Layers 0, 1, 3, 5, and 7 are convolutional layers, layers 2, 4, 6, and 8 are C2f modules, and layer 9 is a Spatial Pyramid Pooling (SPPF) layer. The neck network is responsible for multi-scale feature fusion, enhancing feature representation capabilities by fusing feature maps from different stages of the backbone network. The head is responsible for the final object detection.

[0024] It should be noted that YOLOv8 is a relatively mature technology in this field. Its backbone network and bottleneck network each include multiple convolutional and C2f modules, which will not be elaborated here.

[0025] This embodiment improves the original backbone network of YOLOv8, and the improved backbone network is used for multi-scale feature extraction.

[0026] Step S2: Input the image to be recognized into the improved backbone network and extract multi-scale features. The improved backbone network removes the 7th and 8th layers of the original backbone network and introduces a deformable attention mechanism in the spatial pyramid pooling layer.

[0027] This embodiment improves upon the original YOLOv8 backbone network by reducing the number of layers in the backbone network and increasing the size of the feature map to be detected. It adds feature layers containing rich semantic information of small targets, thereby reducing the network depth while enabling the model to focus more on the detection of small targets.

[0028] Specifically, the input image for the YOLOv8 model is 640×640×3 pixels, and the resulting three feature maps are 80×80×256 pixels, 40×40×512 pixels, and 20×20×1024 pixels, respectively. Its network structure is as follows: Figure 2 As shown, these correspond to the outputs of layers 4, 6, and 9, respectively. When the targets to be detected are mainly small, compressing the minimum feature map size to 20×20×1024 pixels may lead to a significant loss of feature information for small targets. Furthermore, after deep convolution, the features of small targets are easily submerged, thus affecting the detection accuracy.

[0029] Therefore, in the first step of this embodiment, the convolution (Conv) and C2f layers of Stage Layer 4 in the backbone network are removed to reduce the impact of deep convolution, that is, the original 7th and 8th layers of the backbone network are removed.

[0030] Then, a deformable attention mechanism (DAT) was introduced into the Spatial Pyramid Pooling Layer (SPPF). The introduction of the deformable attention mechanism in the Spatial Pyramid Pooling Layer (SPPF_DAT) enables the network to dynamically select sampling points, allowing the model to focus more on the most important regions in the current task.

[0031] When the image reaches the SPPF layer, the result obtained after processing the original SPPF module through multiple pooling layers is first input into a 4×4 non-overlapping convolution with a stride of 4 in the DAT module. Then, a normalization layer compresses the input image size to 1 / 4 of its original size. In the last two stages of the hierarchical feature pyramid, continuous local attention and deformable attention modules are used. Information is aggregated locally through window-based local attention, and then deformable attention blocks are used to model the global relationship between local enhancement labels. This alternative design of attention blocks with local and global receptive fields can collect more effective features without incurring excessive memory and computational costs.

[0032] In this embodiment, the feature map output by the SPPF_DAT module is 40×40×512 pixels.

[0033] In a specific embodiment, the multi-scale features extracted by the backbone network in this embodiment are the output features of the second layer, the fourth layer, and the spatial pyramid pooling layer after introducing the deformable attention mechanism. The output feature map sizes are 160×160×128 pixels, 80×80×256 pixels, and 40×40×512 pixels, respectively.

[0034] It is worth mentioning that, due to the reduction in the number of layers in the backbone network, in order to match the output length and width of the feature maps corresponding to the Concat connection layers in the neck network, the number of layers in the output feature maps has been changed from the original 4, 6, and 9 to 2, 4, and 7.

[0035] This application enlarges the multi-scale feature size to 160×160×128 pixels, 80×80×256 pixels and 40×40×512 pixels. Through this improvement, the model can better preserve the feature information of small targets.

[0036] Step S3: The extracted multi-scale features are processed through the neck network and the detection head to obtain the recognition result.

[0037] After multi-scale features are extracted from the backbone network, they are input into the bottleneck network, and then the final detection result is output through the detection head.

[0038] Example 2 provides a target detection method from the perspective of an unmanned aerial vehicle (UAV). Unlike Example 1, the C2f module in the backbone network and neck network of this example is replaced with the RepNCSPELAN4 module.

[0039] RepNCSPELAN4 is a feature extraction and fusion module in YOLOv9. It uses the GELAN design concept, supports multiple types of computation blocks, and can better adapt to different computing and hardware requirements, effectively improving computing efficiency and performance.

[0040] GELAN integrates the design concepts of CSPNet and ELAN, borrows the segmentation and recombination concepts of CSPNet, and introduces the hierarchical convolution processing method of ELAN in each part, thus reducing the amount of computation and memory usage, and improving inference speed and accuracy.

[0041] Example 3 provides a target detection method from the perspective of an unmanned aerial vehicle (UAV). Unlike Example 1, the inner-ShapeIoU is used instead of the original loss function in the detection head of the network model to improve the algorithm's localization and classification performance of small targets in complex environments.

[0042] The purpose of bounding box regression is to adjust the detection window output by the detector to more closely approximate the real detection window. Since its inception, the intersection over union (IoU) has become the main standard for evaluating the loss of predicted bounding boxes in the field of object detection, and its calculation formula is shown in Equation (1):

[0043] (1)

[0044] in This represents the calculation result of the prediction box. Representing the calculated result of the ground truth bounding boxes, IoU measures the degree of matching by calculating the ratio of the intersection to the union of the predicted and ground truth bounding boxes. IoU-based bounding box regression loss functions have undergone continuous iteration and development, resulting in various versions such as DIoU, GIoU, CIoU, EIoU, and SIoU. These improved loss functions accelerate convergence by introducing new loss terms, but they do not take into account the inherent limitations of IoU itself.

[0045] The Inner-IoU algorithm, based on auxiliary bounding boxes, proposes using auxiliary bounding boxes to calculate IoU, thereby improving the generalization ability of the model. The specific calculation process is shown in equations (2) to (6), where... The center point of the anchor point and the inner anchor point is represented by w and h, respectively. The size of the auxiliary border is adjusted by the scale factor ratio.

[0046] (2)

[0047] (3)

[0048] Equations (2) and (3) can be used to transform the center point of the detection box, thereby obtaining the corner vertices of the auxiliary detection box. Simultaneously, the predicted and ground truth boxes output by the model are transformed accordingly, using... and This represents the calculated results of the ground truth bounding box and the predicted bounding box. Use min and max to get the minimum and maximum values ​​in parentheses.

[0049] (4)

[0050] (5)

[0051] (6)

[0052] Therefore, as shown in equations (4) to (6), inner-IoU actually calculates the IoU between the auxiliary bounding boxes. When ratio∈[0.5,1.5] and When the auxiliary bounding box is smaller than the actual bounding box, the effective range of the regression is smaller than the IoU loss, but the absolute value of the obtained gradient is greater than the gradient of the IoU loss, thus accelerating the convergence of high IoU samples. However, when... When the auxiliary bounding box is larger than the actual bounding box, this expands the effective range of the regression and helps with the regression of low IoU samples.

[0053] ShapeIoU takes into account the influence of the shape and size of the bounding box itself, and samples itself in the bounding box regression, thereby making the bounding box regression more accurate. Its specific calculation process is shown in equations (7) to (10).

[0054] (7)

[0055] (8)

[0056] (9)

[0057] (10)

[0058] in , These represent the weight coefficients in the vertical and horizontal directions, respectively, and their values ​​are related to the values ​​of the ground truth bounding box. `scale` is a scaling factor. This indicates the cost of the shape.

[0059] This embodiment modifies ShapeIoU using the concept of inner-IoU, effectively improving the detection performance by replacing the IoU calculation part. The inner-ShapeIoU calculation formula is shown in equation (11):

[0060] (11)

[0061] in, This indicates the inner-ShapeIoU loss function used in this embodiment for bounding box regression in the detection head. This represents the Inner-IoU loss function.

[0062] like Figure 3 As shown, the object detection network model constructed in this application, abbreviated as DAT-YOLO, introduces a deformable attention mechanism (DAT) on top of YOLOv8. Specifically, it reduces the number of layers in the backbone network and increases the size of the feature map to be detected, adding feature layers containing rich semantic information of small objects. This reduces the network depth while allowing the model to focus more on the detection of small objects. The RepNCSPELAN4 structure, based on the GELAN design concept, replaces the C2f structure in the original model. This structure allows it to support multiple types of computational blocks, making it better adaptable to different computational and hardware requirements and effectively improving computational efficiency and performance. The deformable attention mechanism (DAT) is introduced in the Spatial Pyramid Pooling (SPPF) layer. SPPF_DAT is proposed to enable the network to dynamically select sampling points, allowing the model to focus more on the most important regions in the current task. The ShapeIoU function is improved by using the idea of ​​inner-IoU loss function. The original loss function is replaced by inner-ShapeIoU. The loss is calculated by focusing on the shape and size of the bounding box itself, which improves the algorithm's localization and classification performance of small targets in complex environments.

[0063] When training the constructed object detection network model, an image dataset captured by a drone was collected and preprocessed, including converting the labeled file format and applying data augmentation methods. The images in the training set were resized to 640×640 and used as input to the network model, while the Mosaic data augmentation method was employed to expand the dataset. The training epochs were set to 600, and the patience to 30 to ensure that the model training would automatically stop if no better results were achieved within 30 epochs.

[0064] After training, the corresponding weight model and training results are saved. To evaluate the performance of the improved model, P (precision), R (recall), mAP@50 (multi-class average precision at a 50% IoU threshold), and mAP@50:95 (multi-class average precision between 50% and 95% IoU thresholds) are used as evaluation metrics. The results of training YOLOv8 and DAT-YOLO networks are compared, and the results are as follows: Figure 4 As shown, DAT-YOLO has superior overall performance, with a smoother mAP curve.

[0065] Finally, based on models trained with YOLOv8, PVswin-YOLOv8, GAM-YOLO, and the DAT-YOLO network proposed in this application, object detection was performed on the same image. Under different lighting conditions, object quantity, and size, the model proposed in this application can achieve good detection results. Several typical scenarios are listed below.

[0066] The first scenario: a scene with sparse small targets under good lighting.

[0067] Experimental results show that GAM-YOLO and DAT-YOLO with added small target detection layers identified distant trucks, while the original model and PVswin-YOLOV8s failed to do so.

[0068] The second scenario: a multi-category target scenario with ample lighting.

[0069] The original model misidentified roadblocks as pedestrians, only recognizing pedestrians in the vicinity; while DAT-YOLO did not make any misidentifications, not only recognizing pedestrians but also detecting bicycles under the pedestrians.

[0070] The third scenario: a scene of densely packed small targets at night.

[0071] Compared to the original model, PVswin-YOLOv8s, and GAM-YOLO, DAT-YOLO can also detect more small targets more accurately, such as small vehicles and bicycles in the distance, effectively reducing the chances of missed detections and false detections.

[0072] The results show that, compared with other mainstream models, this application is more suitable for UAV target detection tasks.

[0073] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A target detection method from the perspective of an unmanned aerial vehicle (UAV), characterized in that, The target detection method from the perspective of the UAV includes: Build and train an improved YOLOv8 object detection network model, including an improved backbone network, neck network, and detection head; The image to be identified is input into the improved backbone network to extract multi-scale features. The improved backbone network removes the 7th and 8th layers of the original backbone network and introduces a deformable attention mechanism in the spatial pyramid pooling layer. The extracted multi-scale features are processed through a neck network and a detection head to obtain the recognition results.

2. The target detection method from the perspective of an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The multi-scale features extracted by the improved backbone network are the output features of the second layer, the fourth layer, and the spatial pyramid pooling layer after introducing a deformable attention mechanism.

3. The target detection method from the perspective of an unmanned aerial vehicle (UAV) according to claim 2, characterized in that, The improved backbone network extracts multi-scale features with sizes of 160×160×128 pixels, 80×80×256 pixels, and 40×40×512 pixels, respectively.

4. The target detection method from the perspective of an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The C2f modules in the backbone network and neck network are replaced with RepNCSPELAN4 modules.

5. The target detection method from the perspective of an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, During training, the improved YOLOv8 object detection network model uses ShapeIoU as the boundary regression loss function and replaces the Cross-Union Ratio (IoU) in the ShapeIoU loss function with Inner-IoU loss.

Images