Rotating target detection method and device based on UAV field of view

Abstract

The present invention provides a rotating target detection method and device based on the field of view of an unmanned aerial vehicle, and belongs to the field of target detection technology. The method uses a pre-trained target detection model to perform image segmentation on a high-resolution target image collected by a drone to obtain multiple sub-images, and sets an overlap area between each two sub-images for detecting the target located between the two sub-images; then, the sub-images are subjected to small-size target recognition and rotating target detection by the target detection model to obtain a small-size target prediction frame and a rotating target detection frame; finally, according to the small-size target prediction frame and the rotating target detection frame, the target position and target type of the small-size target and the rotating target are predicted in each sub-image and the overlap area. The present invention can more accurately identify and quickly frame small-size targets and rotating targets, solves the limitations of the traditional rectangular frame detection method, and improves the robustness and practicality of the detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a rotating target detection method and device based on the field of view of an unmanned aerial vehicle. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in monitoring, reconnaissance, agricultural monitoring, disaster assessment and other fields. In these application scenarios, the high-definition cameras carried by drones can capture high-resolution, large-size image data, providing a rich source of information for subsequent target detection and analysis.

[0003] However, traditional deep learning object detection networks face many challenges when processing such high-resolution images. On the one hand, high-resolution images require higher computing resources and storage space, which may lead to low detection efficiency; on the other hand, when drones image targets at high altitudes or long distances, the targets are usually presented in a bird's-eye view, and may be densely arranged, non-horizontal or vertical, which increases the difficulty of detection. In this case, traditional rectangular box detection methods often have problems with overlapping frames and not fitting the target contour, affecting the accuracy and practicality of detection. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a rotating target detection method and device based on the field of view of an unmanned aerial vehicle, so as to solve the problem that the traditional target detection method has low efficiency and low detection accuracy for high-resolution images.

[0005] The technical solution adopted by the present invention is: In a first aspect, the present invention provides a rotating target detection method based on the field of view of an unmanned aerial vehicle, comprising: The high-resolution target image collected by the UAV is segmented using a pre-trained target detection model to obtain multiple sub-images, and an overlap region is set between every two sub-images; the overlap region is used to detect the target between the two sub-images; wherein the target detection model includes an image segmentation layer, a small target detection layer, and a rotation target detection layer; Performing small-size target recognition and rotated target detection on the sub-image through a target detection model to obtain a small-size target prediction frame and a rotated target detection frame; According to the small-sized target prediction box and the rotated target detection box, the target position and target type of the small-sized target and the rotated target are predicted in each sub-image and overlap area.

[0006] Furthermore, the pre-training process of the target detection model includes: Load the preset training dataset and set the candidate box size for object detection based on the image size in the training dataset; Customize the Yolov5 model, add an image segmentation layer, a small target detection layer, and a rotation target detection layer to the Yolov5 model, and initialize the network structure of the customized Yolov5 model based on preset training parameters; The customized Yolov5 model is trained for target detection using a preset training data set, and the model is optimized using a loss function. After the loss function converges, the pre-trained target detection model is output.

[0007] Furthermore, the Yolov5 model is customized and modified, and an image segmentation layer, a small target detection layer, and a rotation target detection layer are added to the Yolov5 model, including: According to the image cutting logic, the image cutting code is added to the detect.py script of the Yolov5 model to form an image cutting layer; the Yolov5 model includes an input end, a backbone network, a neck structure and a head structure; the neck structure includes an SPP structure for pooling and a PAN structure for multi-level feature fusion; the head structure includes the final detection layer Pi; The neck structure of the Yolov5 model is optimized. A small target detection layer is added to the 2nd layer of the backbone network after 4 times downsampling. At the same time, the neck structure is modified to four-scale fusion, and feature fusion is performed with the feature maps generated at 8 times downsampling, 16 times downsampling, and 32 times downsampling through the PAN structure. Based on the rotation target detection algorithm, a rotation angle prediction channel represented by parameter θ is added to the prediction channel of the head structure of the Yolov5 model. The channel dimension of the final detection layer Pi in the head structure is changed from 3×(C+5) to 3×(C+6), so that each anchor frame in the final detection layer Pi has one more prediction channel for regressing parameter θ, forming a rotation target detection layer that can generate a rotated target detection frame, and obtaining a customized Yolov5 model; C represents the category of the target.

[0008] In a second aspect, the present invention provides a rotating target detection device based on the field of view of an unmanned aerial vehicle, comprising: An image cutting module is used to perform image cutting on a high-resolution target image collected by a UAV using a pre-trained target detection model to obtain multiple sub-images, and to set an overlap region between every two sub-images; the overlap region is used to detect a target located between two sub-images; wherein the target detection model includes an image cutting layer, a small target detection layer, and a rotation target detection layer; An object detection and recognition module is used to perform small-size object recognition and rotated object detection on the sub-image through an object detection model to obtain a small-size object prediction frame and a rotated object detection frame; The target marking module is used to predict the target position and target type of small-sized targets and rotated targets in each sub-image and overlap area based on the small-sized target prediction frame and the rotated target detection frame.

[0009] In summary, the beneficial effects of the present invention are as follows: The present invention provides a rotating target detection method based on the field of view of an unmanned aerial vehicle. The method uses a pre-trained target detection model to perform image segmentation on a high-resolution target image collected by the unmanned aerial vehicle to obtain multiple sub-images, and sets an overlap region between each two sub-images to detect the target located between the two sub-images, thereby effectively reducing the processing difficulty of the high-resolution image, and at the same time ensuring the image detection accuracy through the overlap region. Then, the small target detection layer and the rotating target detection layer of the target detection model are used to perform small-size target recognition and rotating target detection on the sub-image with the target detection label format, and the target position and target type of the small-size target and the rotating target are predicted in each sub-image and the overlap region based on the detected small-size target prediction frame and the rotating target detection frame, so that the small-size target and the rotating target can be more accurately recognized and quickly framed, thereby solving the limitations of the traditional rectangular frame detection method and improving the robustness and practicality of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0011] Figure 1 This is a flow chart of a rotating target detection method based on the field of view of an unmanned aerial vehicle according to the present invention; Figure 2 This is a flow chart of target detection model training of the present invention; Figure 3 The framework structure diagram of the Yolov5 algorithm customized and modified for the present invention; Figure 4 It is a schematic diagram of image segmentation of the present invention; Figure 5 This is a schematic diagram of the Yolov5 network structure; Figure 6 This is a detailed diagram of the Head structure of Yolov5; Figure 7 Schematic diagram of the final detection layer structure with the rotation angle prediction channel added. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0013] In view of the problems existing in the above background technology, the present invention aims to solve the following two key technical problems: 1. High-resolution image preprocessing and label format adaptation: How to effectively preprocess high-resolution and large-size images while maintaining high detection accuracy to adapt to the input requirements of the basic deep learning object detection network and generate label formats that meet the requirements of deep learning model training.

[0014] 2. Rotating target detection: Under the conditions of high altitude or long-distance imaging of UAVs, how to design a detection algorithm that can effectively identify and accurately frame rotating targets, avoid the overlap and mismatch problems of traditional rectangular frame detection methods, and improve the accuracy and robustness of target detection. The detailed implementation process of the present invention is shown in the following embodiments.

[0015] Example 1: Reference Figure 1 As shown, Figure 1 This is a flow chart of a rotating target detection method based on the field of view of an unmanned aerial vehicle according to the present invention. Figure 1 As shown, the method of the present invention comprises the following steps: The pre-trained target detection model is used to perform image segmentation on the high-resolution target image collected by the drone to obtain multiple sub-images, and an overlap region is set between every two sub-images. The overlap region is used to detect the target between the two sub-images. The target detection model includes an image segmentation layer, a small target detection layer, and a rotation target detection layer.

[0016] The target detection model is used to perform small-size target recognition and rotated target detection on the sub-image with the target detection label format to obtain the small-size target prediction box and the rotated target detection box.

[0017] According to the small-sized target prediction box and the rotated target detection box, the target position and target type of the small-sized target and the rotated target are predicted in each sub-image and overlap area.

[0018] Further, refer to Figure 2 As shown, the pre-training process of the target detection model in the embodiment of the present invention includes: Load the preset training dataset and set the candidate box size for object detection based on the image size in the training dataset; Customize the Yolov5 model, add an image segmentation layer, a small target detection layer, and a rotation target detection layer to the Yolov5 model, and initialize the network structure of the customized Yolov5 model based on preset training parameters; The customized Yolov5 model is trained for target detection using a preset training data set, and the model is optimized using a loss function. After the loss function converges, the pre-trained target detection model is output.

[0019] Specifically, the present invention proposes a rotating target detection method based on deep learning in the field of view of drones based on the open source model Yolov5. Yolov5 is selected as the basic detection model framework for customized modification. Due to its lightweight structure and high efficiency, it is particularly suitable for drone real-time detection tasks. Figure 3 As shown in the figure, the customized modified Yolov5 algorithm mainly consists of the following: The algorithm framework is mainly composed of three core components: image segmentation layer, small target detection layer and rotation target detection layer. This framework realizes the effective processing of high-resolution images by making targeted customized improvements to Yolov5. First, the image segmentation layer uses image segmentation technology to finely segment the input high-resolution image to obtain an image size that meets the input requirements of the Yolov5 model. Next, the small target detection layer is specially designed to enhance the model's ability to recognize small-sized targets, which is a challenge often faced by traditional target detection algorithms. Finally, the rotation target detection algorithm is integrated to construct the rotation target detection layer, which can generate a rotation rectangle that fits the actual contour of the target more closely, thereby providing more accurate target positioning information.

[0020] Furthermore, in an embodiment of the present invention, the Yolov5 model is customized and modified, and an image segmentation layer, a small target detection layer, and a rotation target detection layer are added to the Yolov5 model, including: According to the image cutting logic, the image cutting code is added to the detect.py script of the Yolov5 model to form an image cutting layer. The Yolov5 model includes an input end, a backbone network, a neck structure and a head structure. The neck structure includes an SPP structure for pooling and a PAN structure for multi-level feature fusion. The head structure contains the final detection layer Pi.

[0021] Specifically, when building the image cutting layer, download the open source Yolov5 open source code, mainly modify the detect.py script in the directory, and add the image cutting layer. First, use image segmentation technology to divide the high-resolution image into multiple smaller sub-images suitable for deep learning model processing. This process not only reduces the computational burden of a single processing, but also maintains the key features of the image. At the same time, the corresponding deep learning detection network label format is generated according to the segmentation results to ensure the accuracy and consistency of the training data.

[0022] Reference Figure 4 The image segmentation diagram shown in the figure is segmented according to the Yolov5 model input size of 640×640. Here, it is assumed that the image is segmented into 4 rows and 4 columns, that is, 16 small images. Some targets are located exactly in the middle of two small images and are just truncated, which may result in the failure to detect the target. In order to avoid this situation, an overlap area is set between the two small images. The overlapping area set here accounts for 20% of the total area (can be set as needed). The width of each overlap area is represented by mix_row and mix_col according to the rows and columns of the cut. When pre-training the target detection model, the label format required for yolov5 training (including the ID and coordinates of the small image, etc.) is generated while completing the image cutting.

[0023] The neck structure of the Yolov5 model is optimized. A small target detection layer is added to the second layer of the backbone network after 4 times downsampling. At the same time, the neck structure is modified to four-scale fusion, and feature fusion is performed with the feature maps generated at 8 times downsampling, 16 times downsampling and 32 times downsampling through the PAN structure.

[0024] Among them, Figure 5The Yolov5 network structure shown in Figure 1 is mainly composed of three parts: Backbone, Neck and Head. Conv means that it is composed of convolutional layers (Conv2d), batch normalization layers (BatchNorm2d) and activation functions (such as SiLU). The Conv module is responsible for extracting features from the image. The C3 module is a key component in Backbone. It consists of three Conv modules and one Bottleneck module. The C3 module further enhances the feature extraction capability of the network through residual connections and feature fusion. SPPF (Spatial Pyramid PoolingFast) represents the SPPF module, which is an improved version of SPP (Spatial Pyramid Pooling) and is used for multi-scale feature fusion. SPPF fuses feature maps of different scales into feature maps of unified scale through pooling operations of different scales, thereby improving the robustness of the network. Detect represents the Detect module: The Detect module in the Head consists of multiple 1x1 convolutional layers, which are used to increase or decrease the dimension of the feature map and generate the final detection results. The Detect module predicts multiple bounding boxes and corresponding category probabilities for each grid cell through the anchor mechanism. Upsample represents the upsampling module. Contact represents the concatenation operation.

[0025] By customizing the Yolov5 model, Figure 5 The small target detection module is introduced in the solid line frame to enhance the model's ability to recognize small targets. The Neck part is the added small target feature fusion part, and the solid line in the Head part is the small target detection head.

[0026] First, optimize the Neck part of Yolov5. Add a small target detection layer at the 4-fold downsampling of the second layer of Backbone, and modify the Neck part of the module to four-scale fusion. After the PAN structure (bottom-up downsampling), the feature maps generated at 8-fold downsampling, 16-fold downsampling, and 32-fold downsampling are fused. After feature fusion, when the input image is 640×640, the final output feature map size is 160×160, which has a smaller receptive field than feature maps of other scales, so as to detect more small targets.

[0027] Based on the rotation target detection algorithm, a rotation angle prediction channel represented by parameter θ is added to the prediction channel of the head structure of the Yolov5 model. The channel dimension of the final detection layer Pi in the head structure is changed from 3×(C+5) to 3×(C+6), so that each anchor frame in the final detection layer Pi has one more prediction channel for regressing parameter θ, forming a rotation target detection layer that can generate a rotated target detection frame, and obtaining a customized Yolov5 model; C represents the category of the target.

[0028] Specifically, based on the Yolov5 model, a rotation target detection algorithm is integrated. The algorithm generates a rotation rectangular box (Oriented Bounding Box, OBB) by predicting the center point, width, height and rotation angle of the target, so as to more accurately fit the actual shape of the target and solve the problems of overlap and mismatch in traditional rectangular box detection.

[0029] like Figure 6 As shown in the figure, the detailed diagram of the Head structure of Yolov5. The embodiment of the present invention adopts the Detect layer as the Head structure. After being processed by the backbone network Backbone and the neck structure Neck, the final detection layer Pi is obtained. The channel dimension of Pi is 3×(C+5), where 3 means that each grid has three anchor frames with different aspect ratios preset, and each anchor frame is responsible for predicting C categories and border parameter information ( x , y , w , h , P r ). x , y Respectively represent the horizontal and vertical coordinates of the center of the anchor box, w , h Represent the width and height of the anchor box respectively. The category prediction channel is ( C 0 , C 1 , ... , C n ) through the binary cross entropy loss function (BCELoss, corresponding to Figure 6 middle L BCE ) is learned. In addition, the HBB (i.e. axis-aligned bounding box) regression channel ( x , y , w , h ) is passed through CIoULoss (Complete Intersection over Union Loss, corresponding to Figure 6 InL CIoU ) to optimize, CIoULoss‌ is a loss function for target detection. P r Represents the foreground confidence of the prediction box. The network uses BCELoss to P r channel to adjust.

[0030] To achieve the prediction of the rotation target detection box, just add a rotation angle prediction channel to the Head structure of the Yolov5 model θ The rotating rectangle uses the five-parameter method to define the long side. θ ∈[-2 / π, 2 / π), through ( x , y , w , h , θ ) to represent OBB. Figure 7 As shown in the figure, the modified Pi detection layer channel dimension is 3×(C+6). This change gives each anchor box an additional prediction channel for the regression parameter θ. This parameter is optimized by the single parameter loss function Smooth L1 Loss. At the same time ( x,y,w,h ) channel still uses CIoU Loss as regression loss. CIoU Loss is used to optimize the position and shape of the long edge definition method HBB, while Smooth L1 Loss is specifically used to optimize the rotation angle of HBB. When predicting, the results of the two loss function optimizations are combined to obtain the final prediction result of the rotated object detection box OBB ( x , y , w , h , θ ).

[0031] The process of rotating target prediction based on the above rotating target detection layer is as follows: Input: Network channel parameters , ;in, Indicates the X coordinate of the center point of the prediction box. Indicates the Y coordinate of the center point of the prediction box. Indicates the width of the prediction box. Indicates the height of the prediction box. Indicates the rotation angle of the prediction box, the value range is [-2 / π, 2 / π). Relative position parameter of the rotated target, which can be expressed as an offset relative to a reference frame. Represents the predicted probability distribution of the different classes detected, usually expressed as a confidence score for each class.

[0032] Output: Prediction box ;in, Represents the confidence of the box, which indicates the confidence level of the target detected in the box. Represents the predicted class label, indicating which class the detected object belongs to.

[0033] (1) First decode the predicted value of the regression channel: Represents the grid index coordinates; Indicates the width and height of the preset anchor box; σ Represents the sigmoid function; , ; , ; ; ; in, Indicates the X coordinate of the center point of the decoded prediction box; Indicates the Y coordinate of the center point of the decoded prediction box; Indicates the width of the decoded prediction box; Indicates the height of the decoded prediction box; Represents the rotation angle of the decoded prediction box; (2) The classification probability is multiplied by the target confidence to obtain the confidence of each category: ; (3) Obtaining multi-category prediction results ,in: ,..., ; (4) Prediction results for multiple categories Perform confidence threshold screening.

[0034] (5) The prediction results of each of the n categories are subjected to non-maximum suppression (NMS) processing using this rotating box: ,..., ; (6) Summarize the prediction box information of all categories: .

[0035] The embodiment of the present invention customizes the Yolov5 model, adds small target and rotating target detection layers, and increases a small amount of computing time. Compared with the original Yolov5 model, the average precision mAP of all classes is improved. 50 The overall improvement is 6.46%, and the mAP of large, medium and small targets is 50 There were increases of 9.57%, 6.7% and 2.71% respectively.

[0036] Embodiment 2: In the embodiment of the present invention, based on the method in Embodiment 1, the embodiment of the present invention further provides a rotating target detection device based on the field of view of a drone, comprising: An image cutting module is used to perform image cutting on a high-resolution target image collected by a UAV using a pre-trained target detection model to obtain multiple sub-images, and to set an overlap region between every two sub-images; the overlap region is used to detect a target located between two sub-images; wherein the target detection model includes an image cutting layer, a small target detection layer, and a rotation target detection layer; An object detection and recognition module is used to perform small-size object recognition and rotated object detection on the sub-image through an object detection model to obtain a small-size object prediction frame and a rotated object detection frame; The target marking module is used to predict the target position and target type of small-sized targets and rotated targets in each sub-image and overlap area based on the small-sized target prediction frame and the rotated target detection frame.

[0037] The rotating target detection device based on the UAV field of view of the embodiment of the present invention has the following significant beneficial effects: 1. Improve detection efficiency and accuracy: Through image preprocessing and segmentation, the processing difficulty of high-resolution images is effectively reduced while ensuring detection accuracy. Combining Yolov5's lightweight model and rotating target detection algorithm, fast and accurate target detection is achieved.

[0038] 2. Enhanced adaptability: In view of the high-altitude or long-distance imaging characteristics of UAVs, the rotating target detection method proposed in the present invention can more accurately identify and frame rotating targets, solve the limitations of traditional rectangular frame detection methods, and improve the robustness and practicality of detection.

[0039] 3. Broad application prospects: This method is not only suitable for drone monitoring, reconnaissance and other fields, but can also be extended to smart agriculture, environmental protection, urban planning and other fields, providing a new solution for target detection in complex scenarios.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rotating target detection method based on drone vision, characterized in that: include: The high-resolution target image collected by the UAV is segmented using a pre-trained target detection model to obtain multiple sub-images, and an overlap region is set between every two sub-images; the overlap region is used to detect the target between the two sub-images; wherein the target detection model includes an image segmentation layer, a small target detection layer, and a rotation target detection layer; Performing small-size target recognition and rotated target detection on the sub-image through a target detection model to obtain a small-size target prediction frame and a rotated target detection frame; According to the small-sized target prediction box and the rotated target detection box, the target position and target type of the small-sized target and the rotated target are predicted in each sub-image and overlap area.

2. The rotating target detection method based on the UAV field of view according to claim 1 is characterized in that: The pre-training process of the target detection model includes: Load the preset training dataset and set the candidate box size for object detection based on the image size in the training dataset; Customize the Yolov5 model, add an image segmentation layer, a small target detection layer, and a rotation target detection layer to the Yolov5 model, and initialize the network structure of the customized Yolov5 model based on preset training parameters; The customized Yolov5 model is trained for target detection using a preset training data set, and the model is optimized using a loss function. After the loss function converges, the pre-trained target detection model is output.

3. The rotating target detection method based on the UAV field of view according to claim 2 is characterized in that: The Yolov5 model is customized and modified to add an image segmentation layer, a small target detection layer, and a rotation target detection layer to the Yolov5 model, including: According to the image cutting logic, the image cutting code is added to the detect.py script of the Yolov5 model to form an image cutting layer; the Yolov5 model includes an input end, a backbone network, a neck structure and a head structure; the neck structure includes an SPP structure for pooling and a PAN structure for multi-level feature fusion; the head structure includes the final detection layer Pi; The neck structure of the Yolov5 model is optimized. A small target detection layer is added to the 2nd layer of the backbone network after 4 times downsampling. At the same time, the neck structure is modified to four-scale fusion, and feature fusion is performed with the feature maps generated at 8 times downsampling, 16 times downsampling, and 32 times downsampling through the PAN structure. Based on the rotation target detection algorithm, a new parameter is added to the prediction channel of the head structure of the Yolov5 model. θ The rotation angle prediction channel represented by , changes the channel dimension of the final detection layer Pi in the head structure from 3×(C+5) to 3×(C+6), so that each anchor box in the final detection layer Pi has one more prediction channel for regression parameters θ , forming a rotating target detection layer that can generate a rotating target detection frame, and obtaining a customized Yolov5 model; C represents the category of the target.

4. A rotating target detection device based on drone vision, characterized in that: include: An image cutting module is used to perform image cutting on a high-resolution target image collected by a UAV using a pre-trained target detection model to obtain multiple sub-images, and to set an overlap region between every two sub-images; the overlap region is used to detect a target located between two sub-images; wherein the target detection model includes an image cutting layer, a small target detection layer, and a rotation target detection layer; An object detection and recognition module is used to perform small-size object recognition and rotated object detection on the sub-image through an object detection model to obtain a small-size object prediction frame and a rotated object detection frame; The target marking module is used to predict the target position and target type of small-sized targets and rotated targets in each sub-image and overlap area based on the small-sized target prediction frame and the rotated target detection frame.

Images