A lightweight unmanned aerial vehicle detection method for embedded devices

Abstract

The application provides a lightweight unmanned aerial vehicle detection method for embedded devices, which uses already deployed resident cameras to collect images, discards the upper computer, improves the model structure of yolov5s, directly deploys the model to the embedded device, and detects whether the target area has an unmanned aerial vehicle in real time, thereby strengthening the supervision of small-scale unmanned aerial vehicles. The method comprises the following steps: Step 1: collecting unmanned aerial vehicle images; Step 2: adding bird negative samples; Step 3: labeling the processed data set, since only unmanned aerial vehicles are detected, only the unmanned aerial vehicle area is labeled during labeling; Step 4: designing an improved yolov5s model structure, which reduces the parameter quantity and proposes a feature fusion mechanism; Step 5: training the model to obtain the trained model weight; Step 6: in the application stage, the trained model weight is loaded into the network and deployed to the embedded device to detect small-scale unmanned aerial vehicles in the video in real time.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, and in particular to a lightweight drone detection method for embedded devices. Background Technology

[0002] With the continuous development of deep learning and the integration of artificial intelligence and smart monitoring, monitoring work is gradually moving towards intelligent integration. Smart monitoring mainly uses artificial intelligence to assist in monitoring, which can automatically extract illegal targets in the video and alert the operators.

[0003] With the increasing proportion of civilian drones year by year, safety accidents and privacy leaks caused by drones are frequent, making the regulation of drones an urgent matter. However, existing target detection models on the market generally have a large number of parameters, making them difficult to deploy directly on embedded devices with limited computing resources. If a method of lower-level machines acquiring images and upper-level machines processing them is used, the computational requirements of the upper-level machines are too high, making it unsuitable for deployment as a detection cluster. Summary of the Invention

[0004] To address the technical problems in the background art, this invention provides a lightweight drone detection method for embedded devices. It utilizes images collected by already deployed residential cameras, eliminates the need for a host computer, improves the YOLOv5s model structure, and directly deploys the model into embedded devices to detect whether drones exist in the target area in real time, thereby strengthening the supervision of small-scale drones.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A lightweight drone detection method for embedded devices is proposed. The method utilizes images collected by already deployed residential cameras, deploys a trained model into the embedded device, and detects in real time whether there are drones in the target area, thereby strengthening the supervision of small-scale drones.

[0007] Includes the following steps:

[0008] Step 1: Acquire drone images;

[0009] Step 2: Add negative bird samples; In order to enhance the network's ability to distinguish between birds and drones, add negative bird samples to the collected drone images, segment the birds from the original images, scale them down to the drone scale, and randomly add them to the drone images.

[0010] Step 3: Label the processed dataset. Since only drones are detected, only the drone area is labeled.

[0011] Step 4: Design an improved YOLOv5s model structure, which reduces the number of parameters and proposes a feature fusion mechanism; reducing the number of parameters makes it usable by embedded devices, and the proposed feature fusion mechanism improves the accuracy after lightweighting without introducing new parameters;

[0012] Step 5: Train the model and obtain the weights of the trained model;

[0013] Step 6: In the application phase, the trained model weights are loaded into the network and deployed to an embedded device to detect small-scale drones in the video in real time.

[0014] Furthermore, in Step 4, the improved YOLOv5s model structure includes:

[0015] 1) Considering that small-scale drones occupy a small proportion in the image and their features are not obvious, by removing shallow features with small receptive fields and using only deep features with receptive fields close to the target size, the output features can contain more effective target features while having a larger receptive field.

[0016] 2) By modifying the structure of the bottleneck layer, the degree of deep feature fusion can be further improved;

[0017] 3) Meanwhile, in order to significantly reduce the number of model parameters and computational load so that it can be used by most embedded devices, the kernel size of the first convolution is adjusted to 3x3 and the number of convolution kernels of all convolution modules and C3 module is reduced, so that the number of model parameters is reduced to 693K, which is about 20 times less than the original 14.1M.

[0018] Furthermore, in Step 4, in order to propose a novel feature fusion structure to improve the accuracy after lightweighting without introducing new parameters, the proposed feature fusion structure includes the following:

[0019] 1) First, considering that shallow features contain rich target detail features, downsampling and fusing with deep features can increase the content of detail features in deep features, which helps the network to locate small-scale targets;

[0020] 2) Then, considering that deep features contain relatively rich semantic features, upsampling and fusing with shallow features helps to increase the semantic features in shallow features, allowing the network to learn the target features more fully.

[0021] 3) Finally, by using steps 1) and 2) multiple times and in multiple layers, bidirectional feature fusion is achieved, and existing features are reused to generate higher-level semantics, enabling the network to take into account feature information at different scales to locate the target position.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] 1) The present invention provides a lightweight drone detection method for embedded devices, which utilizes pre-deployed residential cameras to collect images, eliminates the need for a host computer, and directly deploys the training model into the embedded device to detect whether there are drones in the target area in real time. It does not require the computing power of the host computer and is suitable for deployment as a detection cluster.

[0024] 2) The model structure of YOLOv5S was designed and improved, which significantly reduced the number of model parameters and computational load, making it available to most embedded devices.

[0025] 3) Design an improved model structure for YOLOv5s. Without introducing new parameters, a novel feature fusion structure is proposed to improve the accuracy after lightweighting. After its addition, the accuracy is reduced by only 0.8% compared to the original model. Attached Figure Description

[0026] Figure 1 This is a model structural diagram of the lightweight structure of the present invention;

[0027] Figure 2 This is a structural diagram of module C3 of the present invention;

[0028] Figure 3 This invention presents a novel feature fusion structure diagram.

[0029] Figure 4 This is the final model diagram of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0031] This invention uses deep learning methods to detect whether drones exist in no-fly zones, thereby improving the early warning capabilities of smart monitoring.

[0032] This invention provides a lightweight drone detection method for embedded devices. It utilizes images collected by existing residential cameras, deploys a trained model into the embedded device, and detects in real time whether there are drones in the target area, thereby strengthening the supervision of small-scale drones.

[0033] Includes the following steps:

[0034] Step 1: Acquire drone images;

[0035] Step 2: Add negative bird samples; In order to enhance the network's ability to distinguish between birds and drones, add negative bird samples to the collected drone images, segment the birds from the original images, scale them down to the drone scale, and randomly add them to the drone images.

[0036] Step 3: Label the processed dataset. Since only drones are detected, only the drone area is labeled.

[0037] Step 4: Design an improved YOLOv5s model structure, which reduces the number of parameters and proposes a feature fusion mechanism; reducing the number of parameters makes it usable by embedded devices, and the proposed feature fusion mechanism improves the accuracy after lightweighting without introducing new parameters;

[0038] Step 5: Train the model and obtain the weights of the trained model;

[0039] Step 6: In the application phase, the trained model weights are loaded into the network and deployed to an embedded device to detect small-scale drones in the video in real time.

[0040] Furthermore, in Step 4, the improved YOLOv5s model structure includes:

[0041] 1) Considering that small-scale drones occupy a small proportion in the image and their features are not obvious, by removing shallow features with small receptive fields and using only deep features with receptive fields close to the target size, the output features can contain more effective target features while having a larger receptive field.

[0042] 2) By modifying the structure of the bottleneck layer, the degree of deep feature fusion can be further improved;

[0043] 3) Meanwhile, in order to significantly reduce the number of model parameters and computational load so that it can be used by most embedded devices, the kernel size of the first convolution is adjusted to 3x3 and the number of convolution kernels of all convolution modules and C3 module is reduced, so that the number of model parameters is reduced to 693K, which is about 20 times less than the original 14.1M.

[0044] Furthermore, in Step 4, in order to propose a novel feature fusion structure to improve the accuracy after lightweighting without introducing new parameters, the proposed feature fusion structure includes the following:

[0045] 1) First, considering that shallow features contain rich target detail features, simple downsampling and fusion with deep features can increase the content of detail features in deep features, which helps the network to locate small-scale targets;

[0046] 2) Then, considering that deep features contain relatively rich semantic features, upsampling and fusing with shallow features helps to increase the semantic features in shallow features, allowing the network to learn the target features more fully.

[0047] 3) Finally, by using steps 1) and 2) multiple times and in multiple layers, bidirectional feature fusion is achieved, and existing features are reused to generate higher-level semantics, enabling the network to take into account feature information at different scales to locate the target position.

[0048] The specific implementation process is as follows:

[0049] 1) Training phase:

[0050] First, a large number of small-scale drone images are collected, and negative samples of flying birds are added. Second, drone regions in the images are labeled. Then, the improved scheme is used to train the model weights. Finally, the trained weights are added to the network for application.

[0051] This invention primarily provides a lightweight drone detection scheme, which consists of two steps. First, considering the objective factors of low-altitude drones having relatively small or absolute target sizes, easily blending into the background, and relatively large model sizes, the network structure is optimized, such as... Figure 1 As shown; then, a novel parameterless feature fusion structure is proposed, such as Figure 3 As shown, and an overall detection network is added, such as Figure 4 As shown.

[0052] This invention Figure 1 The following are some of the rules shown in the diagram:

[0053] The number on the right side of each module represents the number of output channels for that module.

[0054] NxC3: This represents the number of times the C3 module is reused.

[0055] The structure of C3 is as follows: Figure 2 As shown.

[0056] Figure 1 The diagram shows an improved YOLOv5s architecture for small-scale UAV detection proposed in this invention. This invention considers that small-scale UAV targets have a small effective proportion, and while shallow networks contain many small target features, their receptive field is insufficient, easily misclassifying local background information as targets. Therefore, from the perspective of improving accuracy, this invention removes shallow features, simplifies the bottleneck layer structure, and fuses the output features of C3_4 and convolution_7 to improve the utilization of deep features; from the perspective of model lightweighting, it significantly reduces the number of convolutional kernels in the detection network.

[0057] Figure 3 The diagram illustrates a novel feature fusion method proposed in this invention. Features 1, 2, and 3 are feature maps from the backbone network, with feature 1 having the largest feature map size and feature 3 the smallest. First, shallow features are downsampled and fused with deep features using max pooling, achieving downward feature fusion. This promotes the fusion of shallow and deep features, increasing the content of detailed features in deep features and helping deep convolutions extract more effective target features. Then, deep features are upsampled and fused with shallow features using linear interpolation, achieving upward feature fusion. This allows the shallow network to consider the semantic information of deep features during feature extraction. Furthermore, by repeatedly using downward and upward feature fusion across multiple layers, the utilization rate of existing features is improved, promoting the fusion of multi-scale features and helping the network to consider multi-scale information of the target for localization. This multi-layered, multi-stage, bidirectional feature fusion method significantly improves the model's detection performance, reducing the model size by 20 times while only decreasing the detection accuracy by 0.8% compared to the original network.

[0058] Figure 4 The image shown is the final model of this invention, in which the FRM module is... Figure 3 The structure shown further fuses three adjacent features with a size difference of 2 times to obtain features with richer detailed and semantic features, which are then used in the subsequent feature fusion stage.

[0059] 2) Application phase:

[0060] Input an image or video, deploy the trained model to an embedded device, perform detection using the model on the embedded device, and output an image or video with a bounding box marked on it. It can also record video frames of drone targets for subsequent review.

[0061] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0066] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A lightweight UAV detection method for embedded devices, characterized in that, The method described above utilizes images collected by existing residential cameras, deploys the trained model to an embedded device, and detects in real time whether there are drones in the target area, thereby strengthening the supervision of small-scale drones. Includes the following steps: Step 1: Acquire drone images; Step 2: Add negative bird samples; In order to enhance the network's ability to distinguish between birds and drones, add negative bird samples to the collected drone images, segment the birds from the original images, scale them down to the drone scale, and randomly add them to the drone images. Step 3: Label the processed dataset. Since only drones are detected, only the drone area is labeled. Step 4: Design an improved YOLOv5s model structure, which reduces the number of parameters and proposes a feature fusion mechanism. The reduced number of parameters enables embedded devices to utilize them, and the proposed feature fusion mechanism improves the accuracy after lightweighting without introducing new parameters. Step 5: Train the model and obtain the weights of the trained model; Step 6: In the application phase, the trained model weights are loaded into the network and deployed to an embedded device to detect small-scale drones in the video in real time. In Step 4, the improved YOLOv5s model structure includes: 1) Considering that small-scale drones occupy a small proportion in the image and their features are not obvious, by removing shallow features with small receptive fields and using only deep features with receptive fields close to the target size, the output features can contain more effective target features while having a larger receptive field. 2) By modifying the structure of the bottleneck layer, the degree of deep feature fusion can be further improved; 3) Meanwhile, in order to significantly reduce the number of model parameters and computational load, so that most embedded devices can use it, the kernel size of the first convolution is adjusted to 3x3 and the number of convolution kernels of all convolution modules and C3 module is reduced, so that the number of model parameters is reduced to 693K. In Step 4, in order to improve the accuracy after lightweighting by proposing a novel feature fusion structure without introducing new parameters, the proposed feature fusion structure includes the following: 1) First, considering that shallow features contain rich target detail features, simple downsampling and fusion with deep features can increase the content of detail features in deep features, which helps the network to locate small-scale targets; 2) Then, considering that deep features contain relatively rich semantic features, upsampling and fusing with shallow features helps to increase the semantic features in shallow features, allowing the network to learn the target features more fully. 3) Finally, by using steps 1) and 2) multiple times and in multiple layers, bidirectional feature fusion is achieved, and existing features are reused to generate higher-level semantics, enabling the network to take into account feature information at different scales to locate the target position.

Images