Rapid target detection method applied to weak computing power airborne chip

Abstract

The invention relates to a rapid target detection method applied to a weak computing power airborne chip, and belongs to the technical field of computer vision. According to the method, the original image is directly sent to the NPU module without the need of the resize operation, so that the time of the resize operation is saved. The structural layers of the target detection network designed by the invention are independent, no cross-layer link exists, and the calculation parallelism is good. A traditional target detection algorithm uses cross-layer connection in order to fuse multi-layer features, and calculation parallelism is poor. The target detection algorithm designed by the invention only has two stages, the target detection network only comprises a shallow-layer network and a middle-layer network, the network is relatively shallow, and the calculation amount is small. A traditional target detection algorithm adopts a deeper network, such as yolov5, including shallow layer, middle layer and deep layer networks, and the calculation amount is large.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a fast target detection method applied to a weak computing power airborne chip. Background Art

[0002] Target detection technology enables drones to autonomously identify targets. Target detection algorithms usually use neural networks as the main module and require large computing resources. However, the computing power of the drone's onboard computing platform is weak, and the drone application scenario requires real-time detection of targets. Therefore, there is a contradiction between weak computing power conditions and real-time requirements in the onboard target detection task. In particular, compared with imported chips such as NVIDIA, domestic chips have lower computing power. As shown in Table 1, for example, the domestic chip HiSilicon 3516AV100 has weaker performance than NVIDIA XAVIER NX.

[0003] Table 1 Comparison between domestic HiSilicon 3516AV300 and imported NVIDIA XAVIER NX

[0004] index HiSilicon 3516AV300 NVIDIA XAVIER NX Processor core 2 cores ARM7 6-core ARM8 Processor Cache 256KB BL2Cache 6MBL2+4MBL3Cache Neural network reasoning computing power 1TOPS 21TOPS Maximum supported memory 2G 16G Video encoding performance 3840×2160 resolution, 20fps 3840×2160 resolution, 60fps Power consumption 5W 15W Process 14nm 7nm Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is to provide a fast target detection method with good computational parallelism and small computational complexity that can be applied to airborne chips with weak computing power.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a fast target detection method applied to a weak computing power airborne chip, comprising the following steps:

[0009] (1) Obtaining the image of the optoelectronic pod, the image of the optoelectronic pod is 1280*720;

[0010] (2) Add 48 rows and 1280 columns of blank pixels below the optoelectronic pod image to change the optoelectronic pod image resolution to 1280*768. The length and width of this resolution can be divided by the total step length of the neural network. Then, the optoelectronic pod image is sent to the NPU module of the airborne chip for target detection to obtain the target detection result.

[0011] (3) Based on the target detection results, low-confidence borders are filtered out, and high-confidence borders are taken out from the NPU module to the memory of the onboard chip;

[0012] (4) Draw the target border in the electro-optical pod image;

[0013] (5) The optoelectronic pod image is compressed according to the H264 standard and sent to the ground station using the RTSP protocol, while the detection result of the current frame image is saved.

[0014] Preferably, the target detection algorithm implemented by the target detection network for target detection in the NPU module designed in step (2) includes two stages in total. The total step size of the backbone network in the first stage is 16. The input image with a resolution of 1280*768 is passed through the backbone network of the first stage to obtain a feature map with a resolution of 80*64, and then passed through the encoder and decoder of the first stage to obtain the detection result of the first stage; the step size of the backbone network in the second stage is 2, and the feature map with a resolution of 80*64 is passed through the backbone network of the second stage to obtain a feature map with a resolution of 40*32, and then passed through the encoder and decoder of the second stage to obtain the detection result of the second stage, and then the detection results of the two stages are fused by the non-maximum suppression algorithm to obtain the final target detection result.

[0015] Preferably, in the target detection network used for target detection in the NPU module designed in step (2), a convolution layer with a step size of 2 is added to the backbone network of the first stage, and the image with a resolution of 1280*768 passes through the convolution layer, and the resolution becomes 640*384.

[0016] Preferably, the confidence threshold is used as a preset value when border filtering is performed in step (3).

[0017] The invention also provides a rapid target detection system for implementing the method.

[0018] The present invention also provides a UAV flight experiment platform implemented based on the system.

[0019] (III) Beneficial effects

[0020] The method of the present invention mainly has the following characteristics:

[0021] 1. This method does not require resizing operation and directly sends the original image to the NPU module, saving the time of resizing operation.

[0022] 2. The structure of the target detection network designed by the present invention is independent between layers, without cross-layer links, and has good computational parallelism. In order to fuse multi-layer features, traditional target detection algorithms use cross-layer connections (such as yolov5), which has poor computational parallelism.

[0023] 3. The target detection algorithm designed by the present invention has only two stages, and the target detection network only includes shallow and middle layers of the network. The network is shallow and the amount of calculation is small. The traditional target detection algorithm uses a deeper network, such as Yolov5, which includes shallow, middle and deep networks and has a large amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the neural network structure designed in the method of the present invention;

[0025] Figure 2 This is a schematic diagram of the traditional yolov5 neural network structure;

[0026] Figure 3 Detection accuracy curves corresponding to different scale targets detected by a specific receptive field network;

[0027] Figure 4 This is an example of the target detection effect of the method of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0029] The present invention proposes a fast target detection method for a weak computing power airborne chip, and the execution process of the method is as follows:

[0030] (1) Obtaining the image of the optoelectronic pod, the image of the optoelectronic pod is 1280*720;

[0031] (2) Add 48 rows and 1280 columns of blank pixels below the electro-optical pod image to make the electro-optical pod image resolution 1280*768. The length and width of this resolution are both divisible by the total step size (32) of the neural network. Since this part of the blank image is continuous, the memory operation time of this part is relatively short.

[0032] The optoelectronic pod image is sent to the NPU module of the airborne chip for target detection (inference) to obtain the target detection result;

[0033] (3) Based on the target detection results, the borders with low confidence (the confidence threshold is a preset value) are filtered out, and the borders with high confidence are taken out from the NPU module to the memory of the onboard chip;

[0034] (4) Draw the target border in the electro-optical pod image;

[0035] (5) The optoelectronic pod image is compressed according to the H264 standard and sent to the ground station using the RTSP protocol, while the detection result of the current frame image is saved.

[0036] The structure of the target detection network used for target detection in the NPU module designed in step (2) above is as follows: Figure 1As shown in the figure, the target detection algorithm implemented by the target detection network includes two stages. The total step size of the backbone network in the first stage is 16. The input image with a resolution of 1280*768 passes through the backbone network of the first stage to obtain a feature map with a resolution of 80*64, and then passes through the encoder and decoder of the first stage to obtain the detection result of the first stage; the step size of the backbone network in the second stage is 2, and the feature map with a resolution of 80*64 passes through the backbone network of the second stage to obtain a feature map with a resolution of 40*32, and then passes through the encoder and decoder of the second stage to obtain the detection result of the second stage, and then the detection results of the two stages are fused through the non-maximum suppression algorithm to obtain the final target detection result.

[0037] The CPU computing power of the onboard chip is relatively weak, and it takes a long time to resize the image. For example, it takes about 10ms to resize a 1280*768 resolution image to a 640*384 image on the HiSilicon 3516AV300 chip. The input image resolution of yolov5 is 640*384, and the method of the present invention changes the input resolution to 1280*768. The target detection network designed by the method of the present invention is relative to the yolov5 neural network structure. In the first stage, a convolution layer with a step size of 2 is added to the backbone network. The image with a resolution of 1280*768 passes through the convolution layer, and the resolution becomes 640*384. This equivalently realizes the resize function. The computing power of the NPU is stronger than that of the CPU. The above-mentioned added convolution step is executed in the NPU, and the running speed is faster. At the same time, the convolution operation adds a feature extraction function compared to the direct resize operation, which helps to improve the detection effect.

[0038] The structure of the above target detection network is independent between layers, there is no cross-layer connection, and the calculation parallelism is good. In order to integrate multi-layer features, traditional target detection algorithms use cross-layer connections, such as cross-layer connections in yolov5 (such as Figure 2 ).

[0039] When the NPU performs neural network reasoning, for networks with the same amount of computing power, the better the parallelism, the shorter the computing time. The encoder and decoder of each layer of the target detection network designed by the present invention can directly output the detection results without waiting for the calculation results of the deep network. In the yolov5 structure, after calculating the backbone network 1, encoder 1 cannot be calculated directly, and it is necessary to wait for the calculation results of backbone network 2, backbone network 3, encoder 2, and encoder 3. Similarly, encoder 2 also needs to wait for the calculation results of backbone network 3 and encoder 3. This kind of "waiting" is very unfriendly to parallel computing and increases the computing time. In the target detection network designed by the method of the present invention, the encoder and decoder structures independent of each layer do not require such "waiting", which effectively improves the computing efficiency.

[0040] In the UAV target recognition task, in order to ensure the stealth and safety of detection, the UAV usually observes the target at a long distance, and the pixel scale of the target in the image is small. The deeper the neural network, the larger its receptive field, and the more forward the neural network, the smaller its receptive field. A neural network with a larger receptive field is sensitive to large-scale targets, and a neural network with a smaller receptive field is sensitive to small-scale targets. A neural network with a specific receptive field can detect targets within a certain scale range, and its sensitivity to the target is normally distributed relative to the square root of the target pixel scale, such as Figure 3 As shown. In order to detect larger-scale targets, general target detection algorithms usually include shallow, middle, and deep networks, among which the deep network is more sensitive to large-scale targets. However, the drone recognition task does not include larger-scale targets, so the method of the present invention only includes shallow and middle networks.

[0041] The method of the present invention is implemented on an airborne computing platform based on the domestic HiSilicon 3516AV300 chip, and the time consumption for detecting one frame of image is about 37ms. The UAV is equipped with an optoelectronic pod and an airborne computing platform for flight experiments. The target detection effect in the flight test is as follows: Figure 4 shown.

[0042] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fast target detection method applied to a weak computing power airborne chip, characterized in that: The following steps are involved: (1) Obtaining the image of the optoelectronic pod, the image of the optoelectronic pod is 1280*720; (2) Add 48 rows and 1280 columns of blank pixels below the optoelectronic pod image to make the optoelectronic pod image resolution 1280*768. The length and width of this resolution are divisible by the total step size of the neural network. Then the optoelectronic pod image is sent to the NPU module of the onboard chip for target detection to obtain the target detection result; (3) Based on the target detection results, low-confidence borders are filtered out, and high-confidence borders are taken out from the NPU module to the memory of the onboard chip; (4) Draw the target border in the electro-optical pod image; (5) The optoelectronic pod image is compressed according to the H264 standard and sent to the ground station using the RTSP protocol, while the detection result of the current frame image is saved.

2. The method according to claim 1, characterized in that The target detection algorithm implemented by the target detection network for target detection in the NPU module designed in step (2) includes two stages in total. The total step size of the backbone network in the first stage is 16. The input image with a resolution of 1280*768 is passed through the backbone network of the first stage to obtain a feature map with a resolution of 80*64, and then passed through the encoder and decoder of the first stage to obtain the detection result of the first stage; the step size of the backbone network in the second stage is 2, and the feature map with a resolution of 80*64 is passed through the backbone network of the second stage to obtain a feature map with a resolution of 40*32, and then passed through the encoder and decoder of the second stage to obtain the detection result of the second stage, and then the detection results of the two stages are fused through the non-maximum suppression algorithm to obtain the final target detection result.

3. The method according to claim 2, characterized in that In the target detection network used for target detection in the NPU module designed in step (2), a convolution layer with a step size of 2 is added to the backbone network of the first stage. The image with a resolution of 1280*768 passes through this convolution layer, and the resolution becomes 640*384.

4. The method according to claim 1, characterized in that When performing border filtering in step (3), the confidence threshold is used as a preset value.

5. The method according to any one of claims 1 to 4, characterized in that This method is applied in the field of computer vision technology.

6. A fast target detection system for implementing the method as claimed in any one of claims 1 to 5.

7. The system according to claim 6, characterized in that The system is designed based on an airborne computing platform.

8. The system according to claim 5 or 6, characterized in that The system is applied in the field of computer vision technology.

9. An unmanned aerial vehicle flight experiment platform implemented based on the system as described in claim 6, 7 or 8.

10. The UAV flight test platform according to claim 9, characterized in that: The UAV flight experiment platform is realized by the UAV carrying an optoelectronic pod and the airborne computing platform.

Images