FPGA-based high energy efficiency embedded uav intrusion detection system

Abstract

The application provides an FPGA-based high-energy-efficiency embedded UAV intrusion detection system, adopts quantitative perception training QAT to retrain and quantize a network, quantizes weight parameters and feature map data to 8 bits under the premise of ensuring detection accuracy, and fuses a batch normalization layer into a convolution layer, so that the consumption of hardware resources is reduced. The overall system is realized through software and hardware collaborative design, reasonable division of software and hardware tasks, and high efficiency of the whole system.

Description

Technical Field

[0001] The present invention relates to the field of drone intrusion technology, and in particular to a high-energy-efficiency embedded drone intrusion detection system based on FPGA. Background Art

[0002] With technological advancements, drones and other aircraft are becoming increasingly maneuverable and flexible. They possess the ability to fly at low altitudes, move slowly, and be difficult to detect. They can also carry payloads and easily penetrate ground defenses, potentially posing a threat to critical ground infrastructure. Therefore, effectively preventing and responding to interference and sabotage activities by drones has become a major global security challenge, primarily due to the difficulty in controlling, detecting, and effectively addressing them.

[0003] In today's low-altitude airspace surveillance, a variety of detection technologies are used to identify and track flying targets, including radar, radio, and photoelectric detection. Photoelectric detection is highly favored due to its excellent anti-interference capabilities, intuitive image quality, and flexible deployment. In complex urban environments, photoelectric detection can effectively assist in the search and identification of targets such as drones. However, photoelectric detection technology also faces a number of challenges. First, it is highly sensitive to environmental conditions, such as visibility and humidity. Harsh weather conditions, such as fog, rain, or smoke, can severely impact photoelectric detection performance. Second, in practical applications, there are numerous unavoidable interference factors and uncertainties. For example, buildings or natural obstructions can block the line of sight, background noise can mask the target signal, and sudden changes in light can degrade image quality. These factors all add additional challenges to the detection and tracking of moving targets. Given these challenges, the research and development of efficient target tracking algorithms is crucial. These algorithms must be able to accurately separate targets from complex backgrounds and maintain stable tracking performance even in obstructed line of sight or adverse environmental conditions. Furthermore, the algorithm should possess a certain degree of self-adaptation, being able to adjust its parameters based on real-time data to cope with environmental changes and interference factors. By applying these advanced algorithms, we can improve the accuracy and reliability of detecting flying targets in low-altitude airspace, thereby effectively ensuring airspace safety.

[0004] At present, the research on drone tracking and detection based on machine vision can be divided into traditional image processing methods and deep learning-based methods. The drone detection methods based on traditional image processing are mainly divided into image preprocessing, image segmentation, feature extraction and recognition detection. Image preprocessing is to expand the effective information of the image through operations such as denoising and enhancement. Image segmentation mainly separates the target area of ​​interest from other background areas. Feature extraction is to extract the color, texture or shape features of the detection target in the area of ​​interest, and finally identify and detect it through feature information. However, when using traditional image processing methods for drone detection, false detection and missed detection may occur when the image is affected by lighting and the defect contrast is not obvious. With the rise of convolutional neural network (CNN) in industrial applications, drone detection methods based on deep learning have been widely used in various detection scenarios. This method adaptively learns parameters and extracts feature information for specific image datasets.

[0005] Automatic detection of drone intrusions for airspace security has certain requirements for inference speed and power consumption. However, most current defect detection research does not use efficient hardware platforms to deploy defect detection algorithms, resulting in high power consumption and low energy efficiency, which does not meet the requirements of airspace security scenarios. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a high-energy-efficiency embedded drone intrusion detection system based on FPGA, deploying the YOLOv7-Tiny target detection network on FPGA for hardware implementation, and used for drone target tracking target detection.

[0007] To achieve the above objectives, the present invention adopts the following technical solutions: a high-efficiency embedded drone intrusion detection system based on FPGA, which uses YOLOv7-Tiny to train a drone target tracking dataset, and optimizes the network at the algorithm level through QAT and layer fusion technology; designs a hardware accelerator with a multiplexed multi-node configurable architecture, and implements different accelerator functions by configuring multiple different nodes; the hardware accelerator with a multiplexed multi-node configurable architecture implements forward reasoning of the network, performs fine-grained optimization of each hardware module, and designs a dual-pipelined structure within and between layers; finally, the entire detection system is finally realized through hardware and software collaborative design.

[0008] The YOLOv7-Tiny specifically includes:

[0009] 1) Layer fusion: The BN layer is integrated into the convolutional layer. The convolutional layer calculation formula is shown in formula (1), where Y conv and X conv Represent the calculation results and input data of the convolutional layer respectively, W and b are weights and bias respectively;

[0010]

[0011] The calculation formula of the BN layer is shown in formula (2), where Y bn and X bn Represent the calculation results and input data of the BN layer respectively, γ and β represent the scaling factor and bias of the BN layer respectively, ε is the sliding coefficient which is a small positive number, μ and δ are the mean and variance of the input data;

[0012]

[0013] Formula (1) converts the calculation result Y of the convolution layer into conv Substitute the input X of the BN layer in formula (2) bn And rearrange them to achieve the fusion of normalization layer and convolution layer. The final new convolution layer formula is shown as follows:

[0014]

[0015] Where X' and Y' are the input data and calculation results of the new convolutional layer, respectively, and W' and b' are the weights and biases of the new convolutional layer;

[0016] 2) QAT model: This model adds pseudo-quantization nodes to the network training to mark the numerical range of the parameters. During training, it simulates quantization to adjust the zero point and scaling factor to reduce the error caused by quantization. After training, the network model is quantized to 8-bit fixed-point numbers by obtaining the optimal zero point and scaling factor.

[0017] During the network forward inference process, the feature map and weight data are quantized to 8-bit width. The feature map data is quantized to the value range of (0, 256), and the weight data is quantized to the value range of (-128, 127). The scaling coefficient S and zero point Z obtained by QAT are used to map the floating point number X to the fixed point number q as shown in formula (4);

[0018] X=S(qZ) (4)

[0019] Substituting formula (4) into (3) yields the following formula:

[0020]

[0021] Among them, since the weight is symmetrically quantized, the zero point Z of the weight can be obtained W =0; take S b =S W S X , the bias adopts symmetric quantization and takes Z Y =0, we can get the following formula;

[0022]

[0023] Among them only is a floating point number; S W and S X The scaling ratio is 8 bits, so S W S X The scaling ratio is 16 bits. In the QAT model, the bias is calculated with a 32-bit data width, and the scaling is done with S W S X To replace S b Using a biased number will result in a loss of half the information.

[0024] In a preferred embodiment, the weights, biases, and quantization factor parameter data of the network model trained using the drone intrusion dataset are rearranged and saved in advance in the format of a binary file on an SD card. Before performing drone target tracking detection, the SD card controller reads the weights, biases, scaling factors, and zero points from the SD card to a preset DDR memory offset address through the hardware interface of the SD card slot. After a single sector is read, the SD card sends an SD_Done interrupt signal to the CPU and starts reading the next sector. The software-side CPU first configures each module through the advanced extensible interface AXI bus and presets the configuration parameters of the accelerator on the software side. The CPU configures the hardware accelerator through the AXI interface based on these parameters, thereby realizing forward reasoning of different network layers in the drone intrusion detection network; the accelerator reads and writes data through the AXIStream interface, and data interaction between the accelerator and the DDR memory is realized by passing the address instruction information and data transmission length of the DDR memory to AXIDMA; after the hardware accelerator is executed once, it will send a Task_Done interrupt signal to the CPU, and the CPU will reconfigure the parameters of the hardware accelerator, and repeat this process until the reasoning of the entire network is finally completed; based on the results of network reasoning, the CPU uses the non-maximum suppression (NMS) algorithm to obtain the coordinate position of the drone target tracking.

[0025] In a preferred embodiment, the CPU calculates the data transmission parameters according to the parallelism of the accelerator. In the next cycle, the CPU configures the hardware accelerator to receive mode through the AXI interface, calculates the transmission parameters for the next cycle, and transfers the feature map data, weight data, and bias data from the DDR memory to the on-chip cache of the accelerator through AXIDMA. After the hardware accelerator receives the data, it sends an interrupt to the CPU to inform the CPU that the reception is complete. After receiving the interrupt signal, the CPU starts the next cycle. The CPU configures the hardware accelerator to receive and inference mode. The hardware accelerator adopts a double-buffer structure. One input feature map data cache is used for the forward reasoning operation of the current cycle, and the other input feature map data cache is used to prepare the input feature map data of the next cycle. When the hardware accelerator completes data reception and data forward reasoning, it also sends an interrupt signal to the CPU to indicate that the execution is completed. After the input channel is cycled through and the calculation of the first batch of output data is completed, the CPU configures the hardware accelerator to send mode, sends the output feature map cache data to the DDR memory, and then starts the next cycle until the feature map data of all output channels are calculated, that is, the reasoning of the network layer is completed.

[0026] In a preferred embodiment, the accelerator first configures a control module, which can analyze the network layer type based on the information, thereby generating enable control signals for other sub-modules; a dual-input cache mechanism is adopted, and the two blocks of input feature map cache space are alternately written and calculated through the cache selector; the filling module is controlled by a filling enable signal, and the filling signal is generated by the control module according to the configured filling value. When the signal is pulled high, the zero point value of the quantized feature map data is output, and when the signal is pulled low, the data output of the input feature map cache space is read; the feature map data needs to be subtracted from the quantized zero point value and the data path is selected according to the configuration information of the convolution type. If it is a 1×1 convolution, the data is directly input into the convolution module for convolution operation, and if it is a 3×3 convolution, the data needs to pass through the row buffer to form a 3×3 sliding window data and then be sent to the convolution module for convolution operation. The row buffer pre-caches the data to be used; the data stream output by the selector is input into the convolution processing unit and LeakyReLU PE for calculation.

[0027] In a preferred embodiment, the convolution module reads the weights and feature map data of different arrangements according to the convolution type selector value to perform convolution operations. After the calculation, two results will be output: one is the value after the convolution calculation without activation function activation, which is suitable for the convolution layer type of the detection layer in the drone intrusion detection network; the other is the value after the convolution calculation after activation function activation, which is suitable for other convolution layer types in the model;

[0028] It also includes inter-layer pipeline structures of convolution modules and upsampling modules, and convolution modules and downsampling modules, realizing inter-layer and intra-layer dual pipeline structures, and selecting the output data stream through the acceleration type selector; the drone intrusion detection network splices the feature maps of different channels together through the splicing layer; in hardware implementation, after quantization calculation, the data is transferred to the DDR memory with a preset offset address to realize data splicing, and finally the result is written into the output feature map cache space.

[0029] In a preferred embodiment, the input feature map data of the convolution module first needs to determine the convolution kernel size. The 3×3 convolution data needs to generate 3×3 sliding window data through the line buffer first, and the 1×1 convolution is directly input into the multiplication array for calculation; in the convolution calculation, the two convolution multiplication operations are integrated into one DSP, and the DSP is optimized; for the weight data of different output channels, the feature map data of the same input channel can be used to calculate the convolution multiplication results of two different output channels; the upper 9 bits of the DSP's A port are filled with the sign bit of the weight of the output channel X, and the lower 18 bits are filled with the sign bit of the weight of the output channel Y, and the B port is filled with the input feature map data. Figure 9 The 8-bit data is sign-extended and padded; the final result of 8-bit wide data multiplied by 9-bit wide data is stored in 17-bit wide data, so the C output port takes the lower 17 bits to output the convolution multiplication result of channel Y, and then takes the next 17 bits of data as the convolution multiplication result of output channel X.

[0030] In a preferred embodiment, the data stream calculated by the convolution module is first written into the data cache space in the upsampling module, and image upsampling is achieved by controlling the read address of the data cache space; after the upsampling module starts working, it counts the rows and columns, and the read address will be increased by 1 only when two data are output from the same read address. After a row of valid data is output, the read address will be subtracted from the row valid data length, and then another row of data identical to the previous row will be output in the same way. Repeating this operation can complete the upsampling operation of the complete feature map data.

[0031] Compared with the existing technology, the present invention has the following beneficial effects: The present invention proposes a high-energy-efficiency embedded drone intrusion detection system based on field programmable gate array FPGA, and adopts quantitative perception training QAT to retrain and quantize the network. Under the premise of ensuring detection accuracy, the weight parameters and feature map data are quantized to 8 bits, and the batch normalization layer is integrated into the convolution layer, which reduces the consumption of hardware resources. The overall system is implemented by software and hardware collaborative design, and the software and hardware tasks are reasonably divided, thereby achieving the efficiency of the entire system. The overall system is implemented by software and hardware collaborative design, and the software and hardware tasks are reasonably divided, thereby achieving the efficiency of the entire system. A hardware accelerator with a multiplexed multi-node configurable architecture is designed, which realizes the acceleration of different network layers through multiple configuration nodes, and performs fine-grained optimization design. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a structural diagram of the ELAN-T module according to a preferred embodiment of the present invention;

[0033] Figure 2 A system framework diagram of a preferred embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the software and hardware coordinated operation mode of a preferred embodiment of the present invention;

[0035] Figure 4 This is a diagram of the architecture of a multi-layer dynamically configurable accelerator according to a preferred embodiment of the present invention;

[0036] Figure 5 This is a diagram of the convolution module architecture of a preferred embodiment of the present invention;

[0037] Figure 6 A configurable row buffer diagram of a preferred embodiment of the present invention;

[0038] Figure 7 DSP optimization design diagram for the preferred embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of the SPP module architecture of a preferred embodiment of the present invention;

[0040] Figure 9 This is a diagram showing the detection effect of the network model after QAT quantization in a preferred embodiment of the present invention;

[0041] Figure 10 This is a diagram of the overall power consumption of the hardware acceleration system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0045] The present invention proposes a high-energy-efficiency embedded drone intrusion detection system based on Xilinx VC707FPGA. YOLOv7-Tiny is used to train the drone target tracking dataset, and the network is optimized at the algorithm level through QAT and layer fusion technology. A hardware accelerator with a multiplexed multi-node configurable architecture is designed. Different accelerator functions are realized by configuring multiple different nodes, and the hardware accelerator is reused to realize the forward reasoning of the network. Each hardware module is optimized in a fine-grained manner, and a dual-pipelined structure within and between layers is designed. Finally, the entire detection system is finally realized through hardware and software collaborative design, achieving an average detection accuracy of 88.8% and a throughput of 177.96GOPS. The power consumption is only 6.5W, and the energy efficiency reaches 27.38GOPS / W. Compared with GPUs and CPUs, it has high energy efficiency and achieves a balance between detection speed, accuracy and power consumption. It is suitable for the application field of rapid drone detection.

[0046] This paper utilizes the YOLOv7-Tiny network to accurately and automatically detect drone targets, and performs algorithm-level optimization to save hardware resources. Using QAT to retrain and quantize the network model, the network's weight and feature map data widths are compressed from 32 bits to 8 bits, while only decreasing the network's detection accuracy by 2.8%. Furthermore, layer fusion is used to integrate normalization layers into convolutional layers, reducing the propagation of quantization errors in QAT while also reducing computational complexity.

[0047] The present invention proposes a hardware accelerator with a multiplexed multi-node configurable architecture, which implements reasoning at different network layers through multiple configurable nodes. This multiplexed shared hardware accelerator is used to implement the YOLOv7-Tiny network, saving hardware resource consumption and system complexity. The present invention also presents the first hardware implementation of the YOLOv7-Tiny network on an FPGA.

[0048] This paper implements various modules within the accelerator, designing configurable line buffers for the convolution, downsampling, and spatial pyramid pooling (SPP) modules. The convolution module also features an optimized digital signal processor (DSP) design and a multiplexed multiplication array, improving hardware resource utilization. Each module is fully pipelined within its own layer and partially pipelined between layers. This dual-pipelined hardware architecture, combining both intra- and inter-layer pipelines, improves the accelerator's throughput.

[0049] The entire embedded drone intrusion detection system is built using a hardware-software collaborative design. A soft-core CPU is installed on the chip, and computationally intensive tasks are delegated to the hardware accelerator, while data control tasks are handled by the CPU. The CPU and hardware accelerator can execute independent computing tasks in parallel, balancing the flexibility of software design and the efficiency of hardware design.

[0050] (1) Target tracking and detection based on machine vision

[0051] The present invention adopts a target detection algorithm based on deep learning to detect and locate intruding drones.

[0052] (2) Target Detection Algorithm

[0053] Currently, deep learning-based object detection algorithms can be categorized into two types based on the algorithm's execution stage: a two-stage region-candidate detection algorithm and a one-stage regression-based detection algorithm. In the first stage of a two-stage region-candidate detection algorithm, candidate bounding boxes are generated containing the detected object. A two-stage classifier then classifies and corrects the position of the detected object within each candidate bounding box.

[0054] This paper will implement drone intrusion target detection based on the YOLOv7-Tiny target detection network.

[0055] (3) FPGA-based CNN accelerator

[0056] The present invention chooses to deploy the drone intrusion detection algorithm on FPGA. Through algorithm optimization and hardware structure optimization, a high-efficiency embedded drone intrusion detection system based on FPGA is realized, which meets the requirements of the drone efficient and accurate detection environment.

[0057] Design of UAV Intrusion Detection System

[0058] 1. Network Optimization Strategy for Drone Intrusion Detection

[0059] Table 1 YOLOv7-Tiny network architecture

[0060]

[0061] Table 1 shows the network structure of YOLOv7-Tiny, which lists the network layer number, network layer type, input feature map size, number of input channels, number of output channels, convolution kernel size, step size, and input data source. Layers 37, 38, and 39 are the detection layers of the network. n represents the number of categories predefined by the network model. The number of output channels of the detection layer depends on the size of n. The ELAN-T module structure is shown in Figure 1. Figure 1 As shown, it consists of four groups of feature calculation blocks and splicing layers. Compared with YOLOv7, two groups of feature calculation blocks are reduced. Each group of feature calculation blocks consists of a convolution layer, a batch normalization layer, and an activation function layer. With a smaller number of parameters and calculations, the network model can learn more feature information. Layers 11-13 are SPP layers that increase the receptive field of the image through three maximum pooling layers of different scales. The present invention uses YOLOv7-Tiny to detect drone intrusion images. The input image size of the drone is 416×416. The output layer can predict at three scales and detect defect targets of three different sizes: large, medium, and small. In order to enable the drone intrusion detection network to be better deployed on hardware, the present invention optimizes the network model and reduces the consumption of hardware multipliers and on-chip storage resources, including integrating the normalization layer (BatchNormalization, BN) into the convolution layer and QAT model.

[0062] 1) Layer Fusion: Batch Normalization (BN) layers have been shown to accelerate network convergence and effectively prevent exploding and vanishing gradients in CNN training. While BN layers perform well during network training, they consume a significant amount of computation during forward inference. Fusion of BN layers into convolutional layers not only reduces computation but also reduces the number of quantization and dequantization steps for QAT, thereby reducing the error accumulation caused by the quantization process.

[0063] The function of the convolution layer in CNN is to extract the feature information of the input feature map. The calculation formula of the convolution layer is shown in formula (1), where Y conv and X conv They represent the calculation results and input data of the convolutional layer respectively, W and b are the weight and bias respectively.

[0064]

[0065] The calculation formula of the BN layer is shown in formula (2), where Y bn and X bn Represent the calculation results and input data of the BN layer respectively, γ and β represent the scaling factor and bias of the BN layer respectively, ε is the sliding coefficient, which is a very small positive number to prevent the denominator from being 0, μ and δ are the mean and variance of the input data.

[0066]

[0067] Formula (1) converts the calculation result Y of the convolution layer into conv Substitute the input X of the BN layer in formula (2) bn And rearrange them to achieve the fusion of normalization layer and convolution layer. The final new convolution layer formula is shown as follows:

[0068]

[0069] Where X' and Y' are the input data and calculation results of the new convolutional layer, respectively, and W' and b' are the weights and bias of the new convolutional layer.

[0070] 2) QAT Model: Network models trained in deep learning frameworks are typically based on 32-bit floating-point numbers. However, deploying 32-bit precision network models directly on FPGAs creates significant computational and storage pressures, hindering their application in drone intrusion detection networks. Furthermore, the data streams in FPGAs are all binary fixed-point numbers, making fixed-point operations more efficient than floating-point operations in FPGAs. Therefore, the network model needs to be quantized.

[0071] This paper uses the QAT algorithm to quantize the YOLOv7-tiny network model used for drone intrusion detection. This algorithm adds pseudo-quantization nodes to the network training to mark the numerical range of parameters. During training, simulated quantization is used to adjust the zero point and scaling factor to reduce quantization errors. After training, the network model is quantized to 8-bit fixed-point numbers using the optimal zero point and scaling factor. Compared to post-training quantization, the quantization-aware algorithm can compress network model parameters while maintaining minimal accuracy loss.

[0072] During the network forward inference process, the feature map and weight data are quantized to 8 bits. The feature map data is quantized to the numerical range of (0, 256), and the weight data is quantized to the numerical range of (-128, 127). The formula for mapping the floating-point number X to the fixed-point number q using the scaling coefficient S and zero point Z obtained by QAT is shown in Equation (4).

[0073] X=S(qZ) (4)

[0074] Substituting equation (4) into equation (3) yields the following equation.

[0075]

[0076] Among them, since the weight is symmetrically quantized, the zero point Z of the weight can be obtained W = 0. In order to make the algorithm more efficient in FPGA, take S b =SW S X , the bias adopts symmetric quantization and takes Z Y =0, we can get the following formula.

[0077]

[0078] Among them only For floating point numbers, just find a suitable n and quantize it by shifting If it becomes a fixed-point number, the entire formula can be converted into a fixed-point number calculation, and finally the result can be obtained by truncation. W and S X The scaling ratio is 8 bits, and S W S X The scaling ratio is 16 bits. In QAT, the offset is calculated with a 32-bit data width. The scaling ratio is S W S X To replace S b The bias value loses half of the information. Due to the redundant parameters of deep CNNs, this loss does not affect the final result in most cases, but it can lead to efficient hardware design.

[0079] 2. UAV Intrusion Detection System Framework

[0080] like Figure 2The following figure shows the architecture of the FPGA-based, energy-efficient, embedded drone target detection system designed by the present invention. This system utilizes a hardware-software collaborative design, with a MicroBlaze soft-core processor on-chip serving as the CPU. The hardware accelerator is designed as a multiplexed, multi-node configurable architecture. The CPU manages overall system scheduling and control tasks, while the hardware accelerator performs network forward reasoning computations. After training with a drone intrusion dataset, the network model's parameters, such as weights, biases, and quantization factors, are rearranged and pre-saved as a binary file on an SD card. Prior to drone target tracking and detection, the SD card controller reads these parameters from the SD card to a preset DDR memory offset address via the SD card slot's hardware interface. Upon completion of a single sector read, the SD card sends an SD_Done interrupt signal to the CPU, initiating the next sector read. The software-side CPU first configures each module via the Advanced Extensible Interface (AXI) bus and presets the accelerator's configuration parameters on the software side. The CPU then configures the hardware accelerator based on these parameters through the AXI interface, enabling forward reasoning at different network layers within the drone intrusion detection network. The accelerator reads and writes data via the AXI Stream interface. Data exchange between the accelerator and the DDR memory is achieved by simply passing the address instruction information and data transfer length of the DDR memory to the AXIDMA. After the hardware accelerator completes a single execution, it sends a Task_Done interrupt signal to the CPU, reconfiguring the hardware accelerator's parameters. This cycle repeats until the entire network's reasoning is complete. Based on the network's reasoning results, the CPU uses the non-maximum suppression (NMS) algorithm to determine the coordinate position of the drone's target for tracking.

[0081] like Figure 3As shown in the figure, the system performs inference on a particular network layer as follows: the CPU calculates data transmission parameters based on the accelerator's parallelism. In the next loop, the CPU configures the hardware accelerator to receive mode via the AXI interface and calculates the transmission parameters for the next loop. It then transfers feature map data, weight data, and bias data from the DDR memory to the accelerator's on-chip cache via AXIDMA. After the hardware accelerator receives the data, it sends an interrupt to the CPU to notify it of the completion of reception. After receiving the interrupt, the CPU begins the next loop. The CPU configures the hardware accelerator to receive and inference mode. The hardware accelerator uses a dual-buffer structure: one input feature map data cache is used for forward inference calculations in the current loop, and the other input feature map data cache is used to prepare input feature map data for the next loop. When the hardware accelerator completes data reception and forward inference, it also sends an interrupt to the CPU to indicate completion. After looping through the input channels and calculating the first batch of output data, the CPU configures the hardware accelerator to send mode and sends the output feature map cache data to the DDR memory. The next loop begins again until feature map data calculations for all output channels are complete, completing the inference for that network layer. The design of parallel operation of the CPU and accelerator can fully reduce system latency and make the system throughput as close as possible to the accelerator's throughput, giving full play to the advantages of CPU scheduling control and accelerator accelerated inference, thereby improving the real-time performance of the entire system and improving the speed and accuracy of defective drone detection.

[0082] Drone Intrusion Detection Hardware Accelerator

[0083] (1) Multi-layer dynamically configurable hardware architecture

[0084] like Figure 4The illustrated architecture is a multi-layer, dynamically configurable one. The hardware accelerator configures itself based on received register information. Using different configuration parameters and appropriate timing control, different network layers within the drone intrusion detection network can be implemented. Multiple hardware optimization designs based on FPGAs are implemented to achieve a highly energy-efficient drone intrusion detection system. The accelerator first configures the control module, which analyzes the network layer type based on the information and generates enable control signals for other submodules. Due to the limited on-chip cache resources of FPGAs, the volume of feature map and parameter data in the drone intrusion detection network is insufficient to store all data simultaneously. Furthermore, there is a memory access delay when reading data from DDR memory into the on-chip cache. This invention utilizes a dual-input cache mechanism. Using a cache selector, data is written and calculated alternately between two input feature map caches, reducing data transmission delay and the impact of memory access delay on the accelerator. The fill module is controlled by a fill enable signal, generated by the control module based on the configured fill value. When this signal is high, the quantized zero value of the feature map data is output. When the signal is low, the data from the input feature map cache is read and output. After quantizing the zero-point value, the feature map data is deducted and the data path is selected based on the convolution type configuration. For 1×1 convolution, the data is directly fed into the convolution module for convolution. For 3×3 convolution, the data is formed into a 3×3 sliding window in the row buffer before being fed into the convolution module for convolution. The row buffer pre-caches the data to be used, thus avoiding the latency caused by data movement and copying. The data stream output by the selector is fed into the convolution processing engine (PE) and the LeakyReLU PE for calculation.

[0085] The convolution module reads weights and feature map data of different arrangements according to the convolution type selector value and performs a convolution operation. After the calculation, it outputs two results: one is the value after the convolution operation without activation function, which is suitable for the convolution layer type of the detection layer in the drone intrusion detection network; the other is the value after the convolution operation with activation function, which is suitable for other convolution layer types in the model. The maximum pooling layer used for image downsampling in the backbone network of the drone intrusion detection network model uses a 2×2 sliding kernel, so a corresponding row buffer for generating a 2×2 sliding window is required. At the same time, to reduce the latency caused by off-chip data transmission, the present invention designs an inter-layer pipeline structure for the convolution module and upsampling module, and the convolution module and downsampling module, realizing an inter-layer and intra-layer dual pipeline structure. The output data stream is selected by the acceleration type selector. At the same time, in order to store the output data of the convolution layer in the inter-layer pipeline structure, a mid-segment output feature map cache space is added. The SPP module is specifically designed to match the maximum pooling of sliding kernels of different sizes. The drone intrusion detection network stitches together feature maps from different channels through the stitching layer, enabling the network to learn more feature information and improve network performance. In hardware implementation, it only requires quantizing the data and transferring it to the DDR memory with a preset offset address to achieve data stitching, and finally write the results into the output feature map cache space.

[0086] (2) Overview of the Convolutional Module Architecture

[0087] like Figure 5 The figure shows a detailed overview of the convolution module architecture. To input feature map data, the convolution kernel size must be determined first. 3×3 convolution data must first be generated into a 3×3 sliding window through a row buffer, while 1×1 convolution is directly input into the multiplication array for calculation.

[0088] like Figure 6 As shown, the present invention designs a length-configurable line buffer, where the output nodes of the line buffer data stream are determined by the image padding value and the line buffer configuration value. The line buffer configuration value sets output nodes with lengths of 13, 26, 52, 104, 208, and 416, respectively, based on the feature map sizes of each level of the network model. The line buffer length can be reconfigured to be extended by 0 or 2 based on the padding value of the convolutional layer in the network model. Different configurations allow line buffering of feature map data of different scales, avoiding the need to design separate line buffers for each scale of feature map data, thus saving hardware resources.

[0089] The largest convolution kernel size in the drone intrusion detection network model is 3×3, so the multiplication array is designed to be 3×3 in size. This 3×3 convolution and 1×1 convolution can share the same 3×3 multiplication array. The corresponding calculations are completed by feeding weight data and feature map data of different permutations into the convolution multiplication array. For convolution operations with different step lengths, it is necessary to control the high-level period of the output data enable signal to selectively output data. The sum of the input channel multiplication array is then calculated over multiple cycles using an addition tree and stored in an accumulation buffer. In the next iteration, the result from the accumulation buffer is read, added to the cumulative sum of the next iteration, and then written back to the accumulation buffer. This loop repeats until all input channels are calculated. Quantization is then performed to obtain the quantized convolution layer result. Finally, the convolution layer output data is activated using the LeakyReLU activation function and output.

[0090] In the convolution calculation, the weight is an 8-bit wide signed number, and the feature map data minus the zero point value is a 9-bit wide signed number. The hardware multiplier DSP48E in the FPGA can perform 18-bit multiplication by 27-bit multiplication. In order to give full play to the performance of the hardware multiplier, we integrate the two convolution multiplication operations into one DSP and optimize the DSP design. Figure 7 As shown in the figure, the weight data of different output channels can share the feature map data of the same input channel to calculate the convolution multiplication results of two different output channels, which provides a prerequisite for the optimization design of DSP. The upper 9 bits of the A port of DSP are filled with the weight of the output channel X after the sign bit is extended, and the lower 18 bits are filled with the weight of the output channel Y after the sign bit is extended. The B port is filled with the input feature map data. Figure 9 The 8-bit data is sign-extended and padded. The final result of multiplying 8-bit wide data by 9-bit wide data can be stored as 17-bit wide data. Therefore, the C output port takes the lower 17 bits to output the convolution multiplication result of channel Y, and then takes the next 17 bits of data as the convolution multiplication result of output channel X. In actual calculations, when the convolution multiplication result of output channel Y is negative, it affects the convolution multiplication result of output channel X. This is because the convolution result of the signed number of output channel Y is treated as an unsigned number. In this case, it is necessary to add 1 to the convolution multiplication result of output channel X to restore the true result.

[0091] (3) Overview of downsampling module architecture

[0092] The backbone network of the drone intrusion detection network uses a maximum pooling layer with a stride of 2 and a sliding kernel size of 2×2 to downsample the image, reducing the size of the feature map by 1 / 4. The activation values ​​calculated by the convolution module are fed into a configurable row buffer to generate a 2×2 sliding window, which is then fed into the pooling module to filter the maximum value. The pooling module performs a two-cycle comparison tree to find the maximum value through pairwise comparison. Due to the stride of 2, the pooling operation requires row and column counting at the beginning. When both the row and column count values ​​are even, the output data valid enable signal is pulled high, and the data is written to the output feature map buffer space.

[0093] (4) Overview of SPP module architecture

[0094] The SPP module enhances the receptive field of the network by using the maximum pooling of sliding kernels of different sizes. Figure 8 As shown in the hardware architecture diagram in [1], to conserve hardware resources, the present invention uses three sliding cores of 3×3, 5×5, and 7×7 sizes, respectively. Six row buffers are designed based on the largest sliding core, capable of simultaneously outputting up to seven rows of data. Because the image padding values ​​for sliding cores of different sizes differ, the row buffers are configured to different lengths based on the image padding value. The row buffers are backward compatible, enabling sliding window data of varying sizes to be obtained by selecting different numbers of rows for output data. The output sliding window data is then selected using a selector. The output sliding window data is filtered out by a comparator over multiple cycles to obtain the maximum value.

[0095] (5) Overview of upsampling module architecture

[0096] The drone intrusion detection network uses two upsampling layers, expanding the 13×13 feature map to 26×26 and the 26×26 feature map to 52×52, respectively. These layers are then concatenated and fused with the feature map from the backbone network, effectively reducing information loss. The data stream computed by the convolution module is first written into the data cache of the upsampling module. Image upsampling is achieved by controlling the read address of the data cache. When the upsampling module begins operation, it counts rows and columns. The read address increments by 1 only after two data points are output from the same read address. After a row of valid data is output, the read address is decremented by the row's valid data length. The same row of data is then output in the same manner, completing the upsampling of a row of feature map data. This process can be repeated to complete the upsampling of the entire feature map data.

[0097] Experimental results

[0098] 1) Network Quantization

[0099] The open-source drone intrusion dataset used in this paper consists of 1057 images of real, defective drones collected from drone manufacturers. YOLOv7-Tiny network training was implemented in the Pytorch deep learning framework, using a binary cross-entropy loss function to calculate classification and localization losses, an intersection-over-union function to calculate confidence loss, and a stochastic gradient descent optimizer to iterate parameters. The network was initially trained and then trained using QAT. Table 2 lists the mean average precision (mAP) of the network obtained from full-precision network training, QAT, and post-training quantization (PTQ).

[0100] Table 2 Accuracy of different types of quantitative networks

[0101]

[0102] This paper uses QAT to quantize the UAV intrusion detection network. The quantized network feature graph data and weight data are 8 bits, and the bias is 32 bits. Compared with the full-scale network, the mAP value only drops by 2.8%, while ensuring accuracy and maximizing the compression of the network size. At the same time, compared with PTQ, QAT can improve the model's adaptability to quantization effects during training, thereby greatly optimizing the accuracy loss problem after network quantization. Figure 9 As shown in Figure 2, (a)-(b) list the detection effects of the network model quantified by QAT on drone intrusion targets.

[0103] 2) Achieve results

[0104] Table 3 Detailed resource usage

[0105]

[0106] This invention uses the Xilinx Virtex-7Vc707 as the deployment platform for a drone intrusion detection system. Table 3 analyzes the usage of the hardware accelerator's core resources. The DSP is the primary hardware multiplier resource. This invention designs 256 3×3 multiplication arrays to implement parallel convolution multiplication operations for 16 input channels and 16 output channels. Furthermore, this invention optimizes the DSP design by combining two convolution multiplication operations into a single DSP. Ultimately, the total DSP resources consumed by all multiplication arrays is 1152, with a small portion of DSP resources used for quantization factor multiplication. Block Random Access Memory (BRAM) is the primary on-chip memory resource, primarily used to implement various cache modules. Using on-chip storage resources to cache data that needs to be reused can reduce data transmission latency. Because the architecture designed in this invention is a multiplexed, multi-node, configurable shared architecture, only partial caching of input and output data is performed. Therefore, compared to a fully pipelined design, this invention utilizes relatively fewer BRAM resources. Look-up tables (LUTs) are primarily used to implement combinational logic in hardware design. Flip-flops (FFs) are primarily used to implement sequential logic in hardware and can be used to temporarily store information using register latches.

[0107] In order to evaluate the performance of the system designed by the present invention, the present invention tested the drone intrusion detection algorithm on three platforms: FPGA, CPU and GPU, and compared various aspects of performance.

[0108] The power consumption of the hardware acceleration system is analyzed by Vivado. The operating frequency of the hardware accelerator is 200MHz. The power consumption diagram is as follows: Figure 10 As shown in the figure, dynamic power consumption is 6.177W, primarily due to internal chip resources such as BRAM, DSP, and clocks. Static power consumption is 0.326W, referring to the chip's power consumption in standby mode, primarily due to leakage current in the chip's internal transistors. Overall system power consumption is 6.5W, achieving the goal of low-power design.

[0109] Table 4 compares the performance of this system with that of CPU and GPU platforms. The CPU was tested on an i5-10400F processor and the GPU on a GTX 2070S. The CPU and GPU were tested on a drone intrusion dataset using the Pytorch framework, and runtime and real-time power consumption were recorded. The CPU inference time for a single image was 81.8ms, with a real-time power consumption of 48.58W. The GPU inference time for a single image was 24ms, with a real-time power consumption of 72.18W. While the FPGA's overall throughput was lower than that of the GPU, its power consumption was lower, with an energy efficiency of 27.38 GOPS / W, surpassing both CPUs and GPUs and meeting the requirements for efficient and accurate drone inspection in drone production environments.

[0110] Table 4 Performance comparison of different platforms

[0111]

[0112] Table 5 compares the performance of network hardware accelerators in the present invention with those in the prior art in recent years. The TinyYOLO network accelerator was implemented on the XCZU3EG platform, performing optimization operations such as network quantization, algorithm simplification, and multi-threaded overall acceleration. The YOLOv3-Tiny network accelerator was implemented on the Ultra96V2, quantizing the network and using general matrix multiplication to construct a convolution accelerator. The YOLOv2-Tiny network accelerator was implemented on the ZC706, proposing a data block transmission strategy. The designed accelerator can reuse output data and designs two 14×14 PE matrices for convolution calculations. The YOLOv7-Tiny network accelerator designed by the present invention has better energy efficiency than accelerators in the prior art, achieving a balance between computing performance and power consumption.

[0113] Table 5 Performance comparison with other accelerators

[0114]

Claims

1. A high-energy-efficiency embedded drone intrusion detection system based on FPGA, characterized by: YOLOv7-Tiny was used to train a drone target tracking dataset, and the network was optimized at the algorithmic level using QAT and layer fusion techniques. A hardware accelerator with a reusable multi-node configurable architecture was designed, which implemented different accelerator functions by configuring multiple nodes. This hardware accelerator implemented forward reasoning of the network, performed fine-grained optimization on each hardware module, and designed a dual-pipelined structure within and between layers. Finally, the entire detection system is realized through software and hardware collaborative design; The YOLOv7-Tiny specifically includes: 1) Layer fusion: The BN layer is integrated into the convolutional layer. The convolutional layer calculation formula is shown in formula (1), where Y conv and X conv Represent the calculation results and input data of the convolutional layer respectively, W and b are weights and bias respectively; The calculation formula of the BN layer is shown in formula (2), where Y bn and X bn Represent the calculation results and input data of the BN layer respectively, γ and β represent the scaling factor and bias of the BN layer respectively, ε is the sliding coefficient which is a small positive number, μ and δ are the mean and variance of the input data; Formula (1) converts the calculation result Y of the convolution layer into conv Substitute the input X of the BN layer in formula (2) bn And rearrange them to achieve the fusion of normalization layer and convolution layer. The final new convolution layer formula is shown as follows: Where X' and Y' are the input data and calculation results of the new convolutional layer, respectively, and W' and b' are the weights and biases of the new convolutional layer; 2) QAT model: This model adds pseudo-quantization nodes to the network training to mark the numerical range of the parameters. During training, it simulates quantization to adjust the zero point and scaling factor to reduce the error caused by quantization. After training, the network model is quantized to 8-bit fixed-point numbers by obtaining the optimal zero point and scaling factor. During the network forward inference process, the feature map and weight data are quantized to 8-bit width. The feature map data is quantized to the value range of (0, 256), and the weight data is quantized to the value range of (-128, 127). The scaling coefficient S and zero point Z obtained by QAT are used to map the floating point number X to the fixed point number q as shown in formula (4); X=S(qZ) (4) Substituting formula (4) into (3) yields the following formula: Among them, since the weight is symmetrically quantized, the zero point Z of the weight can be obtained W =0; take S b =S W S X , the bias adopts symmetric quantization and takes Z Y =0, we can get the following formula; Among them only is a floating point number; S W and S X The scaling ratio is 8 bits, so S W S X The scaling ratio is 16 bits. In the QAT model, the bias is calculated with a 32-bit data width, and the scaling is done with S W S X To replace S b Using a biased number will result in a loss of half the information.

2. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 1 is characterized in that: After training with the drone intrusion dataset, the weights, biases, and quantization factor parameter data of the network model are rearranged and saved in the SD card in binary file format. Before drone target tracking detection, the SD card controller reads the weights, biases, scaling factors, and zero points from the SD card to the preset DDR memory offset address through the hardware interface of the SD card slot. After a single sector is read, the SD card will send an SD_Done interrupt signal to the CPU and start reading the next sector. The software-side CPU will first configure each module through the advanced extensible interface AXI bus and preset the configuration parameters of the accelerator on the software side. The CPU configures the hardware accelerator through the AXI interface based on these parameters, thereby realizing forward reasoning of different network layers in the drone intrusion detection network. The accelerator is connected to the SD card through the AXI Stream interface to read and write data. Data interaction between the accelerator and DDR memory is achieved by passing the address instruction information and data transmission length of the DDR memory to AXIDMA. After the hardware accelerator is executed once, it sends a Task_Done interrupt signal to the CPU, and the CPU reconfigures the parameters of the hardware accelerator. This process is repeated until the entire network reasoning is finally completed. Based on the results of network reasoning, the CPU uses the non-maximum suppression (NMS) algorithm to obtain the coordinate position of the UAV target tracking.

3. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 2 is characterized in that: The CPU calculates the data transmission parameters based on the parallelism of the accelerator. In the next cycle, the CPU configures the hardware accelerator to receive mode through the AXI interface and calculates the transmission parameters for the next cycle. The feature map data, weight data, and bias data are transferred from the DDR memory to the on-chip cache of the accelerator through AXIDMA. After the hardware accelerator receives the data, it sends an interrupt to the CPU to inform the CPU that the reception is complete. After receiving the interrupt signal, the CPU starts the next cycle. The CPU configures the hardware accelerator to receive and inference mode. The hardware accelerator adopts a double-buffer structure. One input feature map data cache is used for the forward reasoning operation of the current cycle, and the other input feature map data cache is used to prepare the input feature map data for the next cycle. When the hardware accelerator completes data reception and data forward reasoning, it also sends an interrupt signal to the CPU to indicate that the execution is completed. After the input channel is cycled through and the calculation of the first batch of output data is completed, the CPU configures the hardware accelerator to send mode, sends the output feature map cache data to the DDR memory, and then starts the next cycle until the feature map data of all output channels is calculated, and the reasoning of the network layer is completed.

4. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 3 is characterized in that: The accelerator first configures the control module, which can analyze the network layer type based on the information and generate enable control signals for other sub-modules; a dual-input cache mechanism is adopted, and the cache selector is used to enable the two input feature map cache spaces to alternately write and calculate data; the filling module is controlled by the filling enable signal, which is generated by the control module according to the configured filling value. When the signal is pulled high, the zero-point value of the quantized feature map data is output, and when the signal is pulled low, the data output of the input feature map cache space is read; the feature map data needs to subtract the quantized zero-point value and then select the data path according to the configuration information of the convolution type. If it is a 1×1 convolution, the data is directly input into the convolution module for convolution operation, and if it is a 3×3 convolution, the data needs to pass through the row buffer to form a 3×3 sliding window data and then be sent to the convolution module for convolution operation. The row buffer pre-caches the data to be used; the data stream output by the selector is input into the convolution processing unit and LeakyReLU PE for calculation.

5. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 4 is characterized in that: The convolution module reads weights and feature map data of different arrangements according to the convolution type selector value and performs convolution operation. After the calculation, it will output two results. One is the value after convolution calculation without activation function activation, which is suitable for the convolution layer type of the detection layer in the drone intrusion detection network; the other is the value after convolution calculation after activation function activation, which is suitable for other convolution layer types in the model; It also includes inter-layer pipeline structures of convolution modules and upsampling modules, and convolution modules and downsampling modules, realizing inter-layer and intra-layer dual pipeline structures, and selecting the output data stream through the acceleration type selector; the drone intrusion detection network splices the feature maps of different channels together through the splicing layer; in hardware implementation, after quantization calculation, the data is transferred to the DDR memory with a preset offset address to realize data splicing, and finally the result is written into the output feature map cache space.

6. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 5, characterized in that: The convolution module's input feature map data first needs to determine the convolution kernel size. 3×3 convolution data must first be converted into 3×3 sliding window data through a row buffer, while 1×1 convolution is directly input into the multiplication array for calculation. In the convolution calculation, two convolution multiplication operations are integrated into a single DSP, and the DSP design is optimized. The weight data of different output channels can share the feature map data of the same input channel to calculate the convolution multiplication results of two different output channels. The upper 9 bits of the DSP's A port are sign-extended and filled with the weight of output channel X, and the lower 18 bits are sign-extended and filled with the weight of output channel Y. The B port is sign-extended and filled with the 9-bit data of the input feature map. The final result of multiplying 8-bit wide data by 9-bit wide data is stored as 17-bit wide data. Therefore, the C output port takes the lower 17 bits to output the convolution multiplication result of channel Y, and then takes the next 17 bits as the convolution multiplication result of output channel X.

7. The high-energy-efficiency embedded UAV intrusion detection system based on FPGA according to claim 6 is characterized in that: The data stream calculated by the convolution module is first written into the data cache space in the upsampling module, and image upsampling is achieved by controlling the read address of the data cache space. After the upsampling module starts working, it counts the rows and columns. When two data are output from the same read address, the read address will be increased by 1. After a row of valid data is output, the read address will be subtracted from the row's valid data length, and then another row of data identical to the previous row will be output in the same way. Repeating this operation completes the upsampling operation of the complete feature map data.

Images