A target tracking method and system based on FPGA

Abstract

The application discloses a target tracking method and system based on FPGA, and the method comprises the following steps: performing inverse Bayer operation processing according to the Bayer interpolation principle to obtain a data set of a Bayer format image (RAWDATA) and training a model by using the data set; designing a Pynq_net network model with configurable parallelism for ARM+FPGA SoC; designing a hardware accelerator according to the principle of two-dimensional parallelism of convolution kernels inside and between the convolution kernels in a convolution layer, modeling the hardware accelerator and internal resources and delay of the FPGA, and seeking the consistency of interlayer delay by using design space exploration under the condition that the internal resources of the FPGA used are not over the standard, so that the lowest delay is achieved; collecting RAWDATA image information by configuring an image sensor and placing the RAWDATA image information in the FPGA hardware accelerator for inference calculation; and using a steering gear holder to be communicatively connected with the FPGA module, so that the steering gear holder and the FPGA module cooperate to identify a target position and control the image sensor to track the target in real time. The application can realize fast detection of a target in motion and accurate analysis of a real-time running track.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target tracking method and system based on FPGA. Background Technology

[0002] With the increasing maturity of deep learning technology, deep learning-based target tracking has attracted widespread attention from researchers both domestically and internationally. Moving object detection and tracking is an important application of computer vision in sports. In ball games, the rapid detection of the ball and the accurate analysis of its real-time trajectory can help coaches make real-time strategy analysis and tactical preparations, providing a basis for athletes' future training and simulated matches.

[0003] Currently, real-time object detection networks on CPUs and GPUs face challenges such as low real-time performance, high power consumption, and high cost. With the continuous development of integrated circuit technology, Field Programmable Gate Arrays (FPGAs) contain increasingly more computing, logic, and storage resources, making them suitable for implementing CNN hardware accelerators. Designing and implementing an FPGA-based CNN hardware accelerator typically requires less time than ASIC-based accelerators, making FPGA-based CNN hardware accelerators the preferred choice in many cases. Furthermore, FPGAs consume less power than CPUs and GPUs when performing the same computational workload. Therefore, using FPGAs to implement CNN hardware accelerators for high-speed, low-power ball tracking tasks offers certain advantages. The winner of the DAC-SDC 2020 System Design Competition proposed ultra_net, which optimizes task processing based on the architectural characteristics of FPGAs, making it suitable for ball detection. In terms of algorithm design, UltraNet reduces the number of parameters in the neural network by employing low-bit quantization technology, resulting in fewer parameters and less storage space required for each parameter. This allows all parameters to be placed in the on-chip BRAM of the FPGA, reducing the overhead caused by frequent parameter loading / unloading during acceleration. In terms of hardware design, it builds an efficient pipelined structure based on FPGA resources, placing all layers of the convolutional neural network on-chip, enabling all layers to be computed in parallel in a pipelined manner. It also designs efficient low-bit matrix computation units, achieving high parallelism under limited resource constraints. Existing object detection systems often use JPEG images as input for object detection and tracking systems. The resulting complex encoding and decoding processes consume significant hardware resources and cause additional latency overhead, severely limiting the inference speed of the model during detection. Summary of the Invention

[0004] Addressing the shortcomings of existing technologies, unlike the common use of JPEG images which requires complex encoding and decoding processes that consume a lot of hardware resources and cause additional latency overhead, this invention utilizes the parallel processing and low power consumption characteristics of FPGAs. It uses image sensors to acquire RAWDATA images to achieve end-to-end processing, eliminating the encoding and decoding process and de-pixelation operation. It proposes an FPGA-based target tracking method and system that achieves accurate analysis of the real-time trajectory of moving targets such as balls with low power consumption. For example, it can help coaches make real-time strategy analysis and tactical preparations in ball games, providing a basis for athletes' future training and simulated combat.

[0005] The technical solution of the present invention is as follows:

[0006] A target tracking method based on FPGA, characterized in that, unlike the JPEG encoding and decoding process, it directly uses a sensor to input RAWDATA format images for end-to-end processing on the FPGA side. The method includes the following steps:

[0007] Step 1: Dataset Creation: Record video using a video input device; process the captured video into an image sequence, label the obtained images containing the target, and perform inverse Bayer operation based on the Bayer interpolation principle to obtain Bayer format images RAWDATA as the original dataset;

[0008] Step 2: Design a Pynq_net network model with configurable parallelism for the ARM+FPGA SoC architecture. Design a hardware accelerator based on the principle of parallelism in two dimensions: within the convolutional kernel and between convolutional kernels in the convolutional layer. Model the accelerator with the internal resources and latency of the FPGA. Under the constraint that the internal resources of the FPGA used do not exceed the limit, use design space exploration to seek the consistency of inter-layer latency, thereby achieving the lowest latency.

[0009] Step 3: Place the raw image data of the target acquired by the image sensor on the FPGA hardware accelerator for inference calculation;

[0010] Step 4: The servo gimbal and FPGA module communicate with each other. The servo gimbal and FPGA module work together to identify the target position and control the image sensor to track the target in real time.

[0011] Further, step 1 specifically involves: processing the image using the inverse Bayer operation principle to obtain the Bayer format image RAWDATA, and dividing the RAWDATA image into a training set, a validation set, and a test set.

[0012] Furthermore, step 2 specifically involves:

[0013] The configurable parallelism within a convolutional kernel is represented by SIMD, and the configurable parallelism between convolutional kernels is represented by PE. The configurable parallelism in both dimensions is modeled in relation to the FPGA's internal memory unit BRAM and computing unit DSP.

[0014] BRAM cov =BRAM FIFO +BRAM SWU +BRAM WM (1)

[0015]

[0016]

[0017]

[0018]

[0019] The delay for each layer is represented as follows:

[0020] Latency SWU =(in_row+2)·[in_col+2+((in_col+2-k) / s+1)*(k*k)] (6)

[0021] Latency WM =((k*k*in_ch) / SIMD)*((out_ch) / PE)*(in_row*in_col) (7)

[0022] BRAM cov This represents the BRAM consumed by the convolutional layer, including the BRAM consumed for storing inter-layer results. FIFO BRAM consumed by the sliding window unit SWU BRAM consumed by matrix convolution operations WM DSP WM Indicates the DSP consumed by matrix convolution operations; Latency SWU Latency represents the delay of a sliding window unit. WM The decrement represents the latency of the matrix convolution operation; Depth represents the depth of the inter-layer FIFO; in_ch represents the input channel; out_ch represents the output channel; in_bit represents the input data bit width; out_bit represents the output data bit width; w_bit represents the weight bit width; k represents the convolution kernel size; s represents the stride; in_row represents the height of the feature map; and in_col represents the width of the feature map.

[0023] Furthermore, step 3 specifically involves:

[0024] Step 3-1: Connect the image sensor to the peripherals of the FPGA hardware platform, and configure the image sensor to work in RAWDATA data pixel format to obtain better image quality;

[0025] Step 3-2: Construct an efficient dataflow architecture based on FPGA hardware platform resources. Based on the Vivado HLS high-level synthesis tool and the computational characteristics of convolutional neural networks, use low-bit matrix computation units to maximize parallelism in the designed Pynq_net network model with configurable parallelism under the limited resource constraints of the FPGA hardware platform. Then, optimize the designed circuit and perform functional simulation. Finally, package the designed acceleration circuit into an IP core.

[0026] Step 3-3: Use Vivado or Vitis for design, call the accelerator IP core, design the corresponding interface scheme, and finally build the SoC system. The image reading and other functions are completed on the ARM processor of the SoC, and the target detection is completed on the FPGA platform of the SoC.

[0027] Steps 3-4: Develop the driver program by writing the corresponding driver program in Python to call the accelerator IP core. Based on the selected FPGA hardware platform, implement the design of the configurable convolutional neural network accelerator and complete system verification and testing.

[0028] Furthermore, step 4 specifically involves:

[0029] The camera's angle is controlled in both horizontal and vertical directions using servos. The angle of the servos in both directions is adjusted according to the target's position in the image to keep the target in the center of the camera's field of view, thus enabling real-time target tracking.

[0030] Accordingly, the present invention also provides an FPGA-based target tracking system, characterized in that the system comprises a camera, a two-dimensional servo gimbal, a SoC main control subsystem, a memory, and a power supply system;

[0031] The camera is used to capture Bayer format RAWDATA images containing the target;

[0032] The two-dimensional servo gimbal is communicatively connected to the FPGA module. The servo gimbal and the FPGA module work together to identify the target position and control the image sensor to track the target in real time.

[0033] The SoC main control subsystem includes an image acquisition module, a target detection module, a target tracking module, and a data transmission module, which are used to implement steps 1 to 4 in the above method;

[0034] The memory is used to store images and provide runtime space for programs.

[0035] Furthermore, the image acquisition module is deployed in the FPGA part of the SoC; it configures the image sensor according to the settings, acquires RAWDATA format images, and sends them to the data transmission module.

[0036] Furthermore, the target detection module is deployed on the FPGA part of the SoC; it executes the target detection logic function according to the settings, and obtains the position information of the target to be tested based on the image acquired by the image acquisition module.

[0037] Furthermore, the target tracking module includes an ARM portion deployed on the SoC, which is connected to the servo gimbal; it executes a set program, namely, by analyzing the target coordinate information obtained by the target detection module to obtain the angle to be adjusted by the servo, converting it into a data frame and sending it to the servo, and controlling the image sensor to follow the target in real time.

[0038] Furthermore, the data transmission module is deployed on the FPGA portion of the SoC; efficient data interaction between the image acquisition module, target detection module, and target tracking module is achieved through DMA.

[0039] Compared with existing technologies, the beneficial effects of this disclosure are: This invention proposes a target tracking method and system based on FPGA. By directly processing RAWDATA images, it achieves higher detection speed, less hardware resource usage, and lower power consumption compared to RGB images. Adopting a hardware-software co-processing approach, it uses an ARM+FPGA SoC to build the hardware platform, and a simplified, customized, and quantized Pynq_net network as the software algorithm. Employing a dataflow architecture, it significantly improves the target detection neural network in terms of latency, power consumption, and cost. Experiments using the method of this invention with an OV5640 sensor and a PYNQ-Z2 hardware platform show an average IOU of 0.853 in the designed Pynq_net network, achieving 53 frames per second (FPS) at 3.0W power consumption. Attached Figure Description

[0040] Figure 1 This is a hardware framework diagram of the FPGA-based target tracking system of the present invention.

[0041] Figure 2 This is a flowchart of the FPGA-based target tracking method of the present invention.

[0042] Figure 3 This is the Pynq_net network model structure in the FPGA-based target tracking method of this invention. Detailed Implementation

[0043] The present invention relates to a target tracking method and system based on FPGA, using ball sports as a specific example. It employs an image sensor to acquire real-time RAW data images of the ball sports during a game. By detecting the ball sports in the acquired RAW data images, the coordinate information of the ball sports is obtained, and this information is converted to determine the angle that the servo motor needs to adjust. This angle is then converted into a data frame and sent to the servo motor to achieve target tracking. This method has low hardware requirements but achieves excellent results, helping coaches to conduct real-time strategy analysis and tactical preparation, and providing a basis for athletes' future training and simulated matches.

[0044] The FPGA-based target tracking method of this invention, see [link to relevant documentation]. Figure 2 As shown, it includes the following steps:

[0045] Step 1: Dataset Creation. Record 20-30 minutes of actual table tennis match video from multiple scenes at different directions and distances using a video input device (e.g., Sony IMX582). Use a Python program to process the captured video into an image sequence and annotate it using the image annotation tool LabelImg. Perform inverse Bayer interpolation on the captured images to obtain Bayer format images (RAWDATA). If image resizing is required, use OpenCV's region interpolation method to directly resize the images or crop them centered on the ball in the image. Divide the resulting images into training, validation, and test sets.

[0046] Step 2: Design a Pynq_net network model with configurable parallelism for the ARM+FPGA SoC architecture. To address the resource constraints of the FPGA board, a common network structure consisting of a backbone network and a detection head is used. The backbone network utilizes a common convolution-normalization-activation module as its model structure, with 3x3 kernels. After sequentially adding convolution-normalization-activation modules to the backbone network, the model undergoes low-bit quantization-aware training. This involves constructing a conventional floating-point computation graph and inserting pseudo-quantization modules at corresponding positions in the convolution computation graph to quantize weights and activation values ​​to 4-bit integers. The trained network model is then tested, and its IOU is recorded. Synthesis is performed using Vivado HLS. A hardware accelerator is designed based on the simultaneous parallel computation of both the convolution kernel's internal and inter-kernel dimensions. The configurable parallelism of these two dimensions is modeled in relation to the FPGA's internal resources and latency. Under the constraint that the FPGA's internal resources are not exceeded, the consistency of inter-layer latency is explored within the design space to achieve the lowest possible latency. The resource usage and latency data from the synthesis report determine whether the current network structure meets the hardware requirements. The customized Pynq_net network model structure is as follows: Figure 3 As shown.

[0047] Step 3: Acquire RAWDATA image information during a table tennis match using an image sensor and perform inference computation on the designed FPGA hardware accelerator. Connect the image sensor to the FPGA hardware platform peripherals and configure it to operate in RAWDATA data pixel format (e.g., if using an OV5640 sensor and PYNQ-Z2, connect the OV5640 sensor to the PYNQ-Z2 via PMODA and PMODB, and use the IIC driver to configure the OV5640 sensor). Use the Vivado HLS tool to build an efficient dataflow architecture based on FPGA resources. Based on the Vivado HLS high-level synthesis tool and the computational characteristics of convolutional neural networks, design a Pynq_net network model with configurable parallelism, using low-bit matrix computation units to maximize parallelism under the limited resource constraints of the PYNQ-Z2. Then, optimize the designed circuit and perform functional simulation. Finally, package the designed acceleration circuit into an IP core. Call the accelerator IP core through the Vivado tool, design the corresponding interface scheme, and finally build the SOC system. The corresponding driver program is written in Python to call the accelerator IP core for driver development. The design of the above-mentioned configurable convolutional neural network accelerator is implemented on the selected FPGA development platform, and the system verification and testing are completed.

[0048] Step 4: After post-processing the data, it is sent to the servo motor for tracking. The servo motor is used to control the camera's angle in both the horizontal and vertical directions. Since the servo motor's three signal lines—power line Vcc, ground line GND, and control signal line (the servo motor's control signal is also a PWM signal)—may be incompatible with the FPGA hardware platform's dual signal lines (one for receiving commands and the other for sending commands). A servo motor adapter board is used to solve this problem. The servo motor's angle in both the horizontal and vertical directions is adjusted according to the ball's position in the image to keep the ball centered in the image (i.e., at the center of the camera's field of view). The servo motor gimbal works with the FPGA module to identify the ball's position and control the image sensor to track the ball in real time.

[0049] Accordingly, the present invention also provides an FPGA-based target tracking system, such as Figure 1As shown, the system is characterized by comprising a camera, a two-dimensional servo motor, a SoC main control subsystem, a memory, and a power supply system. The camera is used to capture images containing the target, conforming to the RAWDATA requirements of step 1; the memory is used to store the images and provide runtime space for the programs involved in steps 3-4; the two-dimensional servo motor is used to execute the tracking actions involved in step 4; and the power supply system ensures that the entire system executes according to the process of steps 3-4. The SoC main control subsystem implements the logic generated in steps 1-3 and executes the programs involved in steps 3-4, mainly including an image acquisition module, a target detection module, a target tracking module, and a data transmission module.

[0050] The image acquisition module is deployed in the FPGA part of the SoC; the image sensor is configured according to the settings in step 1, and RAWDATA format images are acquired and sent to the data transmission module.

[0051] The target detection module is deployed on the FPGA part of the SoC; it executes the target detection logic function according to the settings in steps 1 to 3, and obtains the position information of the target to be detected based on the image acquired by the image acquisition module.

[0052] The target tracking module, including the ARM portion deployed on the SoC, connects to the servo gimbal. It executes the program set in step 4, which involves analyzing the target's coordinate information obtained from the target detection module to determine the angle the servo needs to adjust, converting this into a data frame, sending it to the servo, and controlling the image sensor to follow the target in real time.

[0053] The data transmission module is deployed in the FPGA part of the SoC; it enables efficient data interaction between the image acquisition module, target detection module, and target tracking module through DMA.

[0054] In summary, this invention discloses a target tracking method and system based on FPGA. It uses an image sensor to acquire RAWDATA images as system input, performs target detection processing based on an FPGA hardware platform, and uses a servo gimbal to communicate with the FPGA module. The servo gimbal and the FPGA module work together to identify the position of the moving target and control the image sensor to track the target in real time.

[0055] The experiment compared the frame rate per second (FPS) and power consumption of this method with those of two other hardware platforms. The experimental results are shown in Table 1. The FPS of this method is higher than that of the other two methods, and the power consumption is lower than that of the other two methods.

[0056] Table 1. Detection results for different hardware platforms

[0057]

[0058] *Note: 22 represents the maximum power at startup, and 5 represents the power during network inference.

[0059] The PC power is calculated by subtracting the standby voltage from the voltage at the time of testing.

Claims

1. A target tracking method based on FPGA, characterized in that, Unlike the JPEG encoding and decoding process, this method directly uses sensors to input RAW DATA format images for end-to-end processing on the FPGA side. The method includes the following steps: Step 1: Dataset Creation: Record video using a video input device; process the captured video into an image sequence, label the obtained images containing the target, and perform inverse Bayer operation based on the Bayer interpolation principle to obtain Bayer format images RAWDATA as the original dataset; Step 2: Design a configurable parallel Pynq_net network model for the ARM-FPGA SoC architecture. Based on the principle of parallelism in two dimensions—within and between convolutional kernels—design a hardware accelerator. Model this accelerator in conjunction with the FPGA's internal resources and latency. Under the constraint that the FPGA's internal resources are not exceeded, use design space exploration to seek consistency in inter-layer latency, thereby achieving the lowest possible latency. Step 2 specifically includes: The configurable parallelism within a convolutional kernel is represented by SIMD, and the configurable parallelism between convolutional kernels is represented by PE. The configurable parallelism in both dimensions is modeled in relation to the FPGA's internal memory unit BRAM and computing unit DSP. = + + (1) = · (2) = · (3) = · ∙(EP +1) (4) = PE· + 2*(PE + 1) (5) The delay for each layer is represented as follows: = ( row + 2) [in_col + 2 + ((in_col + 2 - k) / s +1)*(k*k)] (6) = ((k*k* ) / SIMD)*(( ) / PE)*( row*in_col) (7) in This represents the BRAM consumed by the convolutional layer, including the memory consumed for storing results between layers. The amount consumed by the sliding window unit The cost of matrix convolution operations ; This represents the DSP consumed by the matrix convolution operation; Indicates the delay of the sliding window unit. Indicates the delay of matrix convolution operation; Indicates the depth of the inter-layer FIFO. Indicates the input channel. Indicates the output channel. Indicates the input data bit width. Indicates the output data bit width. The bit width indicating the weight. This represents the kernel size, and s represents the stride. row represents the height of the feature map. Indicates the width of the feature map; Step 3: Place the raw image data of the target acquired by the image sensor on the FPGA hardware accelerator for inference calculation; Step 4: The servo gimbal and FPGA module communicate with each other. The servo gimbal and FPGA module work together to identify the target position and control the image sensor to track the target in real time.

2. The FPGA-based target tracking method according to claim 1, characterized in that: Step 1 specifically involves: processing the image using the inverse Bayer interpolation principle to obtain Bayer format RAW DATA images, and then dividing the RAW DATA images into training, validation, and test sets.

3. The FPGA-based target tracking method according to claim 1, characterized in that: Step 3 specifically involves: Step 3-1: Connect the image sensor to the peripherals of the FPGA hardware platform, and configure the image sensor to work in RAW DATA pixel format to obtain better image quality; Step 3-2: Construct an efficient dataflow architecture based on FPGA hardware platform resources. Based on the VivadoHLS high-level synthesis tool and the computational characteristics of convolutional neural networks, use low-bit matrix computation units to maximize parallelism in the designed Pynq_net network model with configurable parallelism under the limited resource constraints of the FPGA hardware platform. Then, optimize the designed circuit and perform functional simulation. Finally, package the designed acceleration circuit into an IP core. Step 3-3: Use Vivado or Vitis for design, call the accelerator IP core, design the corresponding interface scheme, and finally build the SoC system. The image reading and other functions are completed on the ARM processor of the SoC, and the target detection is completed on the FPGA platform of the SoC. Steps 3-4: Develop the driver program by writing the corresponding driver program in Python to call the accelerator IP core. Based on the selected FPGA hardware platform, implement the design of the configurable convolutional neural network accelerator and complete system verification and testing.

4. The FPGA-based target tracking method according to claim 1, characterized in that: Step 4 specifically involves: The camera's angle is controlled in both horizontal and vertical directions using servos. The angle of the servos in both directions is adjusted according to the target's position in the image to keep the target in the center of the camera's field of view, thus enabling real-time target tracking.

5. A target tracking system based on FPGA, characterized in that, The system comprises a camera, a two-dimensional servo motor, a SoC main control subsystem, a memory, and a power supply system. The camera is used to capture Bayer format RAW DATA images containing the target; The gimbal of the two-dimensional servo motor is communicatively connected to the FPGA module. The servo motor gimbal and the FPGA module work together to identify the target position and control the image sensor to track the target in real time. The SoC main control subsystem includes an image acquisition module, a target detection module, a target tracking module, and a data transmission module, used to implement the method described in any one of claims 1, 2, 3, and 4; The memory is used to store images and provide runtime space for programs.

6. The FPGA-based target tracking system according to claim 5, characterized in that, The image acquisition module is deployed in the FPGA part of the SoC; it configures the image sensor according to the settings, acquires RAW DATA format images, and sends them to the data transmission module.

7. The FPGA-based target tracking system according to claim 5, characterized in that, The target detection module is deployed in the FPGA portion of the SoC; The target detection logic function is executed according to the settings, and the position information of the target to be tested is obtained based on the image acquired by the image acquisition module.

8. The FPGA-based target tracking system according to claim 5, characterized in that, The target tracking module includes an ARM portion deployed on the SoC, which is connected to the servo gimbal; it executes a set program, that is, by analyzing the target coordinate information obtained by the target detection module, it obtains the angle to be adjusted by the servo, converts it into a data frame and sends it to the servo, and controls the image sensor to follow the target in real time.

9. The FPGA-based target tracking system according to claim 5, characterized in that, The data transmission module is deployed on the FPGA part of the SoC; it enables efficient data interaction between the image acquisition module, the target detection module, and the target tracking module through DMA.

Images