Target tracking method and system based on ZYNQ platform

Abstract

The invention discloses a target tracking method and system based on a ZYNQ platform, and the method comprises the steps: training and optimizing a neural network model at a PC end to generate a weight and a table look-up file, downloading the weight and the table look-up file to an SD card, and transmitting the weight and the table look-up file to the ZYNQ platform; the camera captures a video stream and then carries out data type conversion and shunt processing; the PS end realizes feature map generation by configuring a convolution accelerator and DMA, and performs activation function processing, maximum pooling, bounding box decoding and non-maximum suppression on output data to finally obtain a category label and bounding box coordinates; the PS end calculates an error through a PID algorithm to generate a control signal, and adjusts a pan-tilt camera in real time, so that the target is always kept in the middle; the system superposes a target detection result to a video picture in real time and outputs the result to a display device, and clear target tracking feedback is provided. According to the invention, efficient and energy efficiency optimized edge calculation target tracking is realized on the ZYNQ platform, and the method is suitable for various target tracking scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and in particular to a target tracking method and system based on a ZYNQ platform. Background Art

[0002] At present, target detection and tracking systems have been widely used in scenarios that require edge computing, such as intelligent monitoring, autonomous driving, drone vision, etc., and these devices have extremely strict requirements on power consumption and volume. Existing target detection systems usually rely on high-performance GPUs to run neural network models (such as YOLO), which can achieve high accuracy and real-time performance when computing resources are sufficient. However, the high power consumption of GPUs will significantly shorten the battery life, making it difficult to meet the needs of long-term, uninterrupted operation of the system. At the same time, the large size of GPUs limits their deployment in space-constrained devices. Therefore, to solve these problems, edge computing urgently needs a target detection solution that can run efficiently in resource-constrained environments to achieve lower power consumption and compact design.

[0003] On the other hand, the ZYNQ platform, with its low power consumption and high flexibility, has become an ideal choice for edge computing-based target detection systems. However, ZYNQ has limited computing power and storage resources, and directly deploying complex neural network models (such as YOLO) on ZYNQ will face multiple challenges. First, the algorithm model based on convolutional neural networks has a large number of convolutional layers and complex operations. ZYNQ's FPGA resources are difficult to support floating-point calculations and large-scale weight storage, and are prone to insufficient resources. Secondly, the model needs to process video streams in real time. The computing power of FPGA is limited, and the lack of parallel structure will lead to a decrease in computing speed and cannot meet real-time requirements. In addition, the convolution calculation in the model requires a large number of multiplication and addition operations. If there is a lack of a dedicated row buffer mechanism and parallel design, the computing efficiency will be further limited and the delay will be increased. The high resource consumption and power consumption of floating-point operations also limit the feasibility of direct deployment. FPGA is not good at high-bitwidth floating-point operations, and the additional power consumption will aggravate the heat dissipation problem. The multi-layer feature maps generated by the model are of different sizes, occupying a large amount of on-chip storage resources, and the lack of dynamic block processing will lead to storage bottlenecks. Ultimately, these bottlenecks affect inference accuracy and model performance, making it difficult to achieve efficient and real-time object detection and tracking.

[0004] Therefore, how to efficiently run the neural network model on the ZYNQ platform and realize the target tracking system based on edge computing has become a challenge for existing technologies. Summary of the invention

[0005] The main technical problem solved by the present invention is to provide a target tracking method and system based on the ZYNQ platform, which realizes multi-dimensional parallel processing, quantitative inference optimization and Ping-Pong Buffer parallel transmission, ensures the continuity and accuracy of data processing, and can improve the speed and accuracy of target tracking and reduce the power consumption of the system. In particular, by optimizing DSP48 resources, adopting a ternary tree addition structure and a flexible row buffer mechanism, high throughput and calculation accuracy are ensured, and the problem of insufficient on-chip storage is effectively solved, making it more real-time and efficient when running neural network models on resource-constrained edge computing devices.

[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a target tracking method based on the ZYNQ platform, characterized in that a neural network model is efficiently run to realize a real-time target tracking system based on edge computing, and the method comprises the following steps:

[0007] Step 1: Train and optimize the neural network model on the PC, including weight quantization and operation simplification, generate 8-bit integer weight files and sigmoid activation function lookup table files, and download them to the SD card to adapt to ZYNQ deployment;

[0008] Step 2: Use the camera to capture video stream data, convert the data type of the video stream data, and divide it into two processing paths: one path is transmitted to the video direct memory access module (VDMA) and cached in the DDR4 memory, and the other path enters the preprocessing module and is transmitted to the VDMA for caching again after preprocessing;

[0009] Step 3: Transfer the weight and lookup table files to the PS side of ZYNQ through the SD card. The PS side configures the convolution accelerator module and direct memory access (DMA) through the AXI4-Lite bus to realize the input image data of the convolution accelerator and the output feature map data after the convolution operation;

[0010] Step 4: After the PS receives the feature map data output by the convolution accelerator, it performs activation function processing, maximum pooling layer processing, and bounding box decoding in sequence, and finally class prediction and non-maximum suppression to obtain the class label and bounding box coordinates (x, y, w, h);

[0011] Step 5: The PS side calculates the difference between the target center and the image center, processes these errors through the PID algorithm and generates control signals to adjust the camera on the 2D gimbal in real time;

[0012] Step 6: The bounding box drawing module overlays the target detection results onto the video screen in real time, and processes them through the display driver module, converting the video stream data into TMDS signals and outputting them to the display device, providing clear target tracking feedback.

[0013] Furthermore, step 2 is specifically as follows:

[0014] The camera is used to collect video stream data in real time. These data are converted into AXI4-Stream data streams through the video stream data type conversion module and copied into two outputs. One path is transferred to the DDR4 memory for caching through VDMA; during this process, the VDMA module configured on the PS side sets the address, data width, resolution and line interval parameters of the frame buffer. The other path processes the data through the video preprocessing module on the PL side, including de-mosaicing, gamma correction, resolution adjustment and scaling operations. Finally, through VDMA, the preprocessed video data is cached to the DDR4 memory, providing a continuous and stable input data source for the convolution accelerator module.

[0015] Furthermore, step 3 is specifically as follows:

[0016] The PS side configures the 7 registers and DMA module of the convolution accelerator through the AXI4-Lite bus to achieve comprehensive control of weight data reading, pooling operations and convolution operations, and transmits the video data cached in the DDR4 memory to the convolution accelerator module to generate feature maps.

[0017] Among the 7 registers, the first register is the control register. The first 5 bits are control words, which respectively control the reception of IFM data, Weight data, Bias data, the transmission of OFM data, and the enablement of convolution operations. The last five bits are status words, which respectively indicate whether the current convolution result needs to be pooled, whether it is the first or last convolution, the memory area pointed to by the Ping-pongBuffer pointer, and whether the current data is valid. The convolution accelerator has 5 types of tasks. After each task is enabled, it needs to be manually reset externally. After the task is completed, the convolution accelerator triggers the interrupt on the PS side by pulling up the AP_DONE signal, informing the CPU to complete the currently set task, thereby realizing feedback.

[0018] The second register is used to set the specific parameters of the convolution operation. The first 16 bits indicate the length of the data in the cache to be convolved, and the next 12 bits indicate how many data there are in a row in the current feature map. The last three bits indicate the type of row cache corresponding to the current feature map.

[0019] The third register is used to set the Scaling Factor in the quantization parameter.

[0020] The fourth register is used to set the input zero point and output zero point in the quantization parameter.

[0021] The fifth, sixth, and seventh registers provide some address information required by the convolution accelerator.

[0022] Furthermore, the convolution accelerator module optimizes convolution operations through the row buffer mechanism and DSP48 resources, and has the ability to dynamically adjust the row buffer depth to adapt to the requirements of different convolution kernel sizes (such as 3x3, 1x1, etc.) and feature map sizes. The row buffer mechanism also supports parallel expansion strategies, allowing convolution calculations to achieve synchronous parallel operations in the row and column dimensions of the convolution kernel and the feature map dimension; in the parallel calculation of the 3x3 convolution kernel, a 3x3 data window is generated in each clock cycle, and 9 multipliers are used for simultaneous calculation.

[0023] Furthermore, the convolution accelerator module processes large-size feature maps through a rectangular blocking strategy, and only convolves a few rows at a time. The convolution accelerator module adopts a full-pipeline structure design, supports the Ping-Pong Buffer structure, realizes the parallel operation of convolution and data transmission, and ensures that the data processing process is unblocked. The full-pipeline structure design includes a row buffer, a multiplier, an adder tree, and a tightly connected stage of accumulation and quantization. The module has multi-channel processing capabilities, can process multiple input feature map channels at the same time, and adopts a ternary tree structure for efficient channel accumulation.

[0024] Furthermore, the convolution accelerator module uses INT8 quantization inference technology to convert 32-bit floating-point weights and activations to 8-bit integers. The product of INT8 values ​​and INT8 values ​​is stored in INT16, and the product is accumulated. The accumulated result is stored in INT18. After the accumulation is completed, the INT18 result is scaled and transformed through the following steps and finally converted to INT8:

[0025] Saturate the INT18 result and truncate it to the INT16 range;

[0026] Multiply the obtained INT16 value by the Scale FactorM0 of type UINT16 to get the INT32 result;

[0027] Perform the "Round right to nearest integer" operation to right-shift the INT32 value by 15+Shift Factor bits, and determine the value of the 14+Shift Factor bit in the INT32 result. If the bit is 1, it means that rounding up is required, and 1 is added to the final result. If the bit is 0, no rounding is performed, and the result remains unchanged. After rounding and right-shifting, the INT16 result is obtained again.

[0028] The obtained INT16 result is saturated again and finally converted to INT8 type. In addition, this method adopts a shared multiplication factor design so that two INT8 multiplication operations can be mapped to the same DSP48 unit, and the window data generated by the line buffer mechanism can be transmitted to multiple DSP48 units at the same time to realize parallel convolution calculation.

[0029] Furthermore, step 4 is specifically as follows:

[0030] Step 4-1: The PS side uses the Sigmoid activation function to perform nonlinear processing on the feature map, converting the linear result output by the convolution layer into nonlinearity.

[0031] Step 4-2: Perform max pooling on the feature map data after the activation function, and decode the bounding box coordinates and category information of the target by selecting the maximum value in the local window of the feature map.

[0032] Step 4-3: Perform bounding box decoding on the feature map data processed by the pooling layer and extract the predicted value t of each anchor box x ,t y ,t w ,t h The bounding box offset in the feature map is mapped to the range [0,1] through Sigmoid function processing, and the center coordinates (x, y) in the actual image are obtained based on the coordinates of the grid unit.

[0033] x=(σ(t x )+c x )×s x

[0034] y=(σ(t y )+c y )×s y

[0035] Where σ(t x ),σ(t y ) is the Sigmoid activation of the predicted value, c x 、c y is the coordinate offset of the grid cell, s x 、s y is the size of the grid.

[0036] The scaling factors of width and height are calculated by exponential function and decoded into the width and height dimensions (w, h) in the actual image in combination with the size of the anchor box.

[0037] w=p w ×e tw

[0038] h=p h ×e th

[0039] where t w ,t h The predicted scaling factor, p w 、p h are the width and height of the anchor box.

[0040] Step 4-4: Use the non-maximum suppression (NMS) algorithm to deduplicate bounding boxes, remove duplicate bounding boxes with low confidence, and retain only the boxes with the highest confidence. The NMS algorithm determines the overlap by calculating the intersection over union (IoU) between bounding boxes, and removes bounding boxes with an IoU value greater than 0.5.

[0041] Furthermore, step 5 is specifically as follows:

[0042] The PS side uses the PID control algorithm to calculate the difference (ex, ey) between the center of the image captured by the camera and the center of the target, and generates two sets of PWM signals that are transmitted to the horizontal and vertical rotation servos of the gimbal respectively to adjust their rotation angles so that the target always remains centered.

[0043] Furthermore, step 6 is specifically as follows:

[0044] The VDMA module reads the video stream data from the DDR4 memory and overlays the target detection results on the video screen in real time through the bounding box drawing module. During this process, the PS side transmits the target detection results, including the category label and bounding box coordinates (x, y, w, h), to the bounding box drawing module through the AXI4-Lite bus. Then, after being processed by the display driver module, the video stream data is converted into a TMDS signal and output to the display device.

[0045] Accordingly, the present invention also provides a target tracking system based on the ZYNQ platform, characterized in that the system comprises a PC terminal, a pan / tilt camera module, a ZYNQ platform, a convolution accelerator module and a display module;

[0046] The PC is used to train and optimize the target detection neural network model and generate an 8-bit integer weight file and a lookup table file of the Sigmoid activation function;

[0047] The pan-tilt camera module includes a camera and a two-dimensional pan-tilt, the camera is used to collect video stream data in real time, and the two-dimensional pan-tilt is used to control the horizontal and vertical rotation of the camera respectively to ensure that the camera is always aimed at the target;

[0048] The ZYNQ platform includes a PL side and a PS side, the PL side includes a VDMA, a video stream data type conversion module, a video preprocessing module, a DMA, a convolution accelerator module, a bounding box drawing module and a display driver module, and the PS side includes a hard-core CPU and a DDR4 memory, which are used to implement the above method;

[0049] The convolution accelerator module is used to perform a convolution operation on the video data to generate feature map data, and the output feature map data has local feature information of the input image;

[0050] The display module is used to output video stream data to a display device.

[0051] The beneficial effects of the present invention are as follows: a target tracking method and system based on the ZYNQ platform of the present invention realizes efficient target detection and tracking through multiple steps, and has significant beneficial effects in each step. First, the camera captures the video stream in real time and pre-processes the data through the video processing module. This step ensures the quality and adaptability of the input data, so that subsequent convolution operations can be performed efficiently. Then, through VDMA, the pre-processed video data is cached to the DDR4 memory. This mechanism provides efficient data transmission and caching capabilities, providing a continuous and stable input data source for the convolution accelerator module.

[0052] By training the target recognition neural network model on the PC side, quantizing the weights and simplifying the operations, generating 8-bit integer weight files and lookup files for the Sigmoid activation function, the computational complexity and storage requirements of the model are significantly reduced. The convolution accelerator module significantly improves the computational efficiency of convolution operations, reduces power consumption, and ensures the real-time responsiveness of the system through line buffering mechanism and pipeline parallel processing. The design of the convolution accelerator makes full use of the hardware resources of the FPGA, and optimizes the DSP48 to achieve efficient multiplication and addition operations in convolution calculations, which greatly improves the inference speed.

[0053] After the target detection is completed, the target tracking module combines the PID control algorithm to dynamically adjust the movement of the two-dimensional pan-tilt head according to the detection results to ensure that the target is always in the center of the screen. The PID controller makes real-time adjustments based on the deviation of the target position, achieving smooth and high-precision target tracking, and effectively improving the system's adaptability in complex environments. Finally, the bounding box drawing module superimposes the target detection results on the video screen in real time, and processes it through the display driver module, converting the video stream data into a TMDS signal and outputting it to the display device, providing clear target tracking feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a structural schematic diagram of a target tracking system based on the ZYNQ platform of the present invention;

[0055] Figure 2 It is a flowchart of a target tracking method based on the ZYNQ platform;

[0056] Figure 3 It is a structural diagram of a camera capturing real-time video stream data;

[0057] Figure 4 It is a schematic diagram of the structure of the convolution accelerator scheduling;

[0058] Figure 5 It is a structural diagram of the display module and the pan-tilt camera module. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0060] See also Figure 1 and Figure 2 , the embodiment of the present invention includes:

[0061] A target tracking method and system based on the ZYNQ platform. The system proposes a solution to the target detection neural network model through hardware acceleration and weight quantization technology to address the problem of limited resources and high real-time requirements of edge computing devices. The system design mainly includes the training and optimization of the target detection neural network model on the PC side, image acquisition, convolution acceleration, target detection and pan-tilt control on the ZYNQ hardware platform, forming a complete closed-loop target tracking system. Through this solution, the system can achieve efficient target tracking with limited hardware resources and ensure real-time performance. Figure 1 The overall hardware architecture of the system is shown. Figure 2 The workflow of the system is demonstrated.

[0062] The hardware architecture of the system is as follows Figure 1 As shown in the figure, it consists of a PC, a gimbal camera module, a display device and a ZYNQ platform. First, the target detection neural network model is trained and optimized on the PC, and then the generated weights and lookup table files are transferred to the PS of ZYNQ through the SD card to adapt to deployment on the ZYNQ platform. Next, the camera on the gimbal camera module collects video stream data and inputs it into the PL for preprocessing before loading it into the DDR4 memory. The PS calls the convolution accelerator on the PL to perform target detection, dynamically adjusts the two-dimensional gimbal motion based on the detection results, and calls the bounding box drawing module and the drive display control module on the PL to superimpose the target detection results into the video frame, drive the display device, and achieve real-time tracking of the target.

[0063] The system workflow is as follows Figure 2 As shown, on the PC side, weight quantization and operation simplification are performed, and the weight values ​​and activation values ​​originally of 32-bit floating point type are quantized to 8-bit integer type, and 8-bit integer weight files and lookup table files of Sigmoid activation functions are generated. Then, the weight and lookup table files are transferred to the PS side of ZYNQ via the SD card, and then the PS side loads the files into the DDR4 memory. This method can significantly reduce the computational complexity and storage requirements of the target detection neural network model. After the preparation is completed, the target tracking system based on the ZYNQ platform can be started.

[0064] Reference Figure 1 , 2 3. First, the camera is used to collect video stream data in real time. These data are converted into AXI4-Stream data streams through the video stream data type conversion module and copied into two outputs. One path is transmitted to the DDR4 memory for caching through VDMA; during this process, the VDMA module configured on the PS side sets the address, data width, resolution, and line interval of the frame buffer to ensure the efficiency of data transmission and storage. The other path processes the data through the video preprocessing module on the PL side, including operations such as de-mosaicing, gamma correction, resolution adjustment, and scaling. This step ensures the quality and adaptability of the input data, so that subsequent convolution operations can be performed efficiently. Finally, through VDMA, the preprocessed video data is cached to the DDR4 memory, thereby providing efficient data transmission and caching capabilities, and providing a continuous and stable input data source for the convolution accelerator module.

[0065] Reference Figure 2 , 4 The PS side calls the convolution accelerator on the PL side to convert the preprocessed video data into feature map data after the convolution operation. The convolution accelerator and DMA module on the PL side need to be controlled through the AXI4-Lite control bus.

[0066] The convolution accelerator is the core of the system. First, it adopts the DSP48 resource optimization strategy. By sharing the multiplication factor, two INT8 multiplication operations can be mapped to the same DSP48 unit at the same time, thereby improving the utilization of the multiplier without increasing the consumption of hardware resources and reducing hardware overhead. Secondly, the addition operation part introduces a ternary tree structure. Compared with the traditional binary tree, this design reduces the delay of one clock cycle and saves logic resources, especially in the multi-operand accumulation of the 3x3 convolution window or the addition operation of the multi-channel feature map.

[0067] In terms of data stream processing, the convolution accelerator supports a variety of convolution kernel sizes (such as 3x3, 1x1, etc.) and feature map sizes by dynamically adjusting the depth of the row buffer mechanism. The row buffer mechanism ensures data alignment while improving the parallelism of the data stream and reducing input and output delays. To meet the processing requirements of large-size feature maps, the convolution accelerator further introduces a rectangular blocking strategy to divide the feature map by row and process several rows of data at a time. This blocking method not only maintains the continuity of the data address, but also effectively reduces complex addressing operations and padding requirements. It is particularly suitable for convolution calculations of large input size feature maps (such as 416x416) in the YOLO network.

[0068] The convolution accelerator has quantized inference capabilities. It uses INT8 quantized inference technology to compress floating-point weights and activation values ​​into 8-bit integers, significantly reducing computational complexity. The INT8 multiplication results within the convolution accelerator are accumulated to INT16, and only the INT18 bit width is used during the accumulation process, which not only saves hardware resources but also avoids the risk of overflow of the stimulus value. After the accumulation is completed, the INT18 results are converted to INT8 through scaling factors and zero point adjustment using a multi-step conversion, including saturation truncation, multiplication scaling, rounding operations (rounding right toward zero), etc., to ensure that the quantized inference results are close to the inference results of the original floating-point model. In addition, the convolution accelerator supports the post-quantization method of the Pytorch framework, which further reduces the inference error through static quantization configuration.

[0069] In terms of overall architecture, the convolution accelerator adopts a full pipeline design to ensure that data flows without blocking in each operation stage (row buffer, multiplier, adder tree, accumulation, quantization, etc.). Through the tightly connected pipeline structure, multiple convolution operations can be efficiently processed in each clock cycle, ensuring the high throughput of the system. Parallel computing is the key to achieving high performance of the convolution accelerator, especially in the row and column dimensions and feature map dimensions of the convolution kernel. Taking the 3x3 convolution kernel as an example, the row buffer mechanism generates a 3x3 data window in each clock cycle, and 9 multipliers calculate the product of these data at the same time, and accumulate the results through the adder tree structure. This design ensures that a complete set of convolution operations is completed in each clock cycle, further improving the overall operation efficiency. At the same time, the convolution accelerator supports 64-bit input and output bandwidth, and can transmit 8 sets of INT8 data at the same time, which significantly improves the parallelism of data transmission.

[0070] In terms of weight management, the convolution accelerator transmits 64-bit weight data streams through DMA, and uses a serial-to-parallel strategy to distribute the data to multiple BRAM memories to meet the needs of different convolution kernels. For example, in the case of a 3x3 convolution kernel, the flattened weight data forms 72 bits, which perfectly matches the hardware structure of BRAM36K, ensuring the efficiency of weight storage and reading.

[0071] In order to achieve efficient hardware control, the convolution accelerator sets 7 registers. The PS side reads and writes these 7 registers through the AXI4-Lite bus to control the convolution accelerator.

[0072] The first register is the control register. The first 5 bits are control words, which respectively control the reception of IFM data, Weight data, Bias data, the transmission of OFM data, and the enablement of convolution operations. The last five bits are status words, which respectively indicate whether the current convolution result needs to be pooled, whether it is the first or last convolution, the memory area pointed to by the Ping-pong Buffer pointer, and whether the current data is valid. It can be seen from the first 5 bits of the control word that the convolution accelerator has 5 types of tasks. After each task is enabled, it needs to be manually reset externally. After the task is completed, the convolution accelerator triggers the interrupt on the PS side by pulling up the AP_DONE signal to inform the CPU to complete the currently set task, thereby achieving feedback.

[0073] The second register is used to set the specific parameters of the convolution operation. The first 16 bits indicate the length of the data in the cache to be convolved, and the next 12 bits indicate how many data there are in a row in the current feature map. The last three bits indicate the type of row cache corresponding to the current feature map.

[0074] The third register is used to set the Scaling Factor in the quantization parameter.

[0075] The fourth register is used to set the input zero point and output zero point in the quantization parameter.

[0076] The fifth, sixth, and seventh registers provide some address information required by the convolution accelerator. The PS configures these seven registers through the AXI4-Lite bus, which can flexibly schedule the various operations of the convolution accelerator and ensure the smooth progress of the convolution calculation.

[0077] But at the same time, the PS side also needs to configure the DMA module to control the reception of IFM data, Weight data, Bias data and the sending of OFM data, so as to realize the input image data of the convolution accelerator and the output feature map data after the convolution operation. The output feature map data has the local feature information of the input image.

[0078] After receiving the feature map output by the convolution accelerator, the PS needs to process the feature map. First, the PS applies the Sigmoid activation function The feature map processed by the activation function introduces nonlinear elements, which converts the linear results of the convolution layer output into nonlinear ones, prevents the output values ​​of the convolution layer from being too large or too small, and suppresses unnecessary activations. Therefore, the obtained feature map can better separate the information of objects of different categories.

[0079] Next, the feature map data processed by the activation function is processed by the maximum pooling layer, which is completed by selecting the maximum value in the local window of the feature map. The resolution of the feature map can be reduced, thereby reducing the amount of calculation and memory usage. At the same time, the maximum value of each local window in the feature map is retained. Therefore, while downsampling, the pooling operation retains the significant characteristics of the local information. Reducing the number of parameters of the feature map by the pooling operation can reduce the risk of overfitting of the model, and will not affect the number of channels of the feature map, but only reduce the spatial dimension (width and height). These feature maps will contain bounding box prediction information, including the center coordinates, width, height, confidence, and category prediction of the tracked target. These prediction values ​​are relative to the feature map and need to be decoded to obtain the bounding box coordinates and category information in the actual image.

[0080] Therefore, it is necessary to perform bounding box decoding on the feature map data processed by the pooling layer and extract the predicted value t of each anchor box. x ,t y ,t w ,t h The bounding box offset in the feature map is mapped to the range [0,1] through Sigmoid function processing, and the center coordinates (x, y) in the actual image are obtained based on the coordinates of the grid unit.

[0081] x=(σ(t x )+c x )×s x

[0082] y=(σ(t y )+c y )×s y

[0083] Where σ(t x ),σ(t y ) is the Sigmoid activation of the predicted value, c x 、c y is the coordinate offset of the grid cell, s x 、s y is the size of the grid.

[0084] The scaling factors of width and height are calculated by exponential function and decoded into the width and height dimensions (w, h) in the actual image in combination with the size of the anchor box.

[0085] w=p w ×etw

[0086] h=p h ×e th

[0087] where t w ,t h The predicted scaling factor, p w 、p h are the width and height of the anchor box.

[0088] The data output by bounding box decoding usually contains multiple overlapping bounding boxes, and each bounding box also outputs a confidence and a category probability. The PS needs to process the confidence and category probability of each bounding box through the Sigmoid function, with the output range between 0 and 1, and select the category with the highest category probability and confidence as the target prediction result. The confidence calculation formula is: confidence = σ(t conf ). Among them, t conf It is the confidence prediction value output by the network, which is used to indicate whether the bounding box contains the target object. The calculation formula of category probability is: P class =σ(t class ). Among them, t class is the class prediction value output by the network.

[0089] In order to obtain the final detection result, non-maximum suppression is required on the PS side. NMS is used to remove duplicate prediction boxes and retain the box with the highest confidence. Sort the bounding boxes by confidence, traverse each box in turn, and calculate its IoU with other boxes. If the IoU is greater than the threshold of 0.5, remove the boxes with higher overlap and only retain the box with the highest confidence.

[0090] After obtaining the target’s category label and bounding box coordinates (x, y, w, h), the system enters the error calculation phase, which is the core of the entire closed-loop control. In order to keep the gimbal aligned with the target, the PS needs to calculate the offset of the target relative to the center of the camera’s field of view. Specifically, the center point x center and center is fixed and determined by the camera resolution, while the actual position of the target is represented by the center x and y of the bounding box output by the neural network model. By calculating the difference between the center of the target and the center of the image, the system can get the current error: horizontal error e x =xx center and vertical error e y =yy center These error values ​​indicate how far the target is out of view.

[0091] Then, the PS uses the PID algorithm to process these errors and generate control signals. The PID algorithm is a classic closed-loop control algorithm that can quickly and accurately adjust the gimbal angle by combining proportional (P), integral (I) and differential (D) control to reduce the deviation between the target and the center of the field of view. The PID algorithm generates the control amount through the following formula:

[0092]

[0093] Among them, K p , K i , K d are proportional, integral and differential coefficients. The proportional control part directly generates a control signal according to the size of the error. The larger the error, the larger the adjustment range. The integral control adjusts according to the accumulation of the error to eliminate the steady-state error in the system. The differential control reduces the oscillation of the system by detecting the speed of the error change to ensure the stability of the control. Through the coordinated adjustment of these three parameters, the gimbal can track the target smoothly and avoid jitter or loss when the target moves suddenly. The PS end combines these control quantities through the PID algorithm to output the horizontal control quantity u x and vertical control amount u y , respectively used to control the left and right and up and down rotation of the gimbal.

[0094] The PTZ control stage is the stage where the PS side directly drives the servo through the PID algorithm. Figure 5 As shown, the PS end calculates the control signal u obtained by the PID controller x and u y , transmitted to the two servos of the gimbal in the form of PWM signals, controlling the horizontal and vertical rotations respectively. The horizontal rotation servo adjusts the left and right viewing angles of the gimbal through the control signal, gradually reducing the horizontal error of the target; the vertical rotation servo controls the up and down viewing angles of the gimbal to eliminate the vertical error. In this way, the gimbal can quickly adjust the direction of the camera according to the PWM signal output by the PS end to ensure that the camera is always aimed at the target.

[0095] The key to the system is its feedback adjustment mechanism, which makes the whole process form a closed loop. Each time the gimbal adjusts the camera direction, the camera will capture new images, which are again input into the neural network model for target detection. Due to the adjustment of the gimbal, the position of the target may change. The system calculates the new error again, and after processing by the PID algorithm, further adjusts the rotation angle of the gimbal. This process is repeated continuously, and the system gradually reduces the error through multiple adjustments until the center of the target's bounding box is aligned with the center of the image, that is, e x ≈0 and e y≈ 0. Through such a feedback adjustment mechanism, the system can continuously track the target. Even if the target is moving, the gimbal can continuously adjust the direction of the camera to maintain real-time tracking of the target.

[0096] Finally, if Figure 5 As shown in the figure, the VDMA module reads the video stream data from the DDR4 memory and superimposes the target detection results on the video screen in real time through the bounding box drawing module. During this process, the PS side transmits the target detection results, including the category label and bounding box coordinates (x, y, w, h), to the bounding box drawing module through the AXI4-Lite bus. At the same time, the PS side also configures the address, data width, resolution, and line interval of the frame buffer in the VDMA module to ensure the efficiency of data processing. Then, after being processed by the display driver module, the video stream data is converted into a TMDS signal and output to the display device. At this stage, the PS side also configures the video standard, resolution, frame rate, line interval, and synchronization signal parameters in the display driver module through the AXI4-Lite bus to ensure the quality and stability of the output video. Through these coordinations, the system can achieve real-time target detection and tracking, ensuring that users can intuitively view the processing results.

[0097] In practical applications, this closed-loop control system based on the target detection neural network model and PID control can be used in various scenarios that require automatic tracking. For example, in a security monitoring system, the camera can automatically track a moving intruder and always keep tracking the target; in an automated production line, the camera can track specific objects to ensure that key steps in the production process are accurately captured; in an automatic tracking system for drones, the camera can adjust the direction of the aircraft according to the movement of the target and always keep tracking the target. Regardless of the application, the PID controller can automatically adjust the viewing angle of the gimbal according to the movement of the target to ensure that the system tracks the target stably and efficiently.

[0098] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target tracking method based on the ZYNQ platform, characterized in that: Efficiently running a neural network model to implement a real-time target tracking system based on edge computing, the method comprises the following steps: Step 1: Train and optimize the neural network model on the PC, including weight quantization and operation simplification, generate 8-bit integer weight files and sigmoid activation function lookup table files, and download them to the SD card to adapt to ZYNQ deployment; Step 2: Use the camera to capture video stream data, convert the data type of the video stream data, and divide it into two processing paths: one path is transmitted to VDMA and cached in DDR4 memory, and the other path enters the preprocessing module and is transmitted to VDMA again for caching after preprocessing; Step 3: Transfer the weight and lookup table files to the PS side of ZYNQ through the SD card. The PS side configures the convolution accelerator module and DMA through the AXI4-Lite bus to realize the input image data of the convolution accelerator and the output feature map data after the convolution operation; Step 4: After the PS receives the feature map data output by the convolution accelerator, it performs activation function processing, maximum pooling layer processing, and bounding box decoding in sequence, and finally class prediction and non-maximum suppression to obtain the class label and bounding box coordinates (x, y, w, h); Step 5: The PS side calculates the difference between the target center and the image center, processes these errors through the PID algorithm and generates control signals, thereby adjusting the camera on the 2D gimbal in real time to keep the target centered at all times; Step 6: The bounding box drawing module overlays the target detection results onto the video screen in real time, and processes them through the display driver module, converting the video stream data into TMDS signals and outputting them to the display device, providing clear target tracking feedback.

2. The target tracking method based on the ZYNQ platform according to claim 1, characterized in that: Step 2 is as follows: the camera is used to collect video stream data in real time. These data are converted into AXI4-Stream data streams through the video stream data type conversion module and copied into two outputs. One path is transmitted to the DDR4 memory for caching through VDMA. In this process, the VDMA configured on the PS side sets the address, data width, resolution and line interval parameters of the frame buffer. The other path processes the data through the video preprocessing module on the PL side, including de-mosaicing, gamma correction, resolution adjustment and scaling operations. Finally, VDMA caches the preprocessed video data to the DDR4 memory to provide a continuous and stable input data source for the convolution accelerator module.

3. The target tracking method based on the ZYNQ platform according to claim 1, characterized in that: Step 3 is as follows: the PS configures the seven registers and DMA module of the convolution accelerator through the AXI4-Lite bus to achieve comprehensive control over weight data reading, pooling operations, and convolution operations, and transmits the video data cached in the DDR4 memory to the convolution accelerator module to generate feature maps; Among the 7 registers, the first register is a control register, the first 5 bits are control words, which respectively control the reception of IFM data, Weight data, Bias data, the transmission of OFM data, and the enablement of convolution operation; the last five bits are status words, which respectively indicate whether the current convolution result needs to be pooled, whether it is the first or last convolution, the memory area pointed to by the pointer of Ping-pongBuffer, and whether the current data is valid. The convolution accelerator has 5 types of tasks. After each task is completed, the convolution accelerator triggers the interrupt of the PS end by pulling up the AP_DONE signal, informing the CPU to complete the currently set task, and then the CPU resets it, thereby realizing feedback; The second register is used to set the specific parameters of the convolution operation. The first 16 bits indicate the length of the data in the cache to be convolved; the middle 12 bits indicate how many data are in a row in the current feature map; the last three bits indicate the type of row cache corresponding to the current feature map; The third register is used to set the Scaling Factor in the quantization parameter; The fourth register is used to set the input zero point and output zero point in the quantization parameter; The fifth, sixth, and seventh registers provide some address information required by the convolution accelerator.

4. The target tracking method based on the ZYNQ platform according to claim 3 is characterized in that: The convolution accelerator module optimizes convolution operations through a row buffer mechanism and DSP48 resources, and has the ability to dynamically adjust the row buffer depth to adapt to the requirements of different convolution kernel sizes and feature map sizes. The row buffer mechanism also supports a parallel expansion strategy, so that the convolution calculation can be implemented in synchronous parallel operation in the row and column dimensions of the convolution kernel and the feature map dimension; in the parallel calculation of the 3x3 convolution kernel, a 3x3 data window is generated in each clock cycle, and 9 multipliers are used for simultaneous calculation.

5. The target tracking method based on the ZYNQ platform according to claim 4, characterized in that: The convolution accelerator module processes large-size feature maps through a rectangular blocking strategy, and only convolves a few rows at a time. The convolution accelerator module adopts a full-pipeline structure design and supports a Ping-Pong Buffer structure to achieve parallel convolution operations and data transmission, ensuring that the data processing process is unblocked. The full-pipeline structure design includes closely connected stages of row buffer, multiplier, adder tree, accumulation and quantization. The module has multi-channel processing capabilities, can process multiple input feature map channels at the same time, and adopts a ternary tree structure for efficient channel accumulation.

6. The target tracking method based on the ZYNQ platform according to claim 5, characterized in that: The convolution accelerator module adopts INT8 quantization inference technology to convert 32-bit floating-point weight values ​​and activation values ​​into 8-bit integer types. The product of INT8 value and INT8 value is stored in INT16, and the product result is accumulated. The accumulated result is stored in INT18 type. The accumulated INT18 result is scaled and transformed through the following steps and finally converted to INT8 type: Saturate the INT18 result and truncate it to the INT16 range; Multiply the obtained INT16 value by the Scale FactorM0 of type UINT16 to get the INT32 result; Perform the "Round right to nearest integer" operation to right-shift the INT32 value by 15+Shift Factor bits, and determine the value of the 14+Shift Factor bit in the INT32 result. If the bit is 1, it means that rounding up is required, and 1 is added to the final result. If the bit is 0, no rounding is performed, and the result remains unchanged. After rounding and right-shifting, the INT16 result is obtained again. The obtained INT16 result is saturated again and finally converted to INT8 type. In addition, this method adopts a shared multiplication factor design so that two INT8 multiplication operations can be mapped to the same DSP48 unit. The window data generated by the row buffer mechanism can be transmitted to multiple DSP48 units at the same time to realize parallel convolution calculation.

7. The target tracking method based on the ZYNQ platform according to claim 1, characterized in that: Step 4 is as follows: Step 4-1: The PS uses the Sigmoid activation function to perform nonlinear processing on the feature map, converting the linear result output by the convolution layer into nonlinearity; Step 4-2: Perform maximum pooling on the feature map data after the activation function, and decode the bounding box coordinates and category information of the target by selecting the maximum value in the local window of the feature map; Step 4-3: Decode the bounding box of the feature map data processed by the pooling layer and extract the predicted value t of each anchor box x ,t y ,t w ,t h , through the Sigmoid function processing, the bounding box offset in the feature map is mapped to the range of [0,1], and the center coordinates (x, y) in the actual image are obtained according to the coordinates of the grid unit. x=(σ(t x )+c x )×s x y=(σ(t y )+c y )×s y Where σ(t x ),σ(t y ) is the Sigmoid activation of the predicted value, c x 、c y is the coordinate offset of the grid cell, s x 、s y is the size of the grid; The width and height scaling factors are calculated by exponential functions and decoded into the width and height dimensions (w, h) in the actual image in combination with the size of the anchor box. w=p w ×e tw h=p h ×e th where t w ,t h Predicted scaling factor, p w 、p h are the width and height of the anchor box; Step 4-4: Use the NMS algorithm to deduplicate the bounding boxes, remove duplicate bounding boxes with low confidence, and retain only the boxes with the highest confidence. The NMS algorithm determines the overlap by calculating the IoU between bounding boxes and removes bounding boxes with an IoU value greater than 0.

5.

8. The target tracking method based on the ZYNQ platform according to claim 1, characterized in that: Step 5 is as follows: The PS side calculates the difference between the center of the image captured by the camera and the center of the target (e x , e y ), generates two sets of PWM signals and transmits them to the horizontal and vertical servos of the gimbal respectively, adjusting their rotation angles so that the target always remains centered.

9. The target tracking method based on the ZYNQ platform according to claim 1, characterized in that: Step 6 is as follows: the VDMA module reads the video stream data from the DDR4 memory and overlays the target detection results on the video screen in real time through the bounding box drawing module. During this process, the PS end transmits the category label and bounding box coordinates (x, y, w, h) to the bounding box drawing module through the AXI4-Lite bus, and then after being processed by the display driver module, the video stream data is converted into a TMDS signal and output to the display device.

10. A target tracking system based on the ZYNQ platform, characterized in that: The system includes a PC, a PTZ camera module, a ZYNQ platform, a convolution accelerator module and a display module; The PC is used to train and optimize the target detection neural network model and generate an 8-bit integer weight file and a lookup table file of the Sigmoid activation function; The pan-tilt camera module includes a camera and a two-dimensional pan-tilt, the camera is used to collect video stream data in real time, and the two-dimensional pan-tilt is used to control the horizontal and vertical rotation of the camera respectively to ensure that the camera is always aimed at the target; The ZYNQ platform includes a PL end and a PS end, the PL end includes a VDMA, a video stream data type conversion module, a video preprocessing module, a DMA, a convolution accelerator module, a bounding box drawing module and a display driver module, and the PS end includes a hard-core CPU and a DDR4 memory, for implementing claims 1 to 9; The convolution accelerator module is used to perform a convolution operation on the video data to generate feature map data, and the output feature map data has local feature information of the input image; The display module is used to output video stream data to a display device.

Images