A high frame rate KCF target tracking device and method based on FPGA

Abstract

This invention discloses a high frame rate KCF target tracking device and method based on FPGA, comprising: a video acquisition module, a high frame rate KCF tracking module, a storage module, and a video output module designed on the FPGA; the video acquisition module is used to acquire video data, convert its format, and store it in the storage module; the high frame rate KCF tracking module includes an image cropping and scaling stage, a KCF template updating stage, a KCF calculation response stage, and a control and scheduling module. The control and scheduling module performs KCF target tracking algorithm calculations on the stored image frames based on three stages: pipeline design and computational resource scheduling control, and generates target tracking boxes in the image frames; the video output module is used to combine the target tracking boxes with the image frames read from the storage module and output them. This can improve computational efficiency while satisfying algorithm robustness, and can also further reduce device power consumption, laying the foundation for the miniaturization of the device.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a high frame rate KCF target tracking device and method based on FPGA. Background Technology

[0002] Object tracking is a crucial task in computer vision and a fundamental task for many computer vision applications, with important applications in video retrieval, motion-based recognition, human-computer interaction, automated monitoring, and vehicle navigation. The emergence of high-definition cameras, the surge in high-end computers, and the growing demand for automated video analysis have all spurred the research and development of object tracking algorithms.

[0003] In recent years, with the rapid development of electronic and computer technologies and significant advancements in semiconductor processes, FPGA devices can meet the requirements of many practical applications for compact and portable implementation platforms. The parallel operation method used in FPGAs greatly improves computational performance, making them an ideal choice for computationally intensive systems. With the development of high-speed cameras, tracking high-speed moving objects has become a challenge. When tracking targets moving at high speeds, the target tracking system must have a high processing speed to ensure accurate and effective tracking. The KCF target tracking algorithm is very popular in the field of correlation filtering and is widely used in engineering applications. The KCF algorithm proposes a new tracker through ridge regression, which is unaffected by the "kerneling penalty," reducing complexity.

[0004] The specific steps of the KCF target tracking algorithm are as follows: Capture the target image and multiple scaled detection images and scale them to a uniform size; extract the HOG features of the target image and detection images; perform a Discrete Fourier Transform on the HOG features of the input target image and calculate its Gaussian response; calculate and store the template parameters based on ridge regression, complete initialization, and calculate the template parameters for the current frame; read the template parameters from the previous frame, perform weighted calculations to update and obtain new template parameters; calculate and generate the Gaussian response of the detection image based on the feature vectors of the detection image and the target image; perform an inverse Fourier Transform after multiplying the Gaussian response of the detection image with the updated template parameters; retain the real part of the result after the inverse Fourier Transform to obtain the real response value, and take the detection image with the largest response value as the tracked target, generating a target tracking box based on the position of the detection pattern.

[0005] The existing technology has the following problems: current target tracking methods cannot achieve high frame rates when acquiring video data, and the target tracking algorithms cannot achieve high speeds and are not hardened. The inability to achieve high frame rates is due to the high latency of video decoding and format conversion via the CPU; the inability to achieve high speeds is due to the large amount of computational resources required by the KCF target tracking algorithm, its long computation process, and the overall time spent on target tracking. It is precisely because of these problems that the existing technology cannot meet the requirements for tracking high frame rate video targets. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a high frame rate KCF target tracking method and apparatus based on FPGA. It uses an algorithm that can be efficiently implemented on logic hardware FPGA to construct part of the KCF target tracking module, performs pipeline design and reuses computing resources, thereby improving computing efficiency while satisfying algorithm robustness, and further reducing device power consumption, laying the foundation for the miniaturization of the device.

[0007] To achieve the above-mentioned objectives, an embodiment provides a high frame rate KCF target tracking device based on FPGA, characterized in that a video acquisition module, a high frame rate KCF tracking module, and a video output module are designed on the FPGA;

[0008] The video acquisition module is used to acquire video data, convert the format, and then store it in the storage module;

[0009] The high frame rate KCF tracking module includes an image cropping and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control and scheduling module. The control and scheduling module performs KCF target tracking algorithm calculation on the stored image frames based on three stages: pipeline design and computing resource scheduling control, and generates target tracking boxes in the image frames.

[0010] The video output module is used to combine the target tracking box with the image frames read from the storage module and then output them.

[0011] Preferably, in the video acquisition module, the acquired video data is converted into RGB format image frames and stored. The acquired video can be a high frame rate video.

[0012] Preferably, the process of converting the acquired video data into RGB format image frames and storing them includes: converting the data output by the camera into parallel data via MIPIIO, then unpacking and extracting the valid image data via the CSI interface, and then converting it into RGB format and storing it in the DDR integrated on the FPGA.

[0013] Preferably, the control and scheduling module operates based on a pipeline design, controlling three stages of operation, including:

[0014] The system controls the input and execution of the image cropping and scaling stage, the KCF template update stage, and the KCF calculation response stage, and caches the output of each stage. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image cropping and scaling; when the KCF calculation response is performed in the current image frame, the KCF template update of the next image frame can be performed in parallel, thus realizing pipelined operation.

[0015] Preferably, in the image cropping and scaling stage, the previous image frame is obtained as a training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is cropped from the training image frame using a floating-point calculation unit. The current image frame is obtained as a detection image frame. After obtaining multiple scale detection positions from the detection image frame based on the target position, the detection image is cropped from the detection image frame at each detection position, and the cropped image is scaled to a uniform scale.

[0016] Preferably, in the KCF template update stage, the HOG features of the cropped image are extracted, and the Gaussian response of the HOG features of the cropped image is calculated by the complex number calculation unit after performing a discrete Fourier transform on the HOG features of the cropped image through the FFT calculation unit. Based on the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating-point calculation unit and the complex number calculation unit. When the first frame image is input during initialization, the template parameters are calculated from the manually cropped image. After each frame image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the new template parameters are obtained by weighted calculation and update.

[0017] Preferably, in the KCF response calculation stage, the Gaussian response of the detection image is generated by calculating the HOG features of the detection image and the HOG features of the training image through a complex number calculation unit; the Gaussian response of the detection image and the updated template parameters are multiplied by a complex number calculation unit, and then an inverse Fourier transform is performed through an FFT calculation unit; the real part of the result after the inverse Fourier transform is retained to obtain the real response value, and the real response value of each detection image is a normalized value; the detection image with the largest real response value is taken as the tracked target, and a target tracking box is generated from it based on the position of the detection pattern.

[0018] Preferably, the control and scheduling module operates based on three stages of computing resource scheduling and control, including:

[0019] Each stage reuses the same computational units that consume a lot of resources, including FFT computational units, floating-point computational units, and complex number computational units.

[0020] Preferably, the computation delay of each stage in the image cropping and scaling stage, the KCF template update stage, and the KCF computation response stage is controlled within 16ms by the control scheduling module.

[0021] To achieve the above-mentioned objectives, an embodiment also provides a high frame rate KCF target tracking method based on FPGA, which employs the aforementioned apparatus. The method includes the following steps:

[0022] The video acquisition module acquires video data at a high frame rate in parallel, converts the format, and stores it in the storage module.

[0023] The high frame rate KCF tracking module generates target tracking boxes in image frames. Specifically, the control and scheduling module, based on pipeline design and computing resource scheduling, controls the image cropping and scaling stage, KCF template update stage, and KCF calculation response stage to perform KCF target tracking algorithm calculation on the stored image frames and generate target tracking boxes in the image frames.

[0024] The target tracking box is combined with the image frames read from the storage module and then output using the video output module.

[0025] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0026] The video acquisition module in this invention is implemented based on FPGA, which utilizes the parallel computing capabilities of FPGA for video decoding and format conversion, thereby reducing device power consumption. When implementing the KCF tracking algorithm based on FPGA, this invention adopts a pipelined design, dividing the KCF tracking algorithm into three stages, with each stage having a calculation delay of less than 16ms. The processing of each frame image is executed in parallel, increasing the video frame rate to 60fps. This improves the calculation speed while reducing resource consumption, and has certain engineering application value. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of the FPGA-based high frame rate KCF target tracking device provided in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of the KCF tracking algorithm provided in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of a high frame rate KCF target tracking method based on FPGA provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0032] The inventive concept of this invention is to solve the technical problems of high power consumption of CPU video acquisition, high resource consumption and slow speed of target tracking algorithms, that is, the problem that target tracking in the prior art cannot achieve high speed and real-time performance. The embodiments of this invention provide a high frame rate KCF target tracking device and method based on FPGA. The acquired video data is unpacked, converted and stored in a different format. The KCF tracking algorithm is implemented through FPGA. The pipeline design and resource reuse are adopted, which can perform real-time target tracking of high frame rate video while consuming low computing resources.

[0033] Figure 1 This is a schematic diagram of the structure of a high frame rate KCF target tracking device based on FPGA provided in an embodiment of the present invention. Figure 1 As shown in the figure, the embodiment provides a high frame rate KCF target tracking device based on FPGA, including: a video acquisition module, a high frame rate KCF tracking module, a storage module, and a video output module designed based on FPGA.

[0034] In this embodiment, the video acquisition module is used to acquire video data from the camera at a high frame rate in parallel, convert the video to RGB format, and store it in DDR. Specifically, the method for acquiring and processing video image data is as follows: Figure 2 As shown, the data output by the camera is converted into parallel data by MIPIIO, then the effective image data is extracted by the CSI-2 unpacking module, then the raw data is converted into RGB format data by format conversion, then the white balance is calibrated by the white balance module, and finally the processed image frame data is written to the storage module, which can be DDR.

[0035] In this embodiment, the high frame rate KCF tracking module is used to perform KCF tracking algorithm calculations on the acquired image frames to generate target tracking boxes in the target image frames. Specifically, the high frame rate KCF tracking module includes an image cropping and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control and scheduling module. The control and scheduling module performs KCF target tracking algorithm calculations on the stored image frames based on three stages: pipeline design and computing resource scheduling control, and generates target tracking boxes in the image frames.

[0036] In the image cropping and scaling stage, the previous image frame is obtained as the training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is cropped from the training image frame through the floating-point calculation unit. The current image frame is obtained as the detection image frame. Based on the target position, multiple scale detection positions are obtained from the detection image frame. The detection image is cropped from each detection position in the detection image frame, and the cropped image is scaled to a uniform scale.

[0037] In the KCF template update phase, the HOG features of the cropped image are extracted. The HOG features of the cropped image are subjected to Discrete Fourier Transform by the FFT calculation unit, and then the Gaussian response is calculated by the complex calculation unit. Based on the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating-point calculation unit and the complex calculation unit. During initialization, i.e., when the first frame image is input, the template parameters are calculated from the manually cropped image. After each frame image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the new template parameters are obtained by weighted calculation.

[0038] In the KCF response calculation phase, the Gaussian response of the detection image is generated by calculating the HOG features of the detection image and the training image through the complex number calculation unit. The Gaussian response of the detection image is multiplied by the updated template parameters through the complex number calculation unit, and then inverse Fourier transform is performed through the FFT calculation unit. The real part of the result after the inverse Fourier transform is retained to obtain the real response value. The real response value of each detection image is a normalized value. The detection image with the largest real response value is taken as the tracked target, and a target tracking box is generated from it based on the position of the detection pattern.

[0039] The control and scheduling module is used to control the three stages of pipelined design, including: controlling the input and execution of the image cropping and scaling stage, the KCF template update stage, and the KCF calculation response stage, and caching the output of each stage. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image cropping and scaling; when the KCF calculation response is performed in the current image frame, the KCF template update of the next image frame can be performed in parallel, thus realizing pipelined operation.

[0040] Specifically, the pipeline design of the high frame rate KCF tracking module is as follows: Figure 3 As shown, a three-stage pipeline design is implemented, consisting of image cropping and scaling, KCF template update, and KCF computation response. When the KCF template is updated in the first frame, the image cropping and scaling of the second frame is performed in parallel. When the KCF computation response is performed in the first frame, the KCF template update of the second frame and the image cropping and scaling of the third frame are performed in parallel, and so on. The computation delay of each stage is 16ms. During output, a target tracking box is output every 16ms to complete 60fps video tracking.

[0041] The control and scheduling module is also used for three phases of work based on computing resource scheduling and control, including:

[0042] Each stage reuses the same computational units that consume a lot of resources, including FFT computational units, floating-point computational units, and complex number computational units.

[0043] In this embodiment, the video output module is used to combine the target tracking box with the image frames read from the storage module and output them. Specifically, it can be output to a display device via an HDMI interface for visualization.

[0044] The embodiment also provides a high frame rate KCF target tracking method based on FPGA, which uses the above-mentioned apparatus, and the method includes the following steps:

[0045] The video acquisition module acquires video data at a high frame rate in parallel, converts the format, and stores it in the storage module.

[0046] The high frame rate KCF tracking module generates target tracking boxes in image frames. Specifically, the control and scheduling module, based on pipeline design and computing resource scheduling, controls the image cropping and scaling stage, KCF template update stage, and KCF calculation response stage to perform KCF target tracking algorithm calculation on the stored image frames and generate target tracking boxes in the image frames.

[0047] The target tracking box is combined with the image frames read from the storage module and then output using the video output module.

[0048] The above-described FPGA-based high frame rate KCF target tracking device and method can directly read video data through a camera, utilize the parallel computing capabilities of the FPGA for video decoding and format conversion, thereby reducing device power consumption. During target tracking, a pipelined design is adopted, dividing the KCF algorithm into three stages, with each stage having a computation delay of less than 16ms. Each frame image is processed in parallel, increasing the video frame rate to 60fps. This improves computation speed while reducing resource consumption, thus providing a high frame rate, low resource consumption target tracking solution.

[0049] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high frame rate KCF target tracking device based on FPGA, characterized in that, include: Design a video acquisition module, a high frame rate KCF tracking module, a storage module, and a video output module on an FPGA; The video acquisition module is used to acquire video data, convert the format, and then store it in the storage module; The high frame rate KCF tracking module includes an image cropping and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control and scheduling module. The control and scheduling module performs KCF target tracking algorithm calculation on the stored image frames based on three stages: pipeline design and computing resource scheduling control, and generates target tracking boxes in the image frames. The control and scheduling module is designed based on a pipeline to control the operation of three stages: image cropping and scaling, KCF template update, and KCF calculation response. It controls the input and execution of each stage and caches the output of each stage. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image cropping and scaling. When the KCF calculation response is performed in the current image frame, the KCF template update of the next image frame can be performed in parallel, thus realizing pipeline operation. The control and scheduling module operates in three stages based on computing resource scheduling and control, including: each stage reuses the same computing units that consume a lot of resources, including FFT computing units, floating-point computing units, and complex number computing units; The computation delay of each stage in the image cropping and scaling stage, KCF template update stage, and KCF computation response stage is controlled within 16ms by the control scheduling module. The video output module is used to combine the target tracking box with the image frames read from the storage module and then output them.

2. The FPGA-based high frame rate KCF target tracking device according to claim 1, characterized in that, The video acquisition module converts the acquired video data into RGB format image frames and stores them.

3. The FPGA-based high frame rate KCF target tracking device according to claim 2, characterized in that, The process involves converting the acquired video data into RGB format image frames and storing them, including: converting the data output from the camera into parallel data via MIPIIO, then unpacking and extracting the valid image data through the CSI interface, and finally converting it into RGB format and storing it in the DDR integrated on the FPGA.

4. The FPGA-based high frame rate KCF target tracking device according to claim 1, characterized in that, In the image cropping and scaling stage, the previous image frame is obtained as the training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is cropped from the training image frame through the floating-point calculation unit. The current image frame is obtained as the detection image frame. Based on the target position, multiple scale detection positions are obtained from the detection image frame. The detection image is cropped from each detection position in the detection image frame, and the cropped image is scaled to a uniform scale.

5. The FPGA-based high frame rate KCF target tracking device according to claim 4, characterized in that, In the KCF template update stage, the HOG features of the cropped image are extracted. The HOG features of the cropped image are subjected to discrete Fourier transform by the FFT calculation unit, and the Gaussian response is calculated by the complex calculation unit. Based on the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating-point calculation unit and the complex calculation unit. When the first frame image is input, the template parameters are calculated from the manually cropped image. After each frame image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the new template parameters are obtained by weighted calculation.

6. The FPGA-based high frame rate KCF target tracking device according to claim 5, characterized in that, In the KCF response calculation stage, the Gaussian response of the detection image is generated by the complex number calculation unit based on the HOG features of the detection image and the HOG features of the training image; the Gaussian response of the detection image and the updated template parameters are multiplied by the complex number calculation unit, and then the inverse Fourier transform is performed by the FFT calculation unit. The real part of the result after the inverse Fourier transform is retained to obtain the real response value. The real response value of each detected image is a normalized value. The detection image with the largest real response value is taken as the tracked target, and a target tracking box is generated from it based on the position of the detection pattern.

7. A high frame rate KCF target tracking method based on FPGA, characterized in that, It employs the apparatus according to any one of claims 1-6, and the method comprises the following steps: The video acquisition module acquires video data at a high frame rate in parallel, converts the format, and stores it in the storage module. The high frame rate KCF tracking module generates target tracking boxes in image frames. Specifically, the control and scheduling module, based on pipeline design and computing resource scheduling, controls the image cropping and scaling stage, KCF template update stage, and KCF calculation response stage to perform KCF target tracking algorithm calculation on the stored image frames and generate target tracking boxes in the image frames. The target tracking box is combined with the image frames read from the storage module and then output using the video output module.

Images