High-frame-rate KCF target tracking device and method based on FPGA

By designing a high-frame rate KCF target tracking device on FPGA, using pipeline design and resource multiplexing, the problem of slow video data acquisition and target tracking in the existing technology is solved, and high-frame rate video tracking of 60fps is achieved, reducing the Resource consumption.

CN119048552AActive Publication Date: 2024-11-29ZHEJIANG UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410998134.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-29
Estimated Expiration
2044-07-24

Smart Images

  • Figure CN119048552A_ABST
    Figure CN119048552A_ABST
Patent Text Reader

Abstract

The invention discloses a high-frame-rate KCF target tracking device and method based on an FPGA. The method comprises the steps that a video collection module, a high-frame-rate KCF tracking module, a storage module and a video output module are designed on the FPGA; the video acquisition module is used for acquiring video data and storing the video data in the storage module after format conversion; the high-frame-rate KCF tracking module comprises an image interception scaling stage, a KCF template updating stage, a KCF calculation response stage and a control scheduling module, and the control scheduling module performs KCF target tracking algorithm calculation on stored image frames based on three stages of pipeline design and calculation resource scheduling control and generates a target tracking frame in the image frames; and the video output module is used for combining and outputting the target tracking frame and the image frames read from the storage module. Therefore, the calculation efficiency can be improved while the robustness of the algorithm is met, the power consumption of the device can be further reduced, and a foundation is laid for miniaturized application of the device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a high frame rate KCF target tracking device and method based on FPGA. Background Art

[0002] Object tracking is an important task in the field of computer vision and is also the basic task of many computer vision applications. It has important applications in video retrieval, motion-based recognition, human-computer interaction, automated monitoring, vehicle navigation, etc. The emergence of high-definition cameras, the surge in high-end computers, and the growing demand for automatic video analysis have all prompted the research progress of object tracking algorithms.

[0003] In recent years, electronic and computer technology has developed rapidly, and semiconductor technology has also made great progress. FPGA devices can meet the requirements of many practical applications for small and portable platforms. FPGA uses parallel operation methods to greatly help improve computing performance, so for computationally intensive systems, FPGA is an ideal choice for implementing target applications. With the development of high-speed cameras, tracking high-speed moving objects has become a challenge. When tracking targets at high speed, the target tracking system is required to have a high processing speed to ensure that the target is tracked correctly and effectively. The KCF target tracking algorithm is very popular in the field of correlation filtering and is widely used in engineering applications. The KCF algorithm proposes a new tracker through ridge regression, which is not affected by the "kernelization penalty" and reduces complexity.

[0004] The specific steps of the KCF target tracking algorithm are as follows: capture the target image and detection images of multiple scales and scale them to a uniform size; extract the HOG features of the target image and the detection image; perform discrete Fourier transform on the HOG features of the input target image and calculate its Gaussian response; obtain the template parameters based on ridge regression calculation and store them, complete initialization, and calculate the template parameters of the current frame; read the template parameters of the previous frame, and update the new template parameters by weighted calculation; generate the Gaussian response of the detection image based on the feature vector of the detection image and the feature vector of the target image; perform inverse Fourier transform based on the Gaussian response of the detection image and the updated template parameters after dot multiplication; retain the real part of the result after inverse Fourier transform to obtain the real response value, take the detection image with the largest response value as the tracked target, and generate the target tracking frame according to the position of the detection pattern.

[0005] The problems with the existing technology are: the current target tracking cannot achieve high frame rate when collecting video data, and the target tracking algorithm cannot reach high speed and is not hardened. The reason why the video data cannot achieve high frame rate is that the delay of video decoding and format conversion through the CPU is high; the reason why it cannot reach high speed is that the KCF target tracking algorithm requires more computing resources and a longer computing process, and the overall target tracking takes a long time. It is precisely because of the above problems in the existing technology that it cannot meet the tracking of high frame rate video targets. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a high frame rate KCF target tracking method and device based on FPGA, which adopts an algorithm that can be efficiently implemented on the logic hardware FPGA to construct part of the KCF target tracking module, and performs pipeline design and computing resource reuse, thereby improving the computing efficiency while meeting the robustness of the algorithm, and can further reduce the power consumption of the device, laying the foundation for the miniaturized application of the device.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a high frame rate KCF target tracking device based on FPGA, characterized in that a video acquisition module, a high frame rate KCF tracking module, and a video output module are designed on the FPGA;

[0008] The video acquisition module is used to acquire video data and store it in the storage module after format conversion;

[0009] The high frame rate KCF tracking module includes an image capture and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control scheduling module. The control scheduling module performs KCF target tracking algorithm calculation on the stored image frames based on the three stages of pipeline design and computing resource scheduling control, and generates a target tracking frame in the image frame;

[0010] The video output module is used to combine the target tracking frame with the image frame read from the storage module and then output it.

[0011] Preferably, in the video acquisition module, the acquired video data is converted into image frames in RGB format and stored. The acquired video may be a high frame rate video.

[0012] Preferably, the collected video data is converted into image frames in RGB format and stored, including: converting the data output by the camera into parallel data through MIPIIO, then unpacking and extracting valid image data through the CSI interface, and then converting it into RGB format and storing it in the DDR integrated on the FPGA.

[0013] Preferably, the control scheduling module controls three phases of work based on pipeline design, including:

[0014] The input and execution of the image capture and scaling stage, the KCF template update stage and the KCF calculation response stage are controlled, and the output of each stage is cached. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image capture and scaling; when the KCF calculation response is performed in the current image frame, the KCF template of the next image frame can be updated in parallel to realize pipeline operation.

[0015] Preferably, in the image capture and scaling stage, the previous image frame is obtained as a training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is captured from the training image frame through a floating-point number calculation unit, and the current image frame is obtained as a detection image frame. After obtaining detection positions of multiple scales from the detection image frame based on the target position, detection images are captured from the detection image frame at each detection position, and the captured images are scaled to a uniform scale.

[0016] Preferably, in the KCF template update stage, the HOG features of the captured image are extracted, and the HOG features of the captured image are discrete Fourier transformed by the FFT calculation unit, and then the Gaussian response is calculated by the complex calculation unit; according to the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating point calculation unit and the complex calculation unit; when initialized, that is, when the first frame of image is input, the template parameters are calculated from the manually captured image; after each frame of image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the weighted calculation is updated to obtain new template parameters.

[0017] Preferably, in the KCF response calculation stage, a Gaussian response of the detection image is calculated and generated through a complex calculation unit based on the HOG features of the detection image and the HOG features of the training image; the Gaussian response of the detection image and the updated new template parameters are point-multiplied through a complex calculation unit, and then an inverse Fourier transform is performed through an FFT calculation unit; the real part of the result after the inverse Fourier transform is retained to obtain a real response value, and the real response values ​​of each detection image are normalized values; the detection image with the largest real response value is taken as the tracked target, and a target tracking frame is generated therefrom according to the position of the detection pattern.

[0018] Preferably, the control scheduling module works in three stages based on the computing resource scheduling control, including:

[0019] Each stage reuses the same computing units that consume more resources, including FFT computing units, floating-point computing units, and complex computing units.

[0020] Preferably, the calculation delay of each stage in the image capture and scaling stage, the KCF template update stage, and the KCF calculation response stage is controlled within 16ms by controlling the scheduling module.

[0021] To achieve the above-mentioned purpose of the invention, the embodiment further provides a high frame rate KCF target tracking method based on FPGA, which adopts the above-mentioned device, and the method comprises the following steps:

[0022] The video acquisition module is used to collect video data in parallel at a high frame rate and store it in a storage module after format conversion;

[0023] The target tracking frame in the image frame is generated by using the high frame rate KCF tracking module. The specific control scheduling module performs KCF target tracking algorithm calculation on the stored image frame based on pipeline design and computing resource scheduling control image capture and scaling phase, KCF template update phase, and KCF calculation response phase, and generates the target tracking frame in the image frame;

[0024] The video output module is used to combine the target tracking frame with the image frame read from the storage module and then output.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The video acquisition module in the present invention is implemented based on FPGA, and the parallel computing capability of FPGA is used to decode the video and convert the format, thereby reducing the power consumption of the device. When the present invention implements the KCF tracking algorithm based on FPGA, a pipeline design is adopted to divide the KCF tracking algorithm into three stages. The calculation delay of each stage is within 16ms, and each frame image processing is executed in parallel, thereby increasing the video frame rate to 60fps, thereby improving the calculation speed and reducing resource consumption, and having certain engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 It is a structural schematic diagram of a high frame rate KCF target tracking device based on FPGA provided in an embodiment of the present invention;

[0029] Figure 2 is a flow chart of a KCF tracking algorithm provided by an embodiment of the present invention;

[0030] Figure 3 It is a flowchart of a high frame rate KCF target tracking method based on FPGA provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0032] The inventive concept of the present invention is: to solve the technical problems of high power consumption of CPU video acquisition, large resource consumption and slow speed of target tracking algorithm, that is, the problem that target tracking in the prior art cannot achieve high speed and real time, the embodiment of the present invention provides a high frame rate KCF target tracking device and method based on FPGA, which unpacks, converts and stores the acquired video data, implements the KCF tracking algorithm through FPGA, adopts pipeline design and performs resource reuse, and can track targets in real time on high frame rate videos while consuming lower computing resources.

[0033] Figure 1 Schematic diagram of a high frame rate KCF target tracking device based on FPGA provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a high frame rate KCF target tracking device based on FPGA, including: a video acquisition module designed based on FPGA, a high frame rate KCF tracking module, a storage module and a video output module.

[0034] In the embodiment, the video acquisition module is used to collect video data from the camera at a high frame rate in parallel, convert the video into RGB format and store it in DDR. Specifically, the method of collecting and processing video image data is as follows: Figure 2 As shown, the data output by the camera is converted into parallel data through MIPIIO, and then the valid image data is extracted through the CSI-2 unpacking module, and then the raw data is converted into RGB format data through format conversion, and then the white balance is calibrated through the white balance module, and finally the processed image frame data is written into the storage module, where the storage module can be DDR.

[0035] In an embodiment, a high frame rate KCF tracking module is used to perform KCF tracking algorithm calculation on the captured image frames to generate a target tracking frame in the target image frame. Specifically, the high frame rate KCF tracking module includes an image capture and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control scheduling module. The control scheduling module performs KCF target tracking algorithm calculation on the stored image frames based on the three stages of pipeline design and computing resource scheduling control, and generates a target tracking frame in the image frame.

[0036] Among them, in the image capture and scaling stage, the previous image frame is obtained as a training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is captured from the training image frame through a floating-point calculation unit, and the current image frame is obtained as a detection image frame. After obtaining detection positions of multiple scales from the detection image frame based on the target position, detection images are captured from the detection image frame at each detection position, and the captured images are scaled to a uniform scale.

[0037] In the KCF template update stage, the HOG features of the captured image are extracted, and the HOG features of the captured image are discrete Fourier transformed by the FFT calculation unit, and then the Gaussian response is calculated by the complex calculation unit; according to the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating point calculation unit and the complex calculation unit; when initialized, that is, when the first frame of image is input, the template parameters are calculated from the manually captured image; after each frame of image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the weighted calculation is updated to obtain the new template parameters.

[0038] In the KCF response calculation stage, the Gaussian response of the detection image is calculated by the complex calculation unit based on the HOG features of the detection image and the HOG features of the training image; the Gaussian response of the detection image and the updated new template parameters are multiplied by the complex calculation unit, and then the inverse Fourier transform is performed by the FFT calculation unit; the real part of the result after the inverse Fourier transform is retained to obtain the real response value, and the real response values ​​of each detection image are normalized values; the detection image with the largest real response value is taken as the tracked target, and the target tracking frame is generated from it according to the position of the detection pattern.

[0039] The control scheduling module is used to control the three-stage work based on pipeline design, including: controlling the input of the image capture and scaling stage, the KCF template update stage and the KCF calculation response stage, and executing and caching the output of each stage. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image capture and scaling; when the KCF calculation response is performed in the current image frame, the KCF template of the next image frame can be updated in parallel to realize pipeline work.

[0040] Specifically, the pipeline design of the high frame rate KCF tracking module is as follows: Figure 3 As shown, a three-stage pipeline design is implemented for the image capture and scaling stage, the KCF template update stage, and the KCF calculation response stage. When the KCF template is updated in the first frame, the image capture and scaling of the second frame is performed in parallel. When the KCF calculation response is performed in the first frame, the KCF template update of the second frame and the image capture and scaling of the third frame are performed in parallel, and so on. The calculation delay of each stage is 16ms, and a target tracking frame is output every 16ms to complete 60fps video tracking.

[0041] The control and scheduling module is also used to schedule and control three phases of work based on computing resources, including:

[0042] Each stage reuses the same computing units that consume more resources, including FFT computing units, floating-point computing units, and complex computing units.

[0043] In an embodiment, the video output module is used to combine the target tracking frame with the image frame read from the storage module and output the combined image. Specifically, the combined image can be output to a display device through an HDMI interface for visualization.

[0044] The embodiment also provides a high frame rate KCF target tracking method based on FPGA, which adopts the above device, and the method includes the following steps:

[0045] The video acquisition module is used to collect video data in parallel at a high frame rate and store it in a storage module after format conversion;

[0046] The target tracking frame in the image frame is generated by using the high frame rate KCF tracking module. The specific control scheduling module performs KCF target tracking algorithm calculation on the stored image frame based on pipeline design and computing resource scheduling control image capture and scaling phase, KCF template update phase, and KCF calculation response phase, and generates the target tracking frame in the image frame;

[0047] The video output module is used to combine the target tracking frame with the image frame read from the storage module and then output.

[0048] The above-mentioned FPGA-based high-frame rate KCF target tracking device and method of the embodiment can directly read video data through the camera, use the parallel computing capability of FPGA to decode the video and convert the format, and reduce the power consumption of the device; when performing target tracking, a pipeline design is adopted, and the KCF algorithm is divided into three stages. The calculation delay of each stage is within 16ms, and each frame image processing is executed in parallel, thereby increasing the video frame rate to 60fps, thereby improving the calculation speed and reducing resource consumption, and providing a high frame rate and low resource consumption target tracking solution.

[0049] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A high frame rate KCF target tracking device based on FPGA, characterized in that: include: Design video acquisition module, high frame rate KCF tracking module, storage module, and video output module on FPGA; The video acquisition module is used to acquire video data and store it in the storage module after format conversion; The high frame rate KCF tracking module includes an image capture and scaling stage, a KCF template update stage, a KCF calculation response stage, and a control scheduling module. The control scheduling module performs KCF target tracking algorithm calculation on the stored image frames based on the three stages of pipeline design and computing resource scheduling control, and generates a target tracking frame in the image frame; The video output module is used to combine the target tracking frame with the image frame read from the storage module and then output it.

2. The high frame rate KCF target tracking device based on FPGA according to claim 1, characterized in that: In the video acquisition module, the acquired video data is converted into image frames in RGB format and stored.

3. The high frame rate KCF target tracking device based on FPGA according to claim 2 is characterized in that: The collected video data is converted into image frames in RGB format and stored, including: converting the data output by the camera into parallel data through MIPIIO, then unpacking and extracting the valid image data through the CSI interface, and then converting it into RGB format and storing it in the DDR integrated on the FPGA.

4. The high frame rate KCF target tracking device based on FPGA according to claim 1, characterized in that: The control and scheduling module controls three phases of work based on pipeline design, including: The input and execution of the image capture and scaling stage, the KCF template update stage and the KCF calculation response stage are controlled, and the output of each stage is cached. When the KCF template is updated in the current image frame, the next image frame can be input in parallel for image capture and scaling; when the KCF calculation response is performed in the current image frame, the KCF template of the next image frame can be updated in parallel to realize pipeline operation.

5. The high frame rate KCF target tracking device based on FPGA according to claim 1, characterized in that: In the image capture and scaling stage, the previous image frame is obtained as a training image frame, and the target position is obtained from the training image frame. Based on the target position, the target image is captured from the training image frame through a floating-point number calculation unit, and the current image frame is obtained as a detection image frame. After obtaining detection positions of multiple scales from the detection image frame based on the target position, detection images are captured from the detection image frame at each detection position, and the captured images are scaled to a uniform scale.

6. The high frame rate KCF target tracking device based on FPGA according to claim 5, characterized in that: In the KCF template update stage, the HOG features of the captured image are extracted, and the HOG features of the captured image are discrete Fourier transformed by the FFT calculation unit, and then the Gaussian response is calculated by the complex calculation unit; according to the Gaussian response and the Gaussian regression label of the image, the template parameters are calculated and stored by the floating point calculation unit and the complex calculation unit; when the first frame of image is initialized, the template parameters are calculated by the manually captured image; after each frame of image is input, the template parameters of the current image frame are calculated, the template parameters of the previous image frame are read, and the weighted calculation is updated to obtain new template parameters.

7. The high frame rate KCF target tracking device based on FPGA according to claim 6, characterized in that: In the KCF response calculation stage, the Gaussian response of the detection image is calculated and generated by the complex calculation unit based on the HOG features of the detection image and the HOG features of the training image; the Gaussian response of the detection image and the updated new template parameters are multiplied by the complex calculation unit, and then the inverse Fourier transform is performed by the FFT calculation unit; The real part of the result after inverse Fourier transformation is retained to obtain a real response value, and the real response value of each detection image is a normalized value; The detection image with the largest real response value is taken as the tracked target, and the target tracking box is generated from it according to the position of the detection pattern.

8. The high frame rate KCF target tracking device based on FPGA according to claim 1, characterized in that: The control and scheduling module works in three stages based on computing resource scheduling and control, including: Each stage reuses the same computing units that consume more resources, including FFT computing units, floating-point computing units, and complex computing units.

9. The high frame rate KCF target tracking device based on FPGA according to claim 1, characterized in that: By controlling the scheduling module, the calculation delay of each stage in the image capture and scaling stage, the KCF template update stage, and the KCF calculation response stage is controlled within 16ms.

10. A high frame rate KCF target tracking method based on FPGA, characterized in that: The method adopts the device according to any one of claims 1 to 9, and the method comprises the following steps: The video acquisition module is used to collect video data in parallel at a high frame rate and store it in a storage module after format conversion; The target tracking frame in the image frame is generated by using the high frame rate KCF tracking module. The specific control scheduling module performs KCF target tracking algorithm calculation on the stored image frame based on pipeline design and computing resource scheduling control image capture and scaling phase, KCF template update phase, and KCF calculation response phase, and generates the target tracking frame in the image frame; The video output module is used to combine the target tracking frame with the image frame read from the storage module and then output.

Citation Information

Patent Citations

  • KCF target tracking method employing CNN in integrated manner

    CN107767405A

  • FPGA-based Ultra-high Speed Target Tracking Method and Computer Vision System

    CN109461170A

  • Infrared target tracking method of improved KCF feature descriptor

    CN110555866A

  • Target tracking hardware implementation system and method

    US20190139232A1