Real-time high-frame-rate high-definition video tracker
Through the hardware architecture of CPU and FPGA, combined with dual-channel video input and minimum system design, the problem of insufficient real-time performance of high frame rate and high-definition video trackers in the existing technology is solved, and low power consumption and efficient video processing are achieved. Improves detection accuracy and speed.
Patent Information
- Application Number
- CN202322845603.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Utility models(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2033-10-23
AI Technical Summary
The prior art is difficult to meet the real-time requirements of high frame rate and high definition video trackers at the same time. The calculation volume is large and the model calculation cost is high, which affects the detection accuracy and speed.
Using the hardware architecture of CPU and FPGA, dual video input is performed through Cameralink-M and Cameralink-B interfaces, FPGA performs protocol analysis and processing, serialization/deserialization of chips to output video, combined with minimal system design, reduce system complexity and cost .
While ensuring high definition and high frame rate, the real-time performance of the video tracker is significantly improved, with the advantages of low power consumption, strong stability, small size and easy to move.
Smart Images
Figure CN222301884U_ABST
Abstract
Description
Technical Field
[0001] The utility model belongs to the field of computer vision, and in particular relates to a real-time high-frame-rate and high-definition video tracker. Background Art
[0002] In computer vision, target detection and target tracking are cross-domains. Target detection is to scan and search for targets in images and videos, locate and identify targets in a scene; while target tracking is to accurately predict the future position and size of the target after the initial state of the target is given, so as to obtain the motion state of the object. In recent years, with the continuous development of deep learning, convolutional neural networks have been widely used in the fields of target detection and target tracking, and have achieved good results.
[0003] The various target tracking algorithms currently used are extremely computationally intensive and the computational cost of the models is very high. In the process of detecting targets, each frame needs to be detected to ensure that the target's motion trajectory is obtained. The position of the target is obtained through the detector, and the accuracy of the detector directly affects the subsequent tracking effect. Therefore, the accuracy of the target detector must be very high, and the detection speed must be very fast, so that accurate results can be obtained in the real-time video stream and used in the subsequent tracking process.
[0004] In the existing technology, it is difficult to simultaneously meet the high frame rate and high definition of the input video and the real-time performance of detection and tracking. Utility Model Content
[0005] In order to solve the above problems existing in the prior art, the utility model provides a real-time high frame rate and high definition video tracker. The technical problem to be solved by the utility model is achieved through the following technical solutions:
[0006] A real-time high frame rate and high definition video tracker, comprising:
[0007] CPU, FPGA, video input module, serialization / deserialization chip, FPGA peripherals and CPU peripherals; among them,
[0008] The video input module is used to input images to the FPGA through a Cameralink-M interface and a Cameralink-B interface;
[0009] The FPGA is used to perform protocol analysis on the received image in combination with the FPGA peripheral and input it to the CPU via the MIPI interface;
[0010] The CPU is used to track and process the received data in combination with the CPU peripherals, and input the processed results into the FPGA;
[0011] The FPGA is also used to process the result received after the CPU processing by using the serialization / deserialization chip and then output the video.
[0012] Preferably, the CPU includes Rockchip's RK3588 chip.
[0013] Preferably, the FPGA includes PG2L100H produced by Tsinghua Unigroup; wherein the FPGA adopts a minimum system design.
[0014] Preferably, the serialization / deserialization chip includes the MS913 chip produced by Hangzhou Ruimeng.
[0015] Preferably, the video input module includes:
[0016] Television equipment and infrared equipment.
[0017] Preferably, the television device transmits data with the FPGA via the Cameralink-M interface and the RS422 / SYNC interface; the FPGA supplies power to the television device via a +5V power supply interface; and the resolution of the television device is 2560×1920@50Hz.
[0018] Preferably, the infrared device transmits data with the FPGA via the Cameralink-B interface and the RS422 / SYNC interface; the FPGA supplies power to the infrared device via a +5V power supply interface; and the resolution of the infrared device is 1280×1024@30Hz.
[0019] Preferably, the FPGA peripherals include:
[0020] 128Mb of QSP FLASH and 1GB of DDR3.
[0021] Preferably, the CPU peripherals include:
[0022] 64GB EMMC, LPDDR4X, storage SD board, YT8521 chip, SIT3232 chip and SIT3088 chip; among them,
[0023] The CPU is connected to the storage SD board via an SDIO interface, connected to the YT8521 chip via an RGMII interface, connected to the SIT3232 chip via a UART interface, and connected to the SIT3088 chip via a MIPI interface.
[0024] Beneficial effects of the utility model:
[0025] In the solution provided by the embodiment of the utility model, the video tracker adopts dual-interface input, uses the CPU to track and process the image analyzed by the FPGA, and inputs the processed result to the FPGA. The FPGA uses the serialization / deserialization chip to process the received result processed by the CPU and then outputs the video. Through the dual-interface video input and the CPU combined with the FPGA to process the input video, the real-time performance is greatly improved while ensuring the high definition and high frame rate of the input video. In addition, the video tracker in this embodiment adopts the minimum system design, which has the advantages of low power consumption, strong stability, small size, low weight and easy mobility. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A structural block diagram of a real-time, high-frame-rate, high-definition video tracker provided by an embodiment of the utility model;
[0027] Figure 2 A hardware composition diagram of a real-time, high-frame-rate, and high-definition video tracker provided in an embodiment of the utility model. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the utility model to clearly and completely describe the technical solutions in the embodiments of the utility model. Obviously, the described embodiments are only part of the embodiments of the utility model, not all of the embodiments. Based on the embodiments in the utility model, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the utility model.
[0029] The utility model embodiment provides a real-time high frame rate and high definition video tracker, such as Figure 1 As shown, including:
[0030] CPU, FPGA, video input module, serialization / deserialization chip, FPGA peripherals and CPU peripherals; among them,
[0031] The video input module is used to input images to the FPGA through the Cameralink-M interface and the Cameralink-B interface;
[0032] FPGA is used to perform protocol analysis on the received image in combination with FPGA peripherals and input it to the CPU through the MIPI interface;
[0033] The CPU is used to track and process the received data in combination with the CPU peripherals, and input the processed results to the FPGA;
[0034] The FPGA is also used to process the received CPU processed results using the serialization / deserialization chip and output the video.
[0035] See also Figure 1 The embodiment of the utility model adopts the hardware architecture of CPU+FPGA, and completes the video input and output functions required by the domestic embedded image acquisition and processing module through the bus expansion peripheral structure of FPGA and CPU; while ensuring the high definition and high frame rate of the input video, the real-time performance is also greatly improved.
[0036] For details, see Figure 2 , Figure 2 This is a specific hardware composition diagram of the video tracker of the present utility model.
[0037] Among them, the video input module includes:
[0038] Television equipment and infrared equipment.
[0039] The TV device transmits data with the FPGA via the Cameralink-M interface and the RS422 / SYNC interface; the FPGA supplies power to the TV device via the +5V power supply interface; the resolution of the TV device is 2560×1920@50Hz.
[0040] The infrared device transmits data with the FPGA through the Cameralink-B interface and the RS422 / SYNC interface; the FPGA supplies power to the infrared device through the +5V power supply interface; the resolution of the infrared device is 1280×1024@30Hz.
[0041] In the utility model, the video input module inputs the image to the FPGA through the Cameralink-M interface and the Cameralink-B interface. It can be seen that the video tracker uses the Cameralink-M interface to transmit the visible light in the image acquired by the TV device in the video input module to the FPGA, and uses the Cameralink-B interface to transmit the infrared light in the image acquired by the infrared device in the video input module to the FPGA, so as to achieve the purpose of dual-channel input.
[0042] The TV device and the infrared device adopt the output mode of external triggering, and the FPGA outputs the trigger signal to communicate with the TV device and the infrared device through the RS422 / SYNC interface.
[0043] In the utility model, the FPGA is used to perform protocol analysis on the received image in combination with the FPGA peripherals and input the image to the CPU through the MIPI interface.
[0044] Specifically, the FPGA includes Unigroup Tongchuang's PG2L100H, wherein the FPGA adopts a minimum system design.
[0045] FPGA peripherals include:
[0046] 128Mb of QSP FLASH and 1GB of DDR3.
[0047] FPGA adopts the minimum system design, with an external 1GB DDR3 memory and 128Mb QSP FLASH, to complete the protocol analysis of the image transmitted by the video input module, and send the RGB format data to the CPU through the MIPI interface.
[0048] The minimum system design is to determine the required logic and external interface according to the functional requirements of the specific application, and implement them on the FPGA. The minimum system design of the FPGA includes determining the system functional modules, external interfaces, clock sources, reset circuits, etc. For details of the clock source and reset circuit, please refer to the publicly available prior art, which will not be described here. The goal of the minimum system design is to achieve the required functions with the minimum resources and cost.
[0049] When selecting an FPGA, the minimum system design is very important for determining the specifications and resource requirements of the FPGA device. Based on the functional modules, I / O interfaces, and other requirements determined in the minimum system design, the required logic resources, memory capacity, number of clocks, etc. can be evaluated. Then, based on these requirements, the appropriate FPGA device is selected to ensure that it has sufficient resources and performance to support the minimum system design.
[0050] Therefore, FPGA minimum system design and FPGA selection are interrelated. Minimum system design provides input requirements for FPGA selection, while FPGA selection provides a suitable hardware platform for minimum system design. This synergy helps to achieve an efficient FPGA system design and development process.
[0051] According to the functional requirements of the video tracker, the pin count of the FPGA minimum system is as follows:
[0052] FPGA has 30 IO interfaces for receiving images sent by the video input module, 48 IO interfaces for inputting the results of image protocol parsing to the CPU, 16 IO interfaces for receiving the processed results output by the CPU, 12 IO interfaces for communication between FPGA and CPU, 51 IO interfaces for connecting 1GB DDR3, 20 IO interfaces for connecting reset circuits, watchdog circuits and other peripherals, and 19 IO interfaces for connecting the serialization / deserialization chip to output video. A total of 196 IO interfaces are required for the above, and specific details can be found in the chip manual.
[0053] According to the requirements of the above IO interfaces, the PG2L100H FPGA model of Unigroup Tongchuang is selected. This FPGA adopts 28nm process and includes modules such as configurable logic module CLM, dedicated 36Kb storage unit DRM, arithmetic processing unit APM, multi-function high-performance IO interface and rich on-chip clock resources. It also integrates hard-core resources such as analog-to-digital conversion module ADC, supports multiple configuration modes, and provides bitstream encryption and authentication, device ID and other functions to protect the user's design security. This FPGA has the throughput capacity of two-way image input and output. As the best choice, PG2L100H-6IFBG484 can be selected. This FPGA package has 285 IO interfaces available for use, meeting the requirements of 196 IO interfaces.
[0054] FPGA uses a minimal system design to reduce the overall cost of the system. It only includes necessary functional modules, which can reduce complexity and development cycle, thereby reducing costs. The reduction of functional modules can reduce signal transmission delays and interference, thereby improving the overall performance of the system.
[0055] In the utility model, the CPU is used to process the image input by the FPGA, and input the processed result to the FPGA.
[0056] Specifically, the CPU includes Rockchip's RK3588 chip.
[0057] CPU peripherals include:
[0058] 64GB EMMC, LPDDR4X, storage SD board, YT8521 chip, SIT3232 chip and SIT3088 chip; among them,
[0059] The CPU is connected to the storage SD board through the SDIO interface, to the YT8521 chip through the RGMII interface, to the SIT3232 chip through the UART interface, and to the SIT3088 chip through the MIPI interface.
[0060] The 64GB EMMC is onboard EMMC storage and supports system booting from EMMC.
[0061] During the verification phase, the CPU expands the SD card storage through the SDIO interface. The storage SD card contains a standard MicroSD card slot, which can be used to insert a TF card. After gluing and reinforcement, it supports the expanded storage of real-time video of the video tracker during movement, which facilitates system debugging and analysis.
[0062] The CPU uses the YT8521 chip to connect to the 100M Ethernet interface, and realizes external 100M network output through Yutaichetong's network YT8521H, outputs H264 or H265 encoding of the video, and supports system upgrades and maintenance.
[0063] The external communication interface of the video tracker is the RS485 interface, which is implemented through the expansion of the interface chip SIT3088. RS485 is a serial communication protocol, commonly used for long-distance data transmission and multi-node communication.
[0064] The debugging serial port of the video tracker is RS232. The video tracker is tested, debugged and maintained through the RS232 communication chip SIT3232 made by China Xinlite. The RS232 communication protocol is suitable for short-distance communication and is usually used between computers and serial port devices.
[0065] Rockchip's RK3588 chip integrates more than 6 billion transistors, uses 4-core Cortex-A76 and 4-core Cortex-A55, 8 EEGPUs and a separate NEON coprocessor. The chip has the characteristics of high computing power and rich interfaces, and supports 8K video encoding. The chip introduces a completely hardware-based image signal processor ISP, and implements many algorithm accelerators, such as: HDR, 3A, LSC, 3DNR, sharpening, dehaze, fisheye correction and gamma correction. The NPU built into the RK3588 chip supports INT4 / INT8 / INT16 / FP16 mixed operations, with a computing power of up to 6TOPS, and has strong compatibility, and can easily convert network models based on a series of frameworks such as TensorFlow / MXNet / PyTorch / Caffe.
[0066] Using Rockchip's RK3588 chip can greatly reduce the time spent on target detection algorithms and target tracking algorithms. The chip has low energy consumption, strong stability, occupies less space and is low in price.
[0067] In the utility model, the FPGA is also used to process the result received after the CPU processing by using the serialization / deserialization chip and then output the video.
[0068] Specifically, the serialization / deserialization chip includes the MS913 chip from Hangzhou Ruimeng.
[0069] The MS913 chip supports input of 25MHz to 100MHz image clock and outputs 10-bit / 12-bit FPD Link. The MS913 chip provides a high-speed FPD-Link interface, a high-speed forward path, and a bidirectional control path. The MS913 chip outputs the CPU-processed results of the PFGA through the FPD-Link interface.
[0070] The power consumption of the video tracker is estimated: the power consumption of the RK3588 chip selected by the embodiment of the utility model is 4.0W, the power consumption of the PG2L100H of Unigroup Tongchuang is 2.5W, the power consumption of the MS913 chip is 0.5W, and the power consumption of other devices is 1W. According to the power efficiency of 85%, the overall power consumption of the video tracker is: 8 / 85%≈9.4W.
[0071] In the solution provided by the embodiment of the utility model, the video tracker adopts dual-interface input, uses the CPU to process the image parsed by the FPGA, and inputs the processed result to the FPGA. The FPGA uses the serialization / deserialization chip to output the result processed by the CPU as a video. Through the dual-interface video input and the CPU combined with the FPGA to process the input video, the real-time performance is greatly improved while ensuring the high definition and high frame rate of the input video. The video tracker in this embodiment adopts the minimum system design, which has the advantages of low power consumption, strong stability, small size, low weight and easy mobility.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A real-time high frame rate and high definition video tracker, characterized in that: include: CPU, FPGA, video input module, serialization / deserialization chip, FPGA peripherals and CPU peripherals; among them, The video input module is used to input images to the FPGA through a Cameralink-M interface and a Cameralink-B interface; The FPGA is used to perform protocol analysis on the received image in combination with the FPGA peripheral and input it to the CPU via the MIPI interface; The CPU is used to track and process the received data in combination with the CPU peripherals, and input the processed results into the FPGA; The FPGA is also used to process the result received after the CPU processing by using the serialization / deserialization chip and then output the video.
2. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The CPU includes Rockchip's RK3588 chip.
3. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The FPGA includes PG2L100H produced by Tsinghua Unigroup; wherein the FPGA adopts a minimum system design.
4. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The serialization / deserialization chip includes the MS913 chip of Hangzhou Ruimeng.
5. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The video input module comprises: Television equipment and infrared equipment.
6. A real-time high frame rate and high definition video tracker according to claim 5, characterized in that: The television device transmits data with the FPGA via the Cameralink-M interface and the RS422 / SYNC interface; the FPGA supplies power to the television device via a +5V power supply interface; and the resolution of the television device is 2560×1920@50Hz.
7. A real-time, high-frame-rate, high-definition video tracker according to claim 5, characterized in that: The infrared device transmits data with the FPGA via the Cameralink-B interface and the RS422 / SYNC interface; the FPGA supplies power to the infrared device via a +5V power supply interface; the resolution of the infrared device is 1280×1024@30Hz.
8. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The FPGA peripherals include: 128Mb of QSP FLASH and 1GB of DDR3.
9. A real-time, high-frame-rate, high-definition video tracker according to claim 1, characterized in that: The CPU peripherals include: 64GB EMMC, LPDDR4X, storage SD board, YT8521 chip, SIT3232 chip and SIT3088 chip; among them, The CPU is connected to the storage SD board via an SDIO interface, connected to the YT8521 chip via an RGMII interface, connected to the SIT3232 chip via a UART interface, and connected to the SIT3088 chip via a MIPI interface.