Vision and radar fusion camera system based on FPGA

Abstract

The invention discloses a vision and radar fusion camera system based on an FPGA (Field Programmable Gate Array), which realizes intelligent detection, low-bandwidth real-time tracking and user customizable operation of a metal target through multi-mode perception fusion, event trigger control and dynamic resource optimization technologies. The system is composed of a millimeter wave radar, an FPGA chip, a high-resolution dynamic camera and a tracking turntable. Compared with the prior art, the method has the advantages that through a radar-vision cooperative triggering mechanism, dynamic ROI compression and closed-loop tracking optimization, the data transmission bandwidth is remarkably reduced while the metal target detection precision and real-time performance are guaranteed; and in combination with a fully-open programmable design, the limitation of function solidification of a traditional system is broken through, and a high-adaptability solution is provided for multi-target monitoring in a complex scene.

Description

Technical Field

[0001] The present invention application relates to a vision and radar fusion camera system based on FPGA, which relates to the technical field of surveillance cameras. Background Art

[0002] With the rapid development of the Internet of Things and intelligent sensing technology, millimeter-wave radar and visual fusion systems have been widely used in security monitoring, industrial detection, and autonomous driving. Although traditional millimeter-wave radars have the characteristics of strong penetration and excellent anti-environmental interference, they are limited in target recognition accuracy and scene visualization; while vision-based monitoring systems can provide intuitive image information, they lack active perception capabilities for metal targets and are easily affected by factors such as illumination and occlusion. In the prior art, FPGA-based millimeter-wave radar and camera fusion solutions mostly use fixed viewing angles or full-scene continuous shooting modes, resulting in data redundancy, high bandwidth occupancy, and difficulty in achieving dynamic tracking and real-time feedback of metal targets. In addition, the existing system functions are solidified, and it is difficult for users to flexibly configure monitoring logic, algorithm parameters, etc. according to actual needs, which limits its adaptability in complex scenarios. Therefore, there is an urgent need for a new monitoring system that combines intelligent metal target recognition, low-bandwidth real-time transmission, and customizable control to meet the needs of high-precision and high-efficiency industrial and security applications. Summary of the invention

[0003] The present invention application discloses a FPGA-based vision and radar fusion camera system, which realizes high-precision detection, tracking and visualization of metal dynamic targets through multi-modal perception fusion and intelligent closed-loop control technology. It is characterized by comprising: FPGA chip, millimeter-wave radar, high-resolution dynamic camera, tracking turntable; The FPGA includes a radar perception and collaborative control module, an image storage module, an image processing module, a turntable control module, a camera control module, and an instruction parsing module; The radar perception and collaborative control module uses the millimeter wave spectrum characteristics to analyze the azimuth coordinates and three-dimensional profile parameters of the metal target in real time, generates a digital coding signal containing the target space vector, and triggers the camera collaborative start instruction. The module integrates an adaptive threshold filtering algorithm to suppress multipath interference and enhance target features on the radar echo signal.

[0004] The image processing module realizes the unification of spatiotemporal benchmarks of multi-source data based on hardware-level nanosecond synchronization technology. 2The EUnet++ segmentation network constructs a multi-scale feature fusion architecture, uses an enhanced decoder to optimize edge feature representation, and combines a dynamic weight combination loss function to improve target segmentation accuracy. At the same time, a cross-modal feature similarity measurement model is established to achieve matching association between radar point clouds and image contours based on geometric coordinate deviations and radiation characteristic differences. Finally, the target attribute information analyzed by the radar is superimposed on the visual perception result in real time through the overlay dynamic rendering technology.

[0005] The turntable control module builds a target motion prediction model based on the Kalman filter algorithm, solves the gimbal azimuth / pitch angle control quantity according to the spatial coordinates output by the radar module, and realizes sub-pixel level stable tracking of the moving target in the center of the image sensor's field of view through closed-loop PID regulation.

[0006] The instruction parsing module adopts a multi-threaded instruction distribution mechanism to perform semantic analysis and priority scheduling on the control protocol issued by the host computer, supports dynamic reconfiguration operations including exposure parameter configuration and tracking mode switching, and realizes cross-module instruction transmission through the hardware abstraction layer interface.

[0007] The camera control module constructs a response logic based on a state machine. By analyzing the event trigger signal of the radar module, it adjusts the acquisition frame rate and region of interest (ROI) of the image sensor in a linked manner. At the same time, it forms a vision-radar dual closed-loop feedback with the turntable control module to ensure the real-time performance and positioning accuracy of target tracking.

[0008] The millimeter wave radar mainly uses a frequency modulated continuous wave (FMCW) millimeter wave radar system to extract the motion trajectory parameters of dynamic targets in real time based on a Doppler-range joint solution algorithm, including target contour feature point cloud distribution, radial velocity and relative distance information.

[0009] In the tracking turntable, based on the target space vector parameters output by the radar module, the turntable azimuth-pitch angle joint control quantity is solved in real time through the kinematic inverse solution model, and the digital signal processor (DSP) is used to implement the incremental PID servo control algorithm to drive the high-precision stepper motor for dynamic tracking with a minimum angular resolution of 0.01°; the subsystem has a built-in sliding mode variable structure compensator to dynamically correct the turntable inertia delay and mechanical hysteresis, ensuring that the target remains stably locked with a center offset of less than 5 pixels within the CMOS imaging field of view.

[0010] This invention application solves the technical bottlenecks of existing systems in metal target detection accuracy, tracking stability and bandwidth efficiency through radar-vision deep fusion, closed-loop servo control optimization and adaptive resource scheduling technology, and has the comprehensive advantages of high real-time performance, low power consumption and strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1The overall block diagram of a vision and radar fusion camera system based on FPGA provided by the present invention application Figure 2 FPGA program block diagram of a vision and radar fusion camera system based on FPGA provided by the present invention Figure 3 A block diagram of an image processing module in a vision and radar fusion camera system based on FPGA provided by the present invention Figure 4 The present invention provides an FPGA-based vision and radar fusion camera system R 2 E-Unet++ algorithm framework DETAILED DESCRIPTION The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used in the specification of the present invention pertain to the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0013] In one embodiment, a vision and radar fusion camera system based on FPGA is provided. The camera is composed of an FPGA chip, a millimeter-wave radar, a high-resolution dynamic camera, and a tracking turntable. Among them, the main functional modules of the FPGA chip include: radar perception and collaborative control module, image storage module, image processing module, turntable control module, camera control module, and instruction parsing module.

[0014] The main task of the radar perception and cooperative control module is to transmit a linear frequency modulation signal, receive the target reflection echo, and then use the fast Fourier transform (FFT) kernel embedded in the FPGA to perform range-Doppler two-dimensional spectrum analysis, combined with the adaptive threshold filtering algorithm (Formula 1) to suppress multipath interference: Threshold = α ⋅ s noise + β ⋅ m clutter in, α , β is the dynamic weight coefficient, snoise is the noise standard deviation, m clutter is the noise mean.

[0015] The target 3D point cloud is generated through multi-channel beamforming technology, the contour feature point cloud distribution and radial velocity are extracted, and the trigger signal is generated based on the target space vector (Formula 2): V target =[ x , y , z , v x , v y , v z ] T The image storage module uses DDR3 SDRAM to build a linear table queue structure, sets a 32-level pipeline cache area, and divides each frame of image data into 848-byte data packets in RGB888 format. The parallel writing of CMV4000 sensor and radar data is realized through mig IP; the storage depth is dynamically adjusted according to the memory capacity. When the amount of single-frame data exceeds the threshold, the LRU (least recently used) replacement strategy is triggered. The Power Down control module is used to switch the working mode of the camera sensor, and can flexibly switch the working mode of the camera sensor. Once the camera sensor completes the task, the module will automatically put it in the energy-saving Power Down mode to optimize the energy utilization in different scenarios and extend the service life of the camera. At the same time, when the camera sensor needs to be reactivated, the Power Down control module will accurately wake up and reinitialize the camera sensor to ensure that it is in the best working state at any time.

[0016] The image processing module is mainly responsible for injecting nanosecond synchronization timestamps into the radar and camera data packets in the FPGA to avoid the mismatch of time and space references due to the difference between the millimeter wave radar sampling rate (30HZ) and the CMV4000 frame rate (60-120fps). 2The EUnet++ network processes the image. The present invention proposes a method for effectively aggregating multi-scale contextual information in the segmentation task to enhance the effective receptive field of the model. By utilizing multi-scale feature maps, this method can capture contextual information of different scales and integrate it into the segmentation process. In the encoder structure, the first layer performs two 3×3 convolutions and Relu operations, and then passes through a 2×2 maximum pooling layer. In the second and third layers, this project uses the Res2Net module module plus a 2×2 maximum pooling layer, and in the fourth and fifth layers, the Bottleneck block module plus a 2×2 maximum pooling layer. The Res2Net module divides the feature map and performs step-by-step convolution operations between each sub-feature group to achieve multi-scale feature fusion. Then, using the RE decoder, the output of the encoder must first pass through the RFB module, which can expand the receptive field and try to avoid the problem of spatial information and contextual information loss caused by downsampling. Then, the output of each layer must pass through the channel attention ECA module, which can avoid dimensionality reduction relative to the SE module and can also interact appropriately across channels. Therefore, the ECA module has the advantages of high learning performance and high efficiency, and the model complexity is small, and the improvement effect is significant. Each layer is connected with 2×2 upsampling. After each upsampling operation, the image size will double. In the RE decoder, each layer will fuse some features extracted from the encoder, and enhance the RE decoder's ability to extract important feature information and capture multi-scale contextual information through the RFB module and the ECA module, enhance the network's ability to represent the correlation between image regions, and improve the ability to segment small targets and edges. Finally, the FM combined loss function is used, which weightedly adds the two loss functions, and easily adjusts different weights according to the segmentation task and data set. 2 The E-Unet++ algorithm framework is shown in the figure.

[0017] At the same time, construct the cost matrix C ij C ij , indicating the i The radar point cloud and j The feature similarity of the image edge patches (Formula 2) matches the radar point cloud with the image contour.

[0018] C ij = w 1⋅∥p i rada r −p j image ∥+ w 2⋅∥RCS i −Gray j ∥ Where RCS is the target contour area A , edge gradient ∇ I Time series correlation model with radar cross section (RCS) RCS( t )= k 1⋅ A ( t )+ k 2⋅∥∇ I ( t )∥+ ϵ ( t ) Finally, the module will also open an overlay in the image frame buffer, and superimpose the target contour, velocity vector arrow and RCS value extracted by the radar onto the original image through Alpha blending technology.

[0019] The turntable control module predicts the target motion trajectory based on Kalman filtering (Formula 3) and solves the azimuth i With pitch angle ϕ Control volume: i k+1 = i k + R vx ⋅Δ t ⋅cos ϕ k ϕ k+1 = ϕ k + R vy ⋅Δ t ⋅sin i k And the incremental PID algorithm (Formula 4) is used to drive the stepper motor, combined with sliding mode variable structure compensation (Formula 5) to suppress mechanical hysteresis: Δ u k = K p ( e k − ek−1 )+ K i e k + Kd ( e k −2 e k−1 + e k−2 )(4) s=e˙+λe,u comp =K smc ⋅sgn(s)(5) The instruction parsing module receives PC serial port instructions, maps the instructions to FPGA register configuration parameters, and supports online updating of radar detection thresholds, PID gain coefficients, and turntable speed angles.

[0020] The camera control module constructs a response logic based on a state machine. By analyzing the event trigger signal of the radar module, it adjusts the acquisition frame rate and the region of interest (ROI) of the image sensor in a linked manner, and forms a visual-radar dual closed-loop feedback with the turntable control module to ensure the real-time performance and positioning accuracy of target tracking. It adopts event-triggered imaging logic, and activates camera shooting only when the radar detects a valid target. Combined with the linear table queue pipeline storage technology and the dynamic ROI area compression strategy, it reduces the data transmission bandwidth by more than 80% compared with the traditional continuous shooting mode, while ensuring the integrity and real-time performance of the target data.

[0021] The millimeter wave radar adopts the 77GHz frequency modulated continuous wave (FMCW) radar system, which transmits linear frequency modulation signals and receives the target's reflected echo, and uses the frequency difference (frequency deviation) to analyze the target's distance and speed information. Combined with the 4×4 MIMO antenna array, it realizes high-resolution detection of azimuth and elevation angles to form three-dimensional point cloud data.

[0022] In order to achieve efficient data reading and writing, optimize memory resource utilization, simplify data management, and reduce the impact of file reading and writing on the system during the configuration file reading and updating process, this design chooses to use the linear storage structure of SDRAM for data writing and reading operations. This design decision enables the system to handle the configuration file reading and writing process more efficiently, thereby improving the system's performance and stability.

[0023] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0024] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the invention patent application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present invention application. Therefore, the protection scope of the present invention patent application shall be subject to the attached claims.

Claims

1. A vision and radar fusion camera system based on FPGA, characterized in that: include: FPGA chip, millimeter wave radar, high-resolution dynamic camera, tracking turntable; the FPGA includes a radar perception and collaborative control module, an image storage module, an image processing module, a turntable control module, a camera control module, and an instruction parsing module; The radar perception and collaborative control module analyzes the azimuth coordinates and three-dimensional profile parameters of the metal target in real time through the millimeter wave spectrum characteristics, generates a digital coding signal containing the target space vector, and triggers the camera collaborative start instruction.

2. The FPGA-based vision and radar fusion camera system according to claim 1, characterized in that: The radar perception and collaborative control module integrates an adaptive threshold filtering algorithm to perform multipath interference suppression and target feature enhancement processing on radar echo signals; The image storage module adopts a pipeline storage unit with a linear table queue structure, and realizes parallel caching of CMOS image sensor and radar data stream through a sequential storage controller; Its storage depth dynamically adapts to the physical capacity of the memory and the data volume threshold of a single frame of image, supporting conflict-free writing and on-demand reading operations of burst data streams.

3. The FPGA-based vision and radar fusion camera system according to claim 2, characterized in that: The image processing module achieves the unification of spatiotemporal benchmarks of multi-source data based on hardware-level nanosecond synchronization technology, builds a multi-scale feature fusion architecture by deploying a lightweight R2EUnet++ segmentation network, uses an enhanced decoder to optimize edge feature representation, and combines a dynamic weight combination loss function to improve target segmentation accuracy. At the same time, a cross-modal feature similarity measurement model is established, and the matching association between radar point cloud and image contour is achieved based on the difference in geometric coordinate deviation and radiation characteristics. Finally, the target attribute information analyzed by the radar is superimposed on the visual perception result in real time through the overlay dynamic rendering technology.

4. The FPGA-based vision and radar fusion camera system according to claim 2, characterized in that: The turntable control module builds a target motion prediction model based on the Kalman filter algorithm, solves the gimbal azimuth / pitch angle control quantity according to the spatial coordinates output by the radar module, and realizes sub-pixel level stable tracking of the moving target in the center of the image sensor's field of view through closed-loop PID regulation.

5. The FPGA-based vision and radar fusion camera system according to claim 2, characterized in that: The instruction parsing module adopts a multi-threaded instruction distribution mechanism to perform semantic analysis and priority scheduling on the control protocol issued by the host computer, supports dynamic reconfiguration operations including exposure parameter configuration and tracking mode switching, and realizes cross-module instruction transmission through the hardware abstraction layer interface.

6. The FPGA-based vision and radar fusion camera system according to claim 2, characterized in that: The camera control module constructs a response logic based on a state machine. By analyzing the event trigger signal of the radar module, it adjusts the acquisition frame rate and region of interest (ROI) of the image sensor in a linked manner. At the same time, it forms a vision-radar dual closed-loop feedback with the turntable control module to ensure the real-time performance and positioning accuracy of target tracking.

7. The FPGA-based vision and radar fusion camera system according to any one of claims 1 to 6, characterized in that: The millimeter wave radar uses a frequency modulated continuous wave millimeter wave radar system and extracts the motion trajectory parameters of the dynamic target in real time based on the Doppler-range joint solution algorithm, including the target contour feature point cloud distribution, radial velocity and relative distance information.

8. The FPGA-based vision and radar fusion camera system according to claim 7, characterized in that: In the tracking turntable, based on the target space vector parameters output by the radar module, the turntable azimuth-pitch angle joint control quantity is solved in real time through the kinematic inverse solution model, and the digital signal processor (DSP) is used to implement the incremental PID servo control algorithm to drive the high-precision stepper motor for dynamic tracking with a minimum angular resolution of 0.01°; the subsystem has a built-in sliding mode variable structure compensator to dynamically correct the turntable inertia delay and mechanical hysteresis, ensuring that the target remains stably locked with a center offset of less than 5 pixels within the CMOS imaging field of view.

Images