FPGA-based time and space synchronization method for laser radar and camera

Abstract

The invention provides an FPGA-based time and space synchronization method for a laser radar and a camera. The method comprises the following steps: an FPGA controller establishes a global high-precision time reference by using a message, a PPS signal and an internal counter; synchronously outputting the message and the PPS signal to a laser radar for time service; a global high-precision time reference is used for carrying out time service on the camera; placing a calibration plate, and collecting image data and point cloud data; coordinates of four corner points of the calibration plate in a camera coordinate system are calculated; obtaining point cloud data of the laser radar at the same time, and obtaining coordinates of four corner points and a central point of the calibration plate; multiple pairs of angular point coordinate data of different angles are obtained, external parameter matrixes of a camera coordinate system and a radar coordinate system are optimized through EPNP and a least square optimization algorithm, and space synchronization is achieved. According to the method, accurate space-time alignment of the laser radar and the camera under various environmental conditions can be realized.

Description

Technical Field

[0001] This invention relates to the field of technology, and more specifically, to a temporal and spatial synchronization method for a LiDAR and camera based on an FPGA. Background Technology

[0002] In the fields of autonomous driving, robot navigation, and intelligent surveying, multi-sensor fusion systems combining LiDAR and cameras have become a key technology for environmental perception. LiDAR provides accurate three-dimensional spatial information, while cameras provide rich texture and color information; their complementarity can significantly improve environmental perception capabilities. However, achieving effective fusion of the two technologies first requires solving two key problems: time synchronization and spatial synchronization.

[0003] Currently, time synchronization between lidar and cameras mostly adopts software timestamp alignment, which is easily affected by system delay and jitter. The synchronization accuracy can usually only reach the millisecond level, which is difficult to meet the requirements of high-precision applications. In terms of spatial synchronization, existing calibration methods mainly rely on professional calibration equipment or complex automatic feature matching algorithms, which have the following problems: (1) For low-resolution lidar, the sparse point cloud makes feature extraction difficult and the calibration accuracy is low; (2) Existing calibration methods are sensitive to lighting conditions and scene textures, and have poor robustness in complex environments; (3) The calibration process is cumbersome and requires multiple parameter adjustments, which has a high operation threshold; (4) Most methods lack an effective time synchronization mechanism, which leads to the distortion of calibration results in dynamic scenes.

[0004] Therefore, there is an urgent need for a method that combines hardware-level high-precision time synchronization with semi-automatic spatial calibration to achieve accurate spatiotemporal alignment between lidar and camera under various environmental conditions, providing a reliable foundation for multi-sensor fusion applications. Summary of the Invention

[0005] To overcome the shortcomings and deficiencies of the existing technology, the present invention aims to provide a temporal and spatial synchronization method for lidar and camera based on FPGA; this method can achieve accurate temporal and spatial alignment of lidar and camera under various environmental conditions, providing a reliable foundation for multi-sensor fusion applications and helping to improve the accuracy of surveying results.

[0006] To achieve the above objectives, the present invention provides a temporal and spatial synchronization method for a lidar and camera based on an FPGA, comprising the following steps:

[0007] S1. Complete the installation of the surveying system; the surveying system includes lidar, camera, GNSS receiver, and FPGA controller;

[0008] S2. The FPGA controller receives messages and PPS signals from the GNSS receiver; it obtains the initial number of seconds by parsing the messages; it obtains a second-level count value by counting the PPS signals; it obtains a nanosecond-level count value by counting the nanosecond-level periodic pulse signals generated internally by the FPGA controller; it integrates the initial number of seconds and the second-level and nanosecond-level count values ​​to obtain a global high-precision time base; the FPGA controller synchronously outputs the messages and PPS signals to the lidar to provide time synchronization for the lidar.

[0009] S3, the FPGA controller uses a global high-precision time base to synchronize the camera with the LiDAR, and triggers the camera to perform exposure to generate image data;

[0010] S4. Place a calibration board in the field of view of the lidar and camera, and the FPGA controller collects image data and point cloud data;

[0011] S5. Perform intrinsic parameter calibration on the camera; identify the calibration board from the image data and calculate the 2D coordinates P of the four corner points of the calibration board in the camera coordinate system. cam Obtain the 2D coordinates of the center point; acquire the point cloud data of the lidar at the same moment;

[0012] S6. Filter out the point cloud region containing the calibration board, and then obtain the 3D coordinates P of the four corner points of the calibration board in the region point cloud through planar fitting and bounding line fitting. corner and the 3D coordinates of the center point P center ;

[0013] S7. Repeat steps S5 and S6 to obtain multiple pairs of corner coordinate data at different angles. Use EPNP and least squares optimization algorithms to optimize the extrinsic parameter matrices of the camera coordinate system and radar coordinate system to achieve spatial synchronization.

[0014] Preferably, step S2 is as follows:

[0015] S21. Create a nanosecond-level time counter and a second-level time counter in the FPGA controller; the nanosecond-level time counter is used to count and statistically analyze the nanosecond-level periodic pulse signals generated by the internal differential crystal oscillator; the second-level time counter is used to count the PPS signal.

[0016] S22. The FPGA controller receives the $GPRMC message sent by the GNSS receiver and parses the year, month, day, hour, minute, and second information in the $GPRMC message into an initial number of seconds. The FPGA controller monitors the level change of the PPS pin of the GNSS receiver: when a rising edge transition is detected on the PPS pin, it determines that a complete second of UTC time has started, clears the nanosecond-level time counter to zero, and increments the second-level time counter by one.

[0017] S23. Integrate the initial number of seconds and the count values ​​of the second-level time counter and the nanosecond-level time counter to obtain a global high-precision time base;

[0018] S24. When the FPGA controller detects the rising edge transition of the PPS signal, it buffers the $GPRMC message data and sends the $GPRMC message data to the lidar. When calculating time in seconds, the lidar uses the same PPS signal as the GNSS receiver. When calculating time between seconds, the lidar uses its internal high-precision counter to complete the lidar's time synchronization.

[0019] Preferably, step S3 is as follows:

[0020] S31. The FPGA controller obtains the global high-precision time base generated in step S2.

[0021] S32. Based on a global high-precision time base, the camera timing module generates trigger pulse signals: the nanosecond-level periodic pulse signals generated inside the FPGA controller are divided to generate trigger pulse signals; combined with phase control logic, the rising edge of each trigger pulse signal is aligned to the global high-precision time base.

[0022] S33. The trigger pulse signal is output through the FPGA controller and connected to the external trigger input port of the camera. When the trigger pulse signal arrives, the camera immediately starts the image exposure and acquisition action. At the same time, within the same clock cycle of the trigger pulse signal, the FPGA controller records the current global high-precision time reference as the timestamp of the image data and sends it to the ARM terminal of the FPGA controller for storage.

[0023] S34. Configure the start time of the lidar frame to form a periodic time alignment structure with the rising edge of the PPS signal; the camera is triggered to acquire images at every whole second and subsequent millisecond intervals; the acquisition start times of the lidar and the camera are aligned to integer multiples of the whole second and millisecond intervals, thus achieving precise synchronization between the lidar and the camera on the time axis.

[0024] Preferably, step S4 is as follows: a calibration board is placed vertically in an open outdoor area; the FPGA controller acquires point cloud data from the LiDAR and image data from the camera; subsequently, the image data passes through the following processing modules on the FPGA:

[0025] S41, Demosaic module: Converts RAW format, single-channel image data into a complete RGB three-channel image, restoring the color information of each pixel;

[0026] S42, White Balance Correction Module: Performs white balance processing on image data, adjusts the gain of each channel; sets the image data of the k-th frame to M. k Image information is represented by a 3D matrix of height × width × 3, where height represents the height of the image data, width represents the width of the image data, and 3 represents the RGB channels of the image data. During white balance processing, the gain of the green channel remains unchanged, while the gain of the red channel G... r Gain G of the blue channel b The calculation formulas are as follows:

[0027]

[0028] Among them, M k (i,j,c) represents the pixel coordinates (i,j) of the k-th frame image and the pixel value of the c channel; c = 0, 1, 2; c = 0 represents the red channel, c = 1 represents the green channel, and c = 2 represents the blue channel.

[0029] For the (k+1)th frame of the image, applying the gain calculated in the kth frame, the values ​​of the red and blue channels for each pixel are corrected as follows:

[0030] M k+1 (i,j,0)=M k+1 (i,j,0)×G r

[0031] M k+1 (i,j,2)=M k+1 (i,j,0)×G b

[0032] S43, Gamma LUT module: Performs Gamma correction on image data, and performs non-linear mapping of pixel values ​​of image data through a lookup table;

[0033] S44, AXI4-Stream Subset Converter module: Converts image data formats to AXI4-Stream protocol-compatible data streams, including bit width adjustment and pixel format conversion;

[0034] S45, VDMA module: Transmits image data to the ARM's memory via AXI VDMA.

[0035] Preferably, step S5 is as follows:

[0036] S51. Perform intrinsic parameter calibration on the camera. By acquiring multiple calibration board images at different angles and distances, calculate the camera's intrinsic parameter matrix K and distortion coefficients D. The intrinsic parameter matrix K includes the camera's focal length f. x ,f yand principal point coordinates c x ,c y The standard form of the intrinsic parameter matrix K is:

[0037]

[0038] S52. The camera acquires image data from the calibration board and performs grayscale conversion. Gaussian filtering is then applied to eliminate noise, and the Canny edge detection algorithm is used to extract image edges. The calculation formula is as follows:

[0039] I gray (x,y)=0.299R+0.588G+0.114B

[0040]

[0041] θ = arctan2(G y G x )

[0042] Where R, G, B represent the pixel values ​​of the corresponding color channels, and I gray (x,y) represents the grayscale value of pixel coordinates (x,y). blur (x,y) represents the Gaussian filtered value at pixel coordinates (x,y), where G(i,j) is the Gaussian kernel, G... x For the horizontal gradient, G y For vertical gradient, θ represents the gradient magnitude, and θ represents the gradient direction.

[0043] S53. Perform contour detection and quadrilateral filtering on the image edges, and map the tilted quadrilaterals to standard squares for image data correction; perform tag decoding and ID recognition on the corrected image data, and perform sub-pixel level optimization on the corner points of the calibration board mark to obtain the ID information of the calibration board mark and the pixel coordinates of the four corner points.

[0044] S54. Define the world coordinate system, with the center point of the calibration plate mark as the origin O. w The coordinates P of the four corner points in the world coordinate system p for:

[0045]

[0046] Where S is the physical side length of the marker;

[0047] Using the calibrated camera intrinsic parameter matrix K, the pose in the camera coordinate system is solved using the EPnP algorithm:

[0048]

[0049] Where λ is the proportionality coefficient. P represents the pixel coordinates corresponding to the corner point. p Let R be the coordinates of the corner point in the world coordinate system, R be the rotation matrix, and t be the translation vector;

[0050] By obtaining the rotation matrix R and the translation vector t, the 2D coordinates P of the four corner points of the calibration plate in the camera coordinate system can be calculated. cam :

[0051] P cam =RP p +t,p=0,1,2,3.

[0052] Preferably, step S6 is as follows:

[0053] S61. Set the distance range threshold based on the origin of the radar coordinate system, including: minimum effective distance threshold d min and the maximum effective distance threshold d max ; Calculate the point cloud data for each point P i =(x i ,y i ,z i The Euclidean distance d from the origin i ; retain those that satisfy d min ≤d i ≤d max The point cloud data is selected, and the rest of the point cloud data is discarded.

[0054] S62. Visualize the retained point cloud data in three dimensions. Manually select the area point cloud containing the calibration board through the interactive interface to form the initial ROI. Only retain the point cloud data within the ROI and eliminate background interference.

[0055] S63. Apply the random sampling consensus algorithm to the point cloud data within the ROI to fit the 3D plane equation:

[0056] ax + by + cz + d = 0

[0057] Where x, y, and z are the three-dimensional coordinates of a point on the 3D plane, and a, b, c, and d are the coefficients of the plane equation. Candidate planes are calculated by iteratively selecting the minimum point set, and the number of interior points of each candidate plane is evaluated. The plane with the most interior points is selected as the fitted plane. All outliers are projected onto the fitted plane to eliminate noise in the X direction. The projection calculation formula is:

[0058]

[0059] Where P represents the original point cloud data, P proj This represents the projected point cloud data;

[0060] Based on the layer information, the point cloud data is separated into layers, and a 3D straight line equation is fitted to each layer of point cloud data: dx + ey + f = 0; where d, e, and f are the coefficients of the 3D straight line equation, respectively; all points of each layer are projected onto the fitted 3D straight line to eliminate noise in the Y and Z planes.

[0061] S64. Utilizing the layer value characteristics of LiDAR point cloud data, identify and retain boundary points to determine four boundary regions; apply least squares regression to the point cloud data of each boundary region to fit the equations of the edge lines; calculate the intersection points of adjacent edge lines to obtain the 3D coordinates P of the four corner points. corner ; 3D coordinates of the center point P of the calibration plate center The mean coordinates of the point cloud data of the projected calibration board are obtained by calculating:

[0062] P corner =Line m ∩Line n

[0063]

[0064] Among them, Line m Line n These represent adjacent edge lines, and N is the total number of valid point clouds on the calibration board surface.

[0065] S65. Verify whether the quadrilateral formed by the four corner points meets the dimensions of the physical calibration board: If the verification fails, readjust the ROI or skip this frame and use the image data and point cloud data from the next moment; the specific formula for verification is:

[0066]

[0067] Among them, S min and S max These are the lengths of the shortest and longest sides, respectively; A calc To calculate the area of ​​a quadrilateral, A the This refers to the actual area of ​​the calibration board.

[0068] Preferably, step S7 is as follows:

[0069] S71. Repeat steps S5 and S6 to obtain the 3D coordinates P of the four corner points from the point cloud data. corner and the 3D coordinates of the center point P center The 2D coordinates P of the four corner points obtained from the camera image data cam Establish a correspondence between the 2D coordinates of the center point and the coordinates of the center point to construct a 3D-2D point pair set;

[0070] S72. Set optimization parameters, including: reprojection error threshold ε, number of iterations N, and confidence level δ;

[0071] Next, from the set of 3D-2D point pairs, randomly select the smallest 3D-2D point pair, and apply the EPnP algorithm to calculate the initial extrinsic parameter matrix M = [R]. o |t o ], where R o Let t be a rotation matrix. o The M-matrix is ​​the translation vector; the 3D coordinates are projected onto the 2D plane of the image data using the M-matrix.

[0072] Calculate the reprojection error e = ||x i -P(X i )‖ 2 , where X i For 3D coordinates, P(X) i ) represents the 3D coordinate X i The reprojection point, x i For 3D coordinates X i The 2D coordinates corresponding to the 3D-2D point pair set; if e < ε, then the 3D-2D point pair is determined as an interior point, and the above steps are repeated until the number of iterations is reached;

[0073] Using the obtained inlier and initial EPNP estimates as input, the extrinsic parameters are refined by iteratively minimizing the reprojection error. In each iteration, the damping factor λ is adaptively adjusted according to the error change until convergence is achieved. Finally, the optimized extrinsic parameter matrix M is obtained, thus completing the spatial synchronization of the camera and lidar.

[0074] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0075] 1. This invention utilizes the FPGA resources of an FPGA controller to construct a nanosecond-level high-precision time reference by combining an internally generated nanosecond-level periodic pulse signal with a GNSS receiver PPS signal. Compared with the millisecond-level precision of software timestamp alignment in existing technologies, this invention achieves hardware-level time synchronization, avoiding the effects of system delay and jitter. The camera trigger signal and lidar timing are both based on the same time reference, ensuring strict alignment of sensor data on the time axis. This high-precision time synchronization is particularly suitable for multi-sensor fusion in dynamic scenarios, solving the calibration result distortion problem caused by time asynchrony in existing technologies, and significantly improving the perception accuracy and reliability of the system in motion.

[0076] 2. This invention innovatively proposes a semi-automated calibration process, combining automatic feature detection and interactive point cloud processing. Addressing the issues of sparse point clouds and difficulty in feature extraction in low-resolution LiDAR, it uses an interactive interface to filter point clouds in the calibration board region and employs methods such as RANSAC plane fitting and edge line fitting to accurately extract corner points. Compared to existing methods that rely on specialized equipment or complex automatic matching algorithms, this process is simple to operate, highly adaptable, requires no large amount of training data, and can handle sensor combinations with different resolutions. Attached Figure Description

[0077] Figure 1 This is a flowchart of the time-space synchronization method for LiDAR and camera based on FPGA according to the present invention;

[0078] Figure 2 This is a flowchart of the high-precision time reference establishment in the time-space synchronization method of FPGA-based lidar and camera of the present invention;

[0079] Figure 3 This is a diagram showing the placement of the calibration board in the FPGA-based time-space synchronization method for lidar and camera of this invention.

[0080] Figure 4 This is a schematic diagram illustrating the conversion of image data from high-speed serial byte stream data to a three-channel color image in the temporal and spatial synchronization method of LiDAR and camera based on FPGA of the present invention.

[0081] Figure 5 This is a point cloud diagram of the four corner points of the calibration board obtained by the temporal and spatial synchronization method of the FPGA-based lidar and camera of this invention.

[0082] Figure 6 This is a diagram illustrating the spatial synchronization effect of the camera and lidar in the temporal and spatial synchronization method of lidar and camera based on FPGA of this invention. Detailed Implementation

[0083] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0084] Example 1

[0085] This embodiment presents a temporal and spatial synchronization method for a LiDAR and camera based on an FPGA, such as... Figure 1 As shown, it includes the following steps:

[0086] S1. Complete the installation of the surveying system; the surveying system includes a lidar, camera, GNSS receiver, inertial measurement unit, and FPGA controller; the lidar is used to acquire three-dimensional spatial information of the surrounding environment; the camera is used to acquire color image data; the GNSS receiver is used to receive satellite signals and send position and time information to the FPGA controller; the inertial measurement unit is used to acquire motion state data, such as three-axis acceleration and three-axis angular velocity data; the FPGA controller contains FPGA resources and an ARM processor, used for data acquisition, storage, and calculation.

[0087] S2. The FPGA controller receives the $GPRMC message and 1Hz PPS signal from the GNSS receiver via a serial port; it obtains the initial number of seconds by parsing the $GPRMC message; it obtains the second-level count value by counting the PPS signal; it obtains the nanosecond-level count value by counting the nanosecond-level periodic pulse signal generated internally by the FPGA controller; it integrates the initial number of seconds and the second-level and nanosecond-level count values ​​to obtain a global high-precision time base; the FPGA controller synchronously outputs the $GPRMC message and PPS signal from the GNSS receiver to the lidar for time synchronization.

[0088] In this embodiment, in step S2, the $GPRMC message, 1Hz PPS signal, and differential crystal oscillator pulse signal emitted by the GNSS receiver are used to establish a high-precision global time for the system and to provide time synchronization for the lidar. The method flow is as follows: Figure 2 As shown, the specific steps include:

[0089] S21. Create a nanosecond-level time counter and a second-level time counter in the FPGA controller; the nanosecond-level time counter is used to count and statistically analyze the nanosecond-level periodic pulse signal with a period of 10ns generated by the internal 100MHz differential crystal oscillator; the second-level time counter is used to count the 1Hz PPS signal.

[0090] S22. The UART RX pin of the FPGA controller is connected to the TX pin of the GNSS receiver. A serial port data reception and data parsing program is written in Verilog language to receive the $GPRMC message sent by the GNSS receiver and parse the year, month, day, hour, minute, and second information in the $GPRMC message into an initial number of seconds. The input pin of the FPGA controller is connected to the PPS pin of the GNSS receiver. The FPGA controller monitors the level change of the PPS pin of the GNSS receiver: when a rising edge transition is detected on the PPS pin, it determines that a complete second of UTC time has started, clears the nanosecond-level time counter to zero, and increments the second-level time counter by one.

[0091] S23. Integrate the initial number of seconds and the count values ​​of the second-level time counter and the nanosecond-level time counter; the value of the second-level time counter represents the current UTC second-level time, and the value of the nanosecond-level time counter represents the time elapsed between the current second and the next second, with 10ns as the smallest scale; the PPS signal is used when calculating the second-level time, while the differential crystal oscillator is used when calculating the nanosecond-level time. The two work together to obtain a global high-precision time reference.

[0092] S24. The $GPRMC message input pin of the lidar is connected to the UART TX pin on the FPGA controller, and the PPS signal input pin of the lidar is connected to the PPS pin of the GNSS receiver. When the FPGA controller detects the rising edge transition of the PPS signal, it buffers the $GPRMC message data, delays for 200 milliseconds, and then sends the $GPRMC message data to the lidar at a baud rate of 9600 Mbps. When calculating time in seconds, the lidar uses the same PPS signal as the GNSS receiver. When calculating time between seconds, the lidar uses its internal high-precision counter to complete the lidar's time synchronization.

[0093] S3. The FPGA controller uses a global high-precision time reference to synchronize the camera with the camera's time. The global high-precision time reference is used as the camera's timestamp to achieve time synchronization between the camera and the LiDAR. The nanosecond-level periodic pulse signal is divided and output to the camera to trigger the camera to expose and generate image data. The timestamp of the trigger moment is recorded as the timestamp corresponding to the image data.

[0094] The process of step S3 is as follows:

[0095] S31. Start the camera timing module in the FPGA controller: First, obtain the global high-precision time reference generated in step S2. This global high-precision time reference uses UTC time as the base and, combined with a high-resolution counter built on a 100MHz differential crystal oscillator inside the FPGA, achieves a continuous time reference with nanosecond-level (10ns) time accuracy. This global high-precision time reference serves as the unified clock source for the entire system, ensuring that all sensors maintain strict alignment in the time dimension.

[0096] S32. Based on a global high-precision time base, the camera timing module generates a precise trigger pulse signal according to a preset 10Hz trigger frequency: the nanosecond-level periodic pulse signal generated by the 100MHz differential crystal oscillator inside the FPGA controller is divided by a counter, and the 100MHz clock is divided 10,000,000 times to generate a trigger pulse signal with a period of 100ms; combined with phase control logic, the rising edge of each trigger pulse signal is strictly aligned to the global high-precision time base.

[0097] S33. The trigger pulse signal is output at TTL level through the GPIO pin of the FPGA controller and connected to the external trigger input port of the camera. When the trigger pulse signal arrives, the camera immediately starts the image exposure and acquisition action. At the same time, within the same clock cycle of the trigger pulse signal, the FPGA controller records the current global high-precision time base as the timestamp of the image data (accurate to 10ns) and sends it to the ARM terminal of the FPGA controller for storage via the AXI bus protocol.

[0098] In steps S34 and S24, the lidar uses the PPS signal and $GPRMC message for timing and has the same clock source as the system. Next, the lidar frame start time is configured to form a periodic time alignment structure with the rising edge of the PPS signal. For example, the lidar scanning frame rate is set to 10Hz, which means it rotates ten times per second. Therefore, the start times of the lidar's 1st, 11th, 21st, ... frames will coincide completely with the rising edge of the PPS signal every second, forming a periodic time alignment structure.

[0099] The camera is triggered to acquire images at every whole second and subsequent millisecond (100ms) intervals; by aligning the acquisition start times of the lidar and the camera to integer multiples of the whole second and millisecond (100ms) intervals, precise synchronization between the lidar and the camera on the time axis is achieved.

[0100] S4. Place the ArUco calibration board in the field of view of the LiDAR and camera, and the FPGA controller collects image data and point cloud data.

[0101] The process in step S4 is as follows: The ArUco calibration board is placed vertically on a fixed support in an open outdoor area; the FPGA controller starts the sensor acquisition program to acquire point cloud data from the LiDAR and image data from the camera, with the following results: Figure 3 As shown; point cloud data is transmitted to the ARM side of the FPGA controller via a gigabit Ethernet interface in the form of UDP packets, and the point cloud data is obtained by parsing the UDP packets; image data is transmitted to the FPGA side of the FPGA controller via the MIPI protocol; in order to leverage the parallel processing capabilities of the FPGA and achieve low-latency, high-throughput real-time image processing, a MIPI CSI-2 receiving module needs to be built on the FPGA side. First, the high-speed serial data is deserialized and parsed to recover the original RAW format image data; subsequently, the image data on the FPGA side goes through the following processing modules in sequence:

[0102] S41, Demosaic module: Converts RAW format, single-channel image data into a complete RGB three-channel image, restoring the color information of each pixel;

[0103] S42, White Balance Correction Module: Performs white balance processing on image data, adjusting the gain of each channel to eliminate color deviation caused by the color temperature of the light source, making the image colors closer to the real scene; sets the image data of the kth frame to M. k Image information is represented by a 3D matrix of height × width × 3, where height represents the height of the image data, width represents the width of the image data, and 3 represents the RGB channels of the image data. During white balance processing, the gain of the green channel remains unchanged, while the gain of the red channel G... r Gain G of the blue channel b The calculation formulas are as follows:

[0104]

[0105] Among them, M k (i,j,c) represents the pixel coordinates (i,j) of the k-th frame image and the pixel value of the c channel; c = 0, 1, 2; c = 0 represents the red channel, c = 1 represents the green channel, and c = 2 represents the blue channel.

[0106] For the (k+1)th frame of the image, applying the gain calculated in the kth frame, the values ​​of the red and blue channels for each pixel are corrected as follows:

[0107] M k+1 (i,j,0)=M k+1 (i,j,0)×G r

[0108] M k+1 (i,j,2)=M k+1 (i,j,0)×G b

[0109] S43, Gamma LUT module: Performs Gamma correction on image data by using a lookup table (LUT) to perform non-linear mapping on the pixel values ​​of the image data to match the visual characteristics of the human eye and optimize the display effect;

[0110] S44, AXI4-Stream Subset Converter module: Converts image data formats into AXI4-Stream protocol-compatible data streams, including bit width adjustment and pixel format conversion, to meet the interface requirements of subsequent modules;

[0111] S45, VDMA module: Efficiently transfers image data to the ARM's memory via AXI VDMA to achieve image caching and subsequent processing.

[0112] Steps S41 to S45 above complete the conversion of image data from high-speed serial byte stream data to a three-channel color image, as illustrated in the diagram below. Figure 4As shown.

[0113] S5. Image Data Processing: Perform intrinsic parameter calibration on the camera to obtain the camera's intrinsic parameter matrix; use the ArUco algorithm to identify the calibration board in the image data. If the calibration board's marking pattern is identified, calculate the 2D coordinates P of the four corner points of the calibration board in the camera coordinate system. cam Obtain the 2D coordinates of the center point; acquire the point cloud data of the lidar at the same moment.

[0114] The process of step S5 is as follows:

[0115] S51. Use a standard chessboard to calibrate the camera's intrinsic parameters. By acquiring multiple calibration board images (usually no less than 20) at different angles and distances, calculate the camera's intrinsic parameter matrix K and distortion coefficients D using the Zhang Zhengyou calibration method. The intrinsic parameter matrix K includes the camera's focal length f. x ,f y and principal point coordinates c x ,c y The standard form of the intrinsic parameter matrix K is:

[0116]

[0117] After obtaining the accurate intrinsic parameter matrix, the system enters the camera-radar joint calibration stage, and then the ArUco calibration board needs to be identified.

[0118] S52. The camera acquires image data from the calibration board and performs grayscale conversion. Gaussian filtering is then applied to eliminate noise, and the Canny edge detection algorithm is used to extract image edges. The calculation formula is as follows:

[0119] I gray (x,y)=0.299R+0.588G+0.114B

[0120]

[0121] θ = arctan2(G y G x )

[0122] Where R, G, B represent the pixel values ​​of the corresponding color channels, and I gray (x,y) represents the grayscale value of pixel coordinates (x,y). blur (x,y) represents the Gaussian filtered value at pixel coordinates (x,y), where G(i,j) is the Gaussian kernel, G... x For the horizontal gradient, G y For vertical gradient, θ represents the gradient magnitude, and θ represents the gradient direction.

[0123] S53. Perform contour detection and quadrilateral filtering on the image edges, and map the tilted quadrilaterals to standard squares for image data correction; perform tag decoding and ID recognition on the corrected image data, and perform sub-pixel level optimization on the corner points of the calibration board mark to obtain the ID information of the calibration board mark and the pixel coordinates of the four corner points.

[0124] S54. Define the world coordinate system, with the center point of the calibration plate mark as the origin O. w The coordinates P of the four corner points in the world coordinate system p for:

[0125]

[0126] Where S is the physical side length of the marker (unit: meters);

[0127] Using the calibrated camera intrinsic parameter matrix K, the pose in the camera coordinate system is solved using the EPnP algorithm:

[0128]

[0129] Where λ is the proportionality coefficient. P represents the pixel coordinates corresponding to the corner point. p Let R be the coordinates of the corner point in the world coordinate system, R be the rotation matrix, and t be the translation vector;

[0130] By obtaining the rotation matrix R and the translation vector t, the 2D coordinates P of the four corner points of the calibration plate in the camera coordinate system can be calculated. cam :

[0131] P cam =RP p +t,p=0,1,2,3.

[0132] S6. Processing point cloud data: Filter out the point cloud regions containing the calibration board, then perform planar fitting and bounding line fitting to obtain the 3D coordinates P of the four corner points of the calibration board in the region point cloud. corner and the 3D coordinates of the center point P center .

[0133] The process of step S6 is as follows:

[0134] S61. Based on steps S2 and S3, the acquired image data and point cloud data have been synchronized on the timeline; next, the raw point cloud data is filtered for distance thresholds. Based on sensor characteristics and the placement of the calibration board, a reasonable distance range threshold is set with the radar coordinate system origin as the reference, including: the minimum effective distance threshold d. min and the maximum effective distance threshold d max ; Calculate the point cloud data for each point P i =(xi ,y i ,z i The Euclidean distance d from the origin i :

[0135]

[0136] Retain the condition that d is satisfied min ≤d i ≤d max The point cloud data is selected, and the rest of the point cloud data is discarded.

[0137] S62. The retained point cloud data is visualized in 3D using open source libraries such as Open3D. The point cloud of the region containing the ArUco calibration board is manually selected through the interactive interface to form the initial ROI. Only the point cloud data within the ROI is retained to eliminate background interference.

[0138] S63. Apply the Random Sample Consensus (RANSAC) algorithm to the point cloud data within the ROI to fit the 3D plane equation:

[0139] ax + by + cz + d = 0

[0140] Where x, y, and z are the three-dimensional coordinates of a point on the 3D plane, and a, b, c, and d are the coefficients of the plane equation. Candidate planes are calculated by iteratively selecting a minimum set of three points, evaluating the number of interior points on each candidate plane, and selecting the plane with the most interior points as the fitted plane. All outliers are projected onto the fitted plane to eliminate noise in the X-direction (depth direction). The projection calculation formula is:

[0141]

[0142] Where P represents the original point cloud data, P proj This represents the projected point cloud data;

[0143] To address the random noise in the Y and Z planes of the lidar data, further layer-by-layer processing (i.e., the laser emission channel) is performed. Based on the layer information, the point cloud data is separated into layers, and a 3D linear equation is fitted to each layer of point cloud data: dx + ey + f = 0; where d, e, and f are the coefficients of the 3D linear equation, respectively. All points in each layer are projected onto the fitted 3D linear equation to eliminate noise in the Y and Z planes.

[0144] S64. Utilizing the layer value characteristics of LiDAR point cloud data, identify and retain boundary points to determine four boundary regions; apply least squares regression to the point cloud data of each boundary region to fit the equations of the edge lines; calculate the intersection points of adjacent edge lines to obtain the 3D coordinates P of the four corner points. corner ; 3D coordinates of the center point P of the ArUco calibration plate centerThe mean coordinates of the point cloud data of the projected calibration board are obtained by calculating:

[0145] P corner =Line m ∩Line n

[0146]

[0147] Among them, Line m Line n These represent adjacent edge lines, and N is the total number of valid point clouds on the calibration board surface.

[0148] S65. Verify whether the quadrilateral formed by the four corner points meets the dimensions of the physical calibration board using geometric consistency verification and area verification methods. If the verification fails, readjust the ROI or skip the current frame and use the image data and point cloud data from the next time step. The specific formula for verification is as follows:

[0149]

[0150] Among them, S min and S max These are the lengths of the shortest and longest sides, respectively, ensuring that the extracted corner points conform to the geometric characteristics of the ArUco calibration plate; A calc To calculate the area of ​​a quadrilateral, A theo This refers to the actual area of ​​the calibration board.

[0151] The above steps S61 to S65 determine the coordinates of the four corner points of the point cloud in the calibration board area, and the results are as follows: Figure 5 As shown.

[0152] S7. Repeat steps S5 and S6 to obtain multiple pairs of corner coordinate data at different angles. Use EPNP and least squares optimization algorithms to optimize the extrinsic parameter matrices of the camera coordinate system and radar coordinate system to achieve spatial synchronization. The results are as follows: Figure 6 As shown.

[0153] The process of step S7 is as follows:

[0154] S71. Repeat steps S5 and S6 to obtain the 3D coordinates P of the four corner points from the point cloud data. corner and the 3D coordinates of the center point P center The 2D coordinates P of the four corner points obtained from the camera image data cam Establish a correspondence between the 2D coordinates of the center point and the coordinates of the center point to construct a 3D-2D point pair set;

[0155] S72. Set optimization parameters, including: reprojection error threshold ε, number of iterations N, and confidence level δ; reprojection error threshold ε = 1.0 pixel, number of iterations N = 1,000,000, confidence level δ = 0.99;

[0156] Next, from the set of 3D-2D point pairs, randomly select the smallest 3D-2D point pairs (4 3D-2D point pairs), and apply the EPnP algorithm to calculate the initial extrinsic parameter matrix M = [R]. o |t o ], where R o Let t be a rotation matrix. o The M-matrix is ​​the translation vector; the 3D coordinates are projected onto the 2D plane of the image data using the M-matrix.

[0157] Calculate the reprojection error e = ||x i -P(X i )‖ 2 , where X i For 3D coordinates, P(X) i ) represents the 3D coordinate X i The reprojection point, x i For 3D coordinates X i The 2D coordinates corresponding to the 3D-2D point pair set; if e < ε, then the 3D-2D point pair is determined as an interior point, and the above steps are repeated until the number of iterations is reached;

[0158] Using the obtained inlier and initial EPNP estimates as input, the extrinsic parameters are refined by iteratively minimizing the reprojection error. In each iteration, the damping factor λ is adaptively adjusted according to the error change until convergence is achieved. Finally, the optimized extrinsic parameter matrix M is obtained, thus completing the spatial synchronization of the camera and lidar.

[0159] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A temporal-space synchronization method for a lidar and camera based on FPGA, characterized in that: Includes the following steps: S1. Complete the installation of the surveying system; the surveying system includes lidar, camera, GNSS receiver, and FPGA controller; S2. The FPGA controller receives messages and PPS signals from the GNSS receiver; it obtains the initial number of seconds by parsing the messages; it obtains a second-level count value by counting the PPS signals; it obtains a nanosecond-level count value by counting the nanosecond-level periodic pulse signals generated internally by the FPGA controller; it integrates the initial number of seconds and the second-level and nanosecond-level count values ​​to obtain a global high-precision time base; the FPGA controller synchronously outputs the messages and PPS signals to the lidar to provide time synchronization for the lidar. S3, the FPGA controller uses a global high-precision time base to synchronize the camera with the LiDAR, and triggers the camera to perform exposure to generate image data; S4. Place a calibration board in the field of view of the lidar and camera, and the FPGA controller collects image data and point cloud data; S5. Perform internal parameter calibration on the camera; The image data is used to identify the calibration board, and the 2D coordinates P of the four corner points of the calibration board in the camera coordinate system are calculated. cam Obtain the 2D coordinates of the center point; acquire the point cloud data of the lidar at the same moment; S6. Filter out the point cloud region containing the calibration board, and then obtain the 3D coordinates P of the four corner points of the calibration board in the region point cloud through planar fitting and bounding line fitting. corner and the 3D coordinates of the center point P center ; S7. Repeat steps S5 and S6 to obtain multiple pairs of corner coordinate data at different angles. Use EPNP and least squares optimization algorithms to optimize the extrinsic parameter matrices of the camera coordinate system and radar coordinate system to achieve spatial synchronization.

2. The temporal and spatial synchronization method for FPGA-based lidar and camera according to claim 1, characterized in that: The process of step S2 is as follows: S21. Create a nanosecond-level time counter and a second-level time counter in the FPGA controller; the nanosecond-level time counter is used to count and statistically analyze the nanosecond-level periodic pulse signals generated by the internal differential crystal oscillator; the second-level time counter is used to count the PPS signal. S22. The FPGA controller receives the $GPRMC message sent by the GNSS receiver and parses the year, month, day, hour, minute, and second information in the $GPRMC message into an initial number of seconds. The FPGA controller monitors the level change of the PPS pin of the GNSS receiver: when a rising edge transition is detected on the PPS pin, it determines that a complete second of UTC time has started, clears the nanosecond-level time counter to zero, and increments the second-level time counter by one. S23. Integrate the initial number of seconds and the count values ​​of the second-level time counter and the nanosecond-level time counter to obtain a global high-precision time base; S24. When the FPGA controller detects the rising edge transition of the PPS signal, it buffers the $GPRMC message data and sends the $GPRMC message data to the lidar. When calculating time in seconds, the lidar uses the same PPS signal as the GNSS receiver. When calculating time between seconds, the lidar uses its internal high-precision counter to complete the lidar's time synchronization.

3. The FPGA-based temporal and spatial synchronization method for LiDAR and camera according to claim 1, characterized in that: The process of step S3 is as follows: S31. The FPGA controller obtains the global high-precision time base generated in step S2. S32. Based on a global high-precision time base, the camera timing module generates a trigger pulse signal: the nanosecond-level periodic pulse signal generated inside the FPGA controller is divided to generate a trigger pulse signal; Combined with phase control logic, the rising edge of each trigger pulse signal is aligned to a global high-precision time base; S33. The trigger pulse signal is output through the FPGA controller and connected to the external trigger input port of the camera; When the trigger pulse signal arrives, the camera immediately starts the image exposure and acquisition action; at the same time, within the same clock cycle of issuing the trigger pulse signal, the FPGA controller records the current global high-precision time base as the timestamp of the image data and sends it to the ARM terminal of the FPGA controller for storage. S34. Configure the start time of the lidar frame to form a periodic time alignment structure with the rising edge of the PPS signal; the camera is triggered to acquire images at every whole second and subsequent millisecond intervals; the acquisition start times of the lidar and the camera are aligned to integer multiples of the whole second and millisecond intervals, thus achieving precise synchronization between the lidar and the camera on the time axis.

4. The FPGA-based temporal and spatial synchronization method for lidar and camera according to claim 1, characterized in that: The process of step S4 is as follows: A calibration board is vertically placed in an open outdoor area; the FPGA controller acquires point cloud data from the LiDAR and image data from the camera; subsequently, the image data passes through the following processing modules on the FPGA: S41, Demosaic module: Converts RAW format, single-channel image data into a complete RGB three-channel image, restoring the color information of each pixel; S42, White Balance Correction Module: Performs white balance processing on image data, adjusts the gain of each channel; sets the image data of the k-th frame to M. k Image information is represented by a 3D matrix of height × width × 3, where height represents the height of the image data, width represents the width of the image data, and 3 represents the RGB channels of the image data. During white balance processing, the gain of the green channel remains unchanged, while the gain of the red channel G... r Gain G of the blue channel n The calculation formulas are as follows: Among them, M k (i,j,c) represents the pixel coordinates (i,j) of the k-th frame image and the pixel value of the c channel; c = 0, 1, 2; c = 0 represents the red channel, c = 1 represents the green channel, and c = 2 represents the blue channel. For the (k+1)th frame of the image, applying the gain calculated in the kth frame, the values ​​of the red and blue channels for each pixel are corrected as follows: M k+1 (i,j,0)=M k+1 (i,j,0)×G r M k+1 (i,j,2)=M k+1 (i,j,0)×G b S43, Gamma LUT module: Performs Gamma correction on image data, and performs non-linear mapping of pixel values ​​of image data through a lookup table; S44, AXI4-Stream Subset Converter module: Converts image data formats to AXI4-Stream protocol-compatible data streams, including bit width adjustment and pixel format conversion; S45, VDMA module: Transmits image data to the ARM's memory via AXI VDMA.

5. The FPGA-based temporal and spatial synchronization method for lidar and camera according to claim 1, characterized in that: The process of step S5 is as follows: S51. Perform intrinsic parameter calibration on the camera. By acquiring multiple calibration board images at different angles and distances, calculate the camera's intrinsic parameter matrix K and distortion coefficients D. The intrinsic parameter matrix K includes the camera's focal length f. x ,f y and principal point coordinates c x ,c y The standard form of the intrinsic parameter matrix K is: S52. The camera acquires image data from the calibration board and performs grayscale conversion. Gaussian filtering is then applied to eliminate noise, and the Canny edge detection algorithm is used to extract image edges. The calculation formula is as follows: I gray (x,y)=0.299R+0.588G+0.114B Where R, G, B represent the pixel values ​​of the corresponding color channels, and I gray (x,y) represents the grayscale value of pixel coordinates (x,y). blur (x,y) represents the Gaussian filtered value at pixel coordinates (x,y), where G(i,j) is the Gaussian kernel, G... x For the horizontal gradient, G y For vertical gradient, θ represents the gradient magnitude, and θ represents the gradient direction. S53. Perform contour detection and quadrilateral filtering on the image edges, and map the tilted quadrilaterals to standard squares for image data correction; perform tag decoding and ID recognition on the corrected image data, and perform sub-pixel level optimization on the corner points of the calibration board mark to obtain the ID information of the calibration board mark and the pixel coordinates of the four corner points. S54. Define the world coordinate system, with the center point of the calibration plate mark as the origin O. w The coordinates P of the four corner points in the world coordinate system p for: Where S is the physical side length of the marker; Using the calibrated camera intrinsic parameter matrix K, the pose in the camera coordinate system is solved using the EPnP algorithm: Where λ is the proportionality coefficient. P represents the pixel coordinates corresponding to the corner point. p Let R be the coordinates of the corner point in the world coordinate system, R be the rotation matrix, and t be the translation vector; By obtaining the rotation matrix R and the translation vector t, the 2D coordinates P of the four corner points of the calibration plate in the camera coordinate system can be calculated. cam : P cam =RP p +t。 6. The FPGA-based temporal and spatial synchronization method for lidar and camera according to claim 1, characterized in that: The process of step S6 is as follows: S61. Set the distance range threshold based on the origin of the radar coordinate system, including: minimum effective distance threshold d min and the maximum effective distance threshold d max ; Calculate the point cloud data for each point P i =(x i ,y i ,z i The Euclidean distance d from the origin i ; retain those that satisfy d min ≤d i ≤d max The point cloud data is selected, and the rest of the point cloud data is discarded. S62. Visualize the retained point cloud data in three dimensions. Manually select the area point cloud containing the calibration board through the interactive interface to form the initial ROI. Only retain the point cloud data within the ROI and eliminate background interference. S63. Apply the random sampling consensus algorithm to the point cloud data within the ROI to fit the 3D plane equation: ax + by + cz + d = 0 Where x, y, and z are the three-dimensional coordinates of a point on the 3D plane, and a, b, c, and d are the coefficients of the plane equation. Candidate planes are calculated by iteratively selecting the minimum point set, and the number of interior points of each candidate plane is evaluated. The plane with the most interior points is selected as the fitted plane. All outliers are projected onto the fitted plane to eliminate noise in the X direction. The projection calculation formula is: Where P represents the original point cloud data, P proj This represents the projected point cloud data; Based on the layer information, the point cloud data is separated into layers, and a 3D straight line equation is fitted to each layer of point cloud data: dx + ey + f = 0; where d, e, and f are the coefficients of the 3D straight line equation, respectively; all points of each layer are projected onto the fitted 3D straight line to eliminate noise in the Y and Z planes. S64. Utilizing the layer value characteristics of LiDAR point cloud data, identify and retain boundary points to determine four boundary regions; apply least squares regression to the point cloud data of each boundary region to fit the equations of the edge lines; calculate the intersection points of adjacent edge lines to obtain the 3D coordinates P of the four corner points. corner ; 3D coordinates of the center point P of the calibration plate center The mean coordinates of the point cloud data of the projected calibration board are obtained by calculating: P corner =Line m ∩Line n Among them, Line m Line n These represent adjacent edge lines, and N is the total number of valid point clouds on the calibration board surface. S65. Verify whether the quadrilateral formed by the four corner points meets the dimensions of the physical calibration board: If the verification fails, readjust the ROI or skip this frame and use the image data and point cloud data from the next moment; the specific formula for verification is: Among them, S min and S max These are the lengths of the shortest and longest sides, respectively; A calc To calculate the area of ​​a quadrilateral, A the This refers to the actual area of ​​the calibration board.

7. The FPGA-based temporal and spatial synchronization method for lidar and camera according to claim 1, characterized in that: The process of step S7 is as follows: S71. Repeat steps S5 and S6 to obtain the 3D coordinates P of the four corner points from the point cloud data. corner and the 3D coordinates of the center point P center The 2D coordinates P of the four corner points obtained from the camera image data cam Establish a correspondence between the 2D coordinates of the center point and the coordinates of the center point to construct a 3D-2D point pair set; S72. Set optimization parameters, including: reprojection error threshold ε, number of iterations N, and confidence level δ; Next, from the set of 3D-2D point pairs, randomly select the smallest 3D-2D point pair, and apply the EPnP algorithm to calculate the initial extrinsic parameter matrix M = [R]. o |t o ], where R o Let t be a rotation matrix. o The M-matrix is ​​the translation vector; the 3D coordinates are projected onto the 2D plane of the image data using the M-matrix. Calculate the reprojection error e = ||x i -P(X i )‖ 2 , where X i For 3D coordinates, P(X) i ) represents the 3D coordinate X i The reprojection point, x i For 3D coordinates X i The 2D coordinates corresponding to the 3D-2D point pair set; if e < ε, then the 3D-2D point pair is determined as an interior point, and the above steps are repeated until the number of iterations is reached; Using the obtained inlier and initial EPNP estimates as input, the extrinsic parameters are refined by iteratively minimizing the reprojection error. In each iteration, the damping factor λ is adaptively adjusted according to the error change until convergence is achieved. Finally, the optimized extrinsic parameter matrix M is obtained, thus completing the spatial synchronization of the camera and lidar.

Images