Camera and LiDAR synchronization method and system for intelligent driving control system
By calculating the synchronization delay time of the camera and performing data fusion processing, the problem of misalignment between the exposure time of the lidar and the camera was solved, achieving high-precision data synchronization and fusion, and improving the performance of the intelligent driving control system.
Patent Information
- Application Number
- CN202110520813.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-05-13
AI Technical Summary
In intelligent driving control systems, it is difficult to align the scanning time of the lidar with the exposure time of the camera at the corresponding installation angle, resulting in a mismatch between time and space data and reducing control accuracy.
By setting the reference angle for LiDAR scanning, calculating the camera's frame rate and installation angle, obtaining the synchronization delay time, controlling the camera's exposure time to align with the LiDAR scanning time, and achieving data fusion through perspective transformation matrix and buffer pool, accurate time and space information is obtained.
It achieves precise synchronization of camera and LiDAR data, improving the detection accuracy and data fusion effect of the intelligent driving control system.
Smart Images

Figure CN113219479B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent driving control technology, and in particular to a camera and laser radar synchronization method and system for an intelligent driving control system. Background Art
[0002] Intelligent driving control technology requires precise synchronization of multiple data such as video, radar, and inertial navigation to achieve temporal and spatial correlation operations. At present, the mainstream sensors in the intelligent driving industry are cameras and lidars. Lidars are connected via Ethernet and perform 360° scanning at 10 or 30rps (revolutions per second). Cameras are installed at 0-360° angles such as the front, side, rear, and rear view. Cameras are connected via MIPI and shoot video data at a frame rate of 10 or 30fps (frames per second). The time interval between two frames of data is large. On high-speed moving vehicles, data at different times correspond to different spatial positions. It is difficult to align the lidar scanning time with the exposure time of the camera at the corresponding installation angle. The application of data that does not correspond to time and space will cause various problems, reducing the control accuracy of intelligent driving. Summary of the invention
[0003] In view of this, on the one hand, the present invention proposes a camera and lidar synchronization method for an intelligent driving control system to solve the problem that it is difficult to align the lidar scanning time of the intelligent driving control system with the exposure time of the camera at the corresponding installation angle.
[0004] The technical solution of the present invention is implemented as follows: a camera and laser radar synchronization method of an intelligent driving control system, comprising:
[0005] Step S1, setting the reference angle of the laser radar scanning, obtaining the frame rate and installation angle of the camera, and calculating the synchronization delay time of the camera exposure time according to the frame rate and installation angle;
[0006] Step S2, obtaining a reference time of the laser radar scanning reference angle, taking the reference time as the starting point, and controlling the corresponding camera exposure after the synchronization delay time has elapsed;
[0007] Step S3, obtaining the camera data and the lidar data at the time of exposure, and fusing the camera data and the lidar data at this time to obtain accurate time and space information.
[0008] Optionally, step S1 includes:
[0009] dly=(1 / fps)*(ang / 360), dly is the synchronization delay time, fps is the frame rate, and ang is the installation angle.
[0010] Optionally, in step S3, the camera data and the lidar data are fused, including:
[0011] Obtain the conversion relationship between the world coordinate system and the camera image coordinate system through the perspective transformation matrix;
[0012] Obtain the conversion relationship between the world coordinate system and the lidar coordinate system;
[0013] Determine the data mapping between camera data and lidar data.
[0014] Optionally, the transformation relationship between the world coordinate system and the camera image coordinate system is obtained through a perspective transformation matrix, including:
[0015]
[0016] u and v are the horizontal and vertical coordinates of a point P in the camera image coordinate system, respectively. Xw, Yw, and Zw are the three-dimensional coordinates of point P in the world coordinate system, respectively. n1 to n11 are the elements in the perspective transformation matrix. The perspective transformation matrix is expressed as
[0017]
[0018] Optionally, obtain the conversion relationship between the world coordinate system and the lidar coordinate system, including:
[0019]
[0020] r is the distance from the laser radar to this point in the laser radar coordinate system, θ is the angle swept by the laser radar, and h is the installation height of the laser radar.
[0021] Optionally, determine the data mapping between camera data and lidar data, including:
[0022] The laser radar and camera are jointly calibrated using the calibration column to obtain the perspective transformation matrix.
[0023] Optionally, in step S3, the camera data and the lidar data are fused, including:
[0024] Buffer pools are set up at the input ends of camera data and lidar data to retain data reached at different times and record their timestamps. The function of each buffer pool is to perform a trigger action after entering a frame of data, save its data value and timestamp, and re-trigger and cycle when the next frame arrives.
[0025] The camera and laser radar synchronization method of the intelligent driving control system of the present invention has the following beneficial effects compared with the prior art:
[0026] (1) The synchronization delay time of the camera exposure time is calculated according to the camera frame rate and installation angle, so that its exposure time is exactly the same as the time it takes the LiDAR to scan this angle. At this time, the LiDAR data and the camera data can be fused to obtain accurate time and space information;
[0027] (2) The coordinate systems of the camera and the lidar are jointly calibrated through the world coordinate system. The calibration column is used to realize the joint calibration of the lidar and the camera, and the perspective transformation matrix is obtained. The lidar data can be mapped to the image plane to realize data fusion.
[0028] (3) The buffer pool-based method realizes time calibration, reduces time deviation, and improves detection accuracy.
[0029] On the other hand, the present invention also proposes a camera and lidar synchronization system for an intelligent driving control system to solve the problem that it is difficult to align the lidar scanning time of the intelligent driving control system with the exposure time of the camera at the corresponding installation angle.
[0030] The technical solution of the present invention is implemented as follows: a camera and laser radar synchronization system of an intelligent driving control system, comprising:
[0031] A synchronization delay time calculation unit is used to set a reference angle for laser radar scanning, obtain the frame rate and installation angle of the camera, and calculate the synchronization delay time of the camera exposure time according to the frame rate and the installation angle;
[0032] The camera exposure control unit is used to obtain the reference time of the laser radar scanning reference angle, and control the corresponding camera exposure after the synchronization delay time has passed, taking the reference time as the starting point;
[0033] The data fusion processing unit is used to obtain the camera data and the lidar data during exposure, and to fuse the camera data and the lidar data at this time to obtain accurate time and space information.
[0034] The advantages of the camera and lidar synchronization system and the above-mentioned camera and lidar synchronization method over the prior art are the same and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1It is a flow chart of the camera and laser radar synchronization method of the present invention;
[0037] Figure 2 It is a schematic diagram of hardware connection of the present invention;
[0038] Figure 3 A schematic diagram of the camera installation angle of the present invention;
[0039] Figure 4 A schematic diagram of the camera data and laser radar data acquisition frequency of the present invention;
[0040] Figure 5 A schematic diagram of the time difference between camera data and laser radar data of the present invention;
[0041] Figure 6 It is a structural block diagram of the camera and laser radar synchronization system of the present invention. DETAILED DESCRIPTION
[0042] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] like Figure 1 As shown, the camera and laser radar synchronization method of the intelligent driving control system of this embodiment includes:
[0044] Step S1, setting the reference angle of the laser radar scanning, obtaining the frame rate and installation angle of the camera, and calculating the synchronization delay time of the camera exposure time according to the frame rate and installation angle;
[0045] Step S2, obtaining a reference time of the laser radar scanning reference angle, taking the reference time as the starting point, and controlling the corresponding camera exposure after the synchronization delay time has elapsed;
[0046] Step S3, obtaining the camera data and the lidar data at the time of exposure, and fusing the camera data and the lidar data at this time to obtain accurate time and space information.
[0047] In this embodiment, the Titan controller uses the GPS module input, and the FPGA parses the pps pulse and GPRMC data into UTC time, synchronizes it to Xavier, sends it to the laser radar, adjusts the synchronization pulse delay according to the camera installation angle, and outputs multiple GPS backup. Hardware connection is as follows Figure 2Assume that the laser radar rotates at 10rps (10 revolutions per second), the camera frame rate fps = 10fps (10 frames per second), set the reference angle (for example, the front is 0°), and install 8 cameras in a clockwise direction, as shown in the figure below. Figure 3 As shown. The laser radar scans and uploads the point cloud data of the corresponding angle in a clockwise direction, and contains angle information at 0°. The controller detects the 0° information and triggers the camera installed at 0° to be exposed synchronously. Cameras with different installation angles ang can set the delay dly relative to 0° according to the rotation speed of the laser radar, so that its exposure time is exactly the same as the time when the laser radar scans this angle. At this time, the laser radar data and the camera data can be fused to obtain accurate time and space information. The calculation formula for the synchronization delay time is: dly = (1 / fps) * (ang / 360), dly is the synchronization delay time, fps is the frame rate, and ang is the installation angle. The set value cnt corresponding to 10ns of the 100MHz counting clock is cnt = dly / 10ns. The synchronization delay time of each camera can be adjusted by Xavier according to the frame rate and installation position. In this way, this embodiment can calculate the synchronization delay time of the camera exposure time according to the frame rate and installation angle of the camera, so that its exposure time is exactly the same as the time when the laser radar scans this angle. At this time, the laser radar data and the camera data can be fused to obtain accurate time and space information. This synchronization technology uses FPGA timing processing. A clock frequency of 100MHz corresponds to a maximum delay of 10ns. Hardware delay is negligible. The SPI communication delay is small and fixed, and can be adjusted by software. The main delay time is the software interrupt and task scheduling delay of the operating system, which can be further optimized by optimizing the real-time processor and real-time operating system.
[0048] After synchronizing the scanning moment of the laser radar and the exposure moment of the camera, since the camera data is established on the image coordinate system, and the laser radar data is established on the basis of the laser radar coordinate system, this involves the transformation relationship between the image coordinate system, the laser radar coordinate system and the self-defined world coordinate system, and the camera data and the laser radar data need to be fused. In this embodiment, in step S3, the camera data and the laser radar data at this time are fused, including: obtaining the transformation relationship between the world coordinate system and the camera image coordinate system through the perspective transformation matrix; obtaining the transformation relationship between the world coordinate system and the laser radar coordinate system; determining the data mapping between the camera data and the laser radar data. In this way, the coordinate systems of the camera and the laser radar can be jointly calibrated through the world coordinate system, and the laser radar data can be mapped to the image plane to achieve data fusion. Specifically, the transformation relationship between the world coordinate system and the camera image coordinate system is obtained through the perspective transformation matrix, including:
[0049]
[0050] u and v are the horizontal and vertical coordinates of a point P in the camera image coordinate system, respectively. Xw, Yw, and Zw are the three-dimensional coordinates of point P in the world coordinate system, respectively. n1 to n11 are the elements in the perspective transformation matrix. The perspective transformation matrix is expressed as
[0051]
[0052] The perspective transformation matrix is a 3×4 matrix. Different points in the world coordinate system in space have unique corresponding points in the image coordinate system; and points in the image can correspond to objects in multiple world coordinate systems, so the matrix is not reversible. When there is a certain pair of world coordinates (Xw, Yw, Zw) and image coordinates (u, v), two equations can be determined. There are 11 position parameters for the perspective matrix, so as long as 6 pairs of world coordinate points and image coordinate points are selected, they can be obtained by solving the equation. The parameter calculation equation is as follows
[0053]
[0054] The above formula can be simplified to M·N=B. When enough world coordinate points and image coordinate points are selected, the least square method can be used to find the perspective transformation matrix (M T M) -1 M T B.
[0055] This embodiment obtains the conversion relationship between the world coordinate system and the image coordinate system through the perspective transformation matrix, and the joint calibration of the camera and the laser radar is to use the world coordinate system as an intermediate quantity to obtain the calibration relationship between the camera, the laser radar and the world coordinate system respectively, so as to determine the data mapping between the camera and the laser radar. In order to obtain the relationship between the laser radar coordinate system and the image coordinate system, it is necessary to determine the relationship between it and the world coordinate system. The vehicle coordinate system is defined as the world coordinate system, so in the actual calibration process, the laser radar is installed horizontally at the midpoint directly in front of the vehicle, so that in this embodiment, the conversion relationship between the world coordinate system and the laser radar coordinate system is obtained, including:
[0056]
[0057] r is the distance from the laser radar to this point in the laser radar coordinate system, θ is the angle swept by the laser radar, and h is the installation height of the laser radar. In this way, the relationship between the two coordinate systems can be obtained.
[0058] In order to obtain the corresponding relationship between (Xw, Yw, Zw) and (u, v), this embodiment preferably determines the data mapping between the camera data and the laser radar data, including: using the calibration column to realize the joint calibration of the laser radar and the camera to obtain the perspective transformation matrix. First, the coordinates (u, v) in the image space are obtained through the image information by obtaining the edge. Then, the laser radar is used to scan the calibration object, and the coordinates of the scanned part of the cylindrical surface are obtained. Due to the width problem, several data points are usually scanned. These data points are clustered to obtain the center point, which is used as the center of the cylinder, and its coordinates are set to (Xw, Yw, Zw). In order to obtain the perspective transformation matrix by the least squares method, the position of the calibration column is selected as 24, and four different positions are placed in each row, with a total of six rows. In this way, the data in the laser radar data space can be mapped to the image data space.
[0059] In this embodiment, due to the different acquisition frequencies of the camera and the laser radar, when the camera collects a frame of image data, the corresponding time label is t1, and the time label corresponding to the laser radar acquisition data frame is t2. These two times are divided into the recording time when the corresponding computer collects the data, which causes the inconsistency between the two in time and space, that is, the image data time and the laser radar data time are not the same. Although the data acquisition time is different, it can still be used for the joint calibration of the laser radar and the camera. Once the intelligent vehicle moves at high speed, the time difference between the two will affect the final detection. Therefore, how to reduce the synchronization problem of the camera and the laser radar sensor is another key issue in realizing data fusion. In short, joint calibration is the calibration of the camera and laser radar data space, while synchronization is the calibration of the camera and laser radar in time. Only when the calibration is complete in both "time" and "space" can the final experimental results be more accurate.
[0060] Figure 4The figure shows the relationship between the camera acquisition frequency and the laser radar acquisition frequency, where T1 represents the visual acquisition frequency, T2 represents the laser radar acquisition frequency, and the black arrow represents the trigger signal. When the trigger signal is generated, a frame of laser radar data and a frame of image data are collected at the same time. The acquisition principle uses the data frame closest to the trigger signal. In this way, due to the different acquisition frequencies, the time points between the two acquired data frames do not match. It can be seen from the figure that ΔT is the acquisition synchronization time difference. In order to complete time synchronization, this embodiment sets buffer pools at the two input ends to retain data reached at different times and record their timestamps. The function of each buffer pool is to perform a trigger action after entering a frame of data, save its data value and timestamp, and re-trigger and cycle when the next frame arrives. After buffer pool A is triggered, buffer pool B is notified immediately. Buffer pool B is set to the synchronization state after receiving the message. During this period of time, once data from the data source enters and is triggered, buffer pool A is notified, and then the two pools simultaneously bundle their respective data and add a unified timestamp (this time is the timestamp of the later data) to send to the subsequent processing module. In order to test the effectiveness of this method, a period of data is collected and the time difference of the data in the two cache pools is analyzed. Figure 5 As shown in the figure, the time difference is between 0.01s and 0.06s, and the average time difference is 0.02s, which is about 20ms. For vehicle detection, considering the length and width of the vehicle, a time error of 20ms will basically not affect the result.
[0061] like Figure 6 As shown, this embodiment also provides a camera and laser radar synchronization system for an intelligent driving control system, including:
[0062] A synchronization delay time calculation unit is used to set a reference angle for laser radar scanning, obtain the frame rate and installation angle of the camera, and calculate the synchronization delay time of the camera exposure time according to the frame rate and the installation angle;
[0063] The camera exposure control unit is used to obtain the reference time of the laser radar scanning reference angle, and control the corresponding camera exposure after the synchronization delay time has passed, taking the reference time as the starting point;
[0064] The data fusion processing unit is used to obtain the camera data and the lidar data during exposure, and to fuse the camera data and the lidar data at this time to obtain accurate time and space information.
[0065] The camera and lidar synchronization system of this embodiment can calculate the synchronization delay time of the camera exposure time according to the frame rate and installation angle of the camera, so that its exposure time is exactly the same as the time for the lidar to scan this angle. At this time, the lidar data and camera data can be fused to obtain accurate time and space information.
[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A camera and laser radar synchronization method for an intelligent driving control system, characterized in that: include: Step S1, setting the reference angle of the laser radar scanning, obtaining the frame rate and installation angle of the camera, and calculating the synchronization delay time of the camera exposure time according to the frame rate and installation angle; The reference angle of the laser radar scan is set in whole seconds based on the GNSS PPS signal. The GNSS PPS signal is connected to the laser radar and camera synchronization device at the same time. When the rising edge of the PPS signal arrives, the laser radar will rotate to the preset angle and obtain the 0 scale corresponding to the preset angle. The 0 scale is the reference angle of the laser radar scan. The synchronization delay time between the camera exposure and the 0 scale is calculated according to the frame rate of each camera, the relative angle with the laser radar, and the rotation frequency of the laser radar. Step S2, obtaining a reference time of the laser radar scanning reference angle, taking the reference time as the starting point, and controlling the corresponding camera exposure after the synchronization delay time has elapsed; Obtaining the reference time of the laser radar scanning reference angle, taking the reference time as the starting point, including: obtaining the moment of the second pulse PPS in each cycle of the laser radar as the 0th moment, taking the 0th moment as the reference, and performing camera exposure according to the synchronization delay time set in step S1; Step S3, acquiring the camera data and the laser radar data during exposure, and fusing the camera data and the laser radar data at this time to obtain accurate time and space information; In step S3, the camera data and the lidar data are fused, including: Obtain the conversion relationship between the world coordinate system and the camera image coordinate system through the perspective transformation matrix; Obtain the conversion relationship between the world coordinate system and the lidar coordinate system; Determine the data mapping between camera data and lidar data; Obtain the conversion relationship between the world coordinate system and the lidar coordinate system, including: r is the distance from the laser radar to this point in the laser radar coordinate system, θ is the angle swept by the laser radar, and h is the installation height of the laser radar; Determine the data mapping between camera data and lidar data, including: Use the calibration column to realize the joint calibration of LiDAR and camera and obtain the perspective transformation matrix; In step S3, the camera data and the lidar data are fused and processed, which also includes: Buffer pools are set up at the input ends of camera data and lidar data to retain data received at different times and record their timestamps. The function of each buffer pool is to perform a trigger action after entering a frame of data, save its data value and timestamp, and re-trigger and cycle when the next frame arrives. The transformation relationship between the world coordinate system and the camera image coordinate system is obtained through the perspective transformation matrix, including: u and v are the horizontal and vertical coordinates of a point P in the camera image coordinate system, respectively. Xw, Yw, and Zw are the three-dimensional coordinates of point P in the world coordinate system, respectively. n1 to n11 are the elements in the perspective transformation matrix, which is expressed as: Select multiple pairs of world coordinate points and image coordinate points to obtain parameter equations. The parameter equations are as follows:
2. The camera and laser radar synchronization method of the intelligent driving control system according to claim 1, characterized in that: Step S1 includes: dly=(1 / fps)*(ang / 360), dly is the synchronization delay time, fps is the frame rate, and ang is the installation angle.
3. A camera and laser radar synchronization system for an intelligent driving control system, characterized in that: include: A synchronization delay time calculation unit is used to set a reference angle for laser radar scanning, obtain the frame rate and installation angle of the camera, and calculate the synchronization delay time of the camera exposure time according to the frame rate and the installation angle; The reference angle of the laser radar scan is set in whole seconds based on the GNSS PPS signal. The GNSS PPS signal is connected to the laser radar and camera synchronization device at the same time. When the rising edge of the PPS signal arrives, the laser radar will rotate to the preset angle and obtain the 0 scale corresponding to the preset angle. The 0 scale is the reference angle of the laser radar scan. The synchronization delay time between the camera exposure and the 0 scale is calculated according to the frame rate of each camera, the relative angle with the laser radar, and the rotation frequency of the laser radar. The camera exposure control unit is used to obtain the reference time of the laser radar scanning reference angle, and control the corresponding camera exposure after the synchronization delay time has passed, taking the reference time as the starting point; Obtaining the reference time of the laser radar scanning reference angle, taking the reference time as the starting point, including: obtaining the moment of the second pulse PPS in each cycle of the laser radar as the 0th moment, taking the 0th moment as the reference, and performing camera exposure according to the synchronization delay time set in step S1; A data fusion processing unit is used to obtain the camera data and the lidar data during exposure, and to fuse the camera data and the lidar data at this time to obtain accurate time and space information; The camera data and lidar data are fused and processed, including: Obtain the conversion relationship between the world coordinate system and the camera image coordinate system through the perspective transformation matrix; Obtain the conversion relationship between the world coordinate system and the lidar coordinate system; Determine the data mapping between camera data and lidar data; Obtain the conversion relationship between the world coordinate system and the lidar coordinate system, including: r is the distance from the laser radar to this point in the laser radar coordinate system, θ is the angle swept by the laser radar, and h is the installation height of the laser radar; Determine the data mapping between camera data and lidar data, including: Use the calibration column to realize the joint calibration of LiDAR and camera and obtain the perspective transformation matrix; In step S3, the camera data and the lidar data are fused and processed, which also includes: Buffer pools are set up at the input ends of camera data and lidar data to retain data received at different times and record their timestamps. The function of each buffer pool is to perform a trigger action after entering a frame of data, save its data value and timestamp, and re-trigger and cycle when the next frame arrives. The transformation relationship between the world coordinate system and the camera image coordinate system is obtained through the perspective transformation matrix, including: u and v are the horizontal and vertical coordinates of a point P in the camera image coordinate system, respectively. Xw, Yw, and Zw are the three-dimensional coordinates of point P in the world coordinate system, respectively. n1 to n11 are the elements in the perspective transformation matrix, which is expressed as: Select multiple pairs of world coordinate points and image coordinate points to obtain parameter equations. The parameter equations are as follows:
Citation Information
Patent Citations
Laser radar and camera synchronization method and device, equipment and storage medium
CN111435162A
Target detection and motion state estimation method based on vision and laser radar
CN111951305A
Space-time synchronization system and device for laser radar and camera data and readable medium
CN112230240A
Panoramic camera and laser radar fusion device and calibration method thereof
CN112308927A
Sensor fusion system for controlling autonomous emergency braking in car
KR101921074B1
