A Compensation-Optimized Multi-Sensor Time Synchronization Method and System for Autonomous Driving

Abstract

This application provides a compensation-optimized multi-sensor time synchronization method for autonomous driving, comprising the following steps: GNSS is used to synchronize the time of LiDAR and IMU, and linear interpolation is performed on their timestamps to obtain a coarse time synchronization estimate; the difference between the IMU time-varying time offset and the LiDAR time-varying time offset is constructed, and this difference is added as a system state variable; the difference is compensated by IMU pre-integration to construct an IMU pre-integration residual containing the time-varying time offset difference; the LiDAR point cloud is processed by feature extraction to obtain line points and planar points, and the residuals of the line points and planar points are constructed to form the LiDAR residual; based on the tight coupling principle, the IMU pre-integration residual and the LiDAR residual are jointly optimized, and the time-varying time offset interpolation is iteratively optimized to a minimum value to eliminate its additional error influence on the positioning result, achieving accurate time synchronization between the IMU and LiDAR, thus ensuring the accuracy and robustness of multi-sensor fusion positioning. This application also provides a compensation-optimized multi-sensor time synchronization system for autonomous driving.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to a compensation-optimized method and system for multi-sensor time synchronization in autonomous driving. Background Technology

[0002] In the field of autonomous driving, localization methods based on multi-sensor fusion have gradually become a research hotspot, with the fusion of LiDAR and inertial measurement unit (IMU) being the most widely used. Sensor timestamps are typically affected by trigger and transmission delays, leading to varying degrees of time skew between data acquisition and transmission between different sensors, usually ranging from a few milliseconds to hundreds of milliseconds. Furthermore, after the processing unit receives the data, there is often misalignment of timestamps between sensors due to factors such as algorithm processing time and different time bases. This results in completely misaligned data streams from different sensors, severely impacting the algorithm's localization accuracy and robustness.

[0003] Therefore, it is necessary to provide a method and system for time synchronization of multiple sensors in autonomous driving based on compensation optimization. Summary of the Invention

[0004] The purpose of this application is to provide a compensation-optimized multi-sensor time synchronization method and system for autonomous driving. GNSS is used to synchronize the time of LiDAR and IMU, and linear interpolation is performed on their timestamps to obtain a rough time synchronization estimate. The difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR is constructed and added as a system state variable. This difference is compensated by IMU pre-integration to construct an IMU pre-integration residual containing the time-varying time offset difference. The LiDAR point cloud is divided into line points and planar points, and the residuals of the line points and planar points are constructed to form the LiDAR residual. Based on the tight coupling principle, the IMU pre-integration residual and the LiDAR residual are jointly optimized, and the time-varying time offset is iteratively optimized to a minimum value to eliminate its additional error influence on the positioning result, achieving precise time synchronization between the IMU and LiDAR, thus ensuring the accuracy and robustness of multi-sensor fusion positioning.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] A compensation-optimized method for time synchronization of multiple sensors in autonomous driving includes the following steps:

[0007] S1: Use GNSS to provide unified time synchronization for IMU and LiDAR, and perform linear interpolation on the timestamps of LiDAR and IMU to achieve coarse alignment of the timestamps of LiDAR and IMU;

[0008] S2: Construct the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, add this difference as a system state variable, compensate for this difference through IMU pre-integration, and construct the IMU pre-integration residual containing the time-varying time offset difference;

[0009] S3: Calculate the curvature of the point cloud in the LiDAR data, extract feature points that conform to line and surface features from the point cloud based on the curvature, project the feature points of the current frame to the previous frame, find the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and construct the point-line residual and the point-surface residual. The point-line residual and the point-surface residual together constitute the LiDAR residual.

[0010] S4: Based on the tight coupling principle, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

[0011] Preferably, the process of "linearly interpolating the IMU and LiDAR timestamps" in step S1 is as follows: the initial IMU timestamp is... The corresponding LiDAR timestamps before and after are respectively The corresponding LiDAR data at that time is IMU corresponding time points are obtained through linear interpolation. LiDAR data To achieve coarse synchronization of IMU and LiDAR timestamps, the interpolation formula is as follows:

[0012]

[0013] Preferably, the system's state variables are represented as follows:

[0014]

[0015] In the formula, X(t) represents the system state variable at time t; X b (t) represents the IMU state variables at time t, including translations relative to the world coordinate system. speed and rotation and accelerometer zero bias b a and gyroscope zero bias b g ;X bl The external parameters of the LiDAR coordinate system and the IMU coordinate system include the relative translation p. bl and relative rotation q bl ; The difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR; T is the matrix transpose operation;

[0016] The IMU measurement model is represented as follows:

[0017]

[0018]

[0019] In the formula, These represent the measured values ​​of acceleration and angular velocity at time t, respectively. These represent the true values ​​of acceleration and angular velocity at time t, respectively. For rotation from the world coordinate system to the IMU coordinate system; g w The acceleration due to gravity in the world coordinate system. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. The values ​​are Gaussian white noise for the accelerometer and gyroscope, respectively.

[0020] Preferably, i′ l and i′ b These are the data acquisition times for LiDAR and IMU, respectively. l j l i represents the time interval between two consecutive LiDAR data transmissions. b j b i represents the data transmission times of two consecutive IMUs. b Time to j l At time, then from i b Time to j l At any given moment, the process of updating the system's translation, velocity, and rotation is as follows:

[0021]

[0022]

[0023]

[0024] In the formula, They represent j respectively l Translation, velocity, and rotation of the time system; Representing i respectively b Translation, velocity, and rotation of the time system; Δt represents the translation, velocity, and rotation of two adjacent LiDAR data transmission times i. l and j l The time interval, Δt=j l -i l ; Indicates the current time relative to i b The rotation of time;

[0025] i b Time to jl At any given time, the translational, velocity, and rotational pre-integral quantities, which are only related to the IMU measurements, are expressed as:

[0026]

[0027]

[0028]

[0029] In the formula, These represent the translational, velocity, and rotational pre-integral values ​​related to the IMU measurements, respectively.

[0030] pre-integral quantity As a measurement value pair i b to j l Constraining the state variables between time points to construct a time difference containing time-varying time offsets The IMU pre-integration residual is expressed as:

[0031]

[0032] In the formula This represents the IMU pre-integration residual. Indicates the measured or estimated value of noise: and These are the translational residuals, velocity residuals, rotational residuals, and zero-bias residuals of the accelerometer and gyroscope, respectively.

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] In the formula, Indicates the world coordinate system to i b Rotation in the IMU coordinate system at any given moment; Indicate i l Translation, velocity, and rotation of the time system; Indicates the world coordinate system to i l Rotation in the IMU coordinate system at any given moment; Indicate i l Rotation of the time system; [·] xyz This refers to the operation of taking the imaginary part of a quaternion; Indicate jl Accelerometer bias at any given moment; Indicate i b Accelerometer bias at any given moment; Indicate j l The gyroscope bias at any given moment; Indicate j b The gyroscope bias at any given moment.

[0039] Preferably, step S3 specifically includes the following steps:

[0040] S31: Integrate the angular velocity and acceleration information obtained from the IMU measurement to calculate the relative change in pose between two cloud frames. The calculation process is expressed as follows:

[0041]

[0042]

[0043] In the formula, Let be the relative translational variable between the two cloud frames at time points; This represents the relative velocity variable between two cloud frames at different times; t represents the relative change in rotation between two cloud frames at time points; k t k+1 These represent the k-th and k+1-th timestamps, respectively; b k Indicates t k The IMU coordinate system at time t, where t represents [t k ,t k+1 At any time in ]; express n w Indicates gyroscope noise. This represents the measured angular velocity value. This represents the measured acceleration value. This indicates the current time t relative to t. k The rotational change in the IMU coordinate system at any given time. This represents the relative change in pose between two cloud frames;

[0044] S32: Based on timestamp t i ∈[t k ,t k+1 and relative change in pose For each point cloud coordinate P in the current frame k Linear interpolation is performed to obtain the distortion-free point cloud coordinates. The formula is:

[0045]

[0046] S33: Divide the current frame point cloud into several equal subsets, and then process each point P in each subset. i Select 5 points on each side to form a point set S, and calculate the curvature using the following formula:

[0047]

[0048] After calculating the curvature, the curvature of the point is compared with a preset curvature threshold. If it is greater than the threshold, the point is determined to be a line point; otherwise, the point is a plane point.

[0049] S34: Project the feature points whose categories have been determined onto the previous frame. For line points, find the two nearest points; for planar points, find the three nearest non-collinear points. Construct the LiDAR residual based on the distance relationship between line and planar points. The calculation process is represented as follows:

[0050]

[0051] In the formula, r l e r l s represents the LiDAR residuals constructed from line points and planar points, respectively; m is the feature line direction vector constructed from the point cloud; These are the feature points of the current frame and the previous frame, respectively; e i To find points on the feature line; They are respectively arrive The rotation and translation transformations; n is the normal vector of the characteristic surface, and D is the constant term in the general equation of the characteristic surface.

[0052] Preferably, step S4 specifically includes the following steps:

[0053] Joint optimization obtains the maximum a posteriori estimate by minimizing the sum of the prior and Mahalanobis norms of the IMU pre-integration residuals and the LiDAR residuals. The joint optimization process is expressed as follows:

[0054]

[0055] In the formula, and These are the LiDAR residuals and the IMU pre-integration residuals, respectively. Indicates r l e r l s sum.

[0056] This application also provides a compensation-optimized multi-sensor time synchronization system for autonomous driving, comprising:

[0057] Timestamp coarse alignment module: Uses GNSS to provide unified time synchronization for IMU and LiDAR, and performs linear interpolation on the timestamps of LiDAR and IMU to achieve coarse alignment of the timestamps of LiDAR and IMU;

[0058] IMU pre-integration residual calculation module: Constructs the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, adds this difference as a system state variable, compensates for this difference through IMU pre-integration, and constructs the IMU pre-integration residual containing the time-varying time offset difference;

[0059] The LiDAR residual calculation module calculates the curvature of the point cloud in the LiDAR data, extracts feature points that conform to line and surface features from the point cloud based on the curvature, projects the feature points of the current frame onto the previous frame, finds the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and constructs point-line residuals and point-surface residuals. The point-line residuals and point-surface residuals together constitute the LiDAR residuals.

[0060] Joint optimization module: Based on the principle of tight coupling, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

[0061] The beneficial effects of this application are as follows:

[0062] (1) LiDAR and IMU are unified to GNSS timing and linear interpolation is performed to eliminate the system error caused by different clock sources and to achieve rough alignment of timestamps between different frequency sensors.

[0063] (2) IMU measurements were used to distort the LiDAR point cloud, and LiDAR provided initial pose values, which improved the robustness of the laser odometry.

[0064] (3) The time-varying time offset is compensated as an additional state variable and tightly coupled with other state variables of the system for optimization. The time-varying time offset is iteratively optimized to the minimum value to achieve precise time synchronization between sensors. Attached Figure Description

[0065] Figure 1 A flowchart illustrating the compensation-optimized multi-sensor time synchronization method for autonomous driving provided in this application;

[0066] Figure 2 A schematic diagram showing the time offset of IMU and LiDAR;

[0067] Figure 3 A schematic diagram illustrating linear interpolation of IMU and LiDAR timestamps;

[0068] Figure 4This diagram illustrates the data processing flow for IMU and LiDAR data that takes into account time-varying time offsets. Detailed Implementation

[0069] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] Please refer to the following: Figure 1-4 This invention provides a compensation-optimized method and system for multi-sensor time synchronization in autonomous driving, comprising the following steps:

[0071] S1: GNSS (Global Navigation Satellite System) is used to provide unified time synchronization for IMU (Inertial Measurement Unit) and LiDAR (Light Detection and Ranging), and the timestamps of LiDAR and IMU are linearly interpolated to achieve coarse alignment between the timestamps of LiDAR and IMU.

[0072] There is typically a fixed time offset between IMUs and LiDARs. This fixed time offset refers to the misalignment of timestamps caused by the time it takes for the sensor to send data to the processing unit and for the processing unit to process the data. Fixed time offsets can be directly read from the system time without additional optimization. For example... Figure 2 As shown, the top is the LiDAR timeline, and the bottom is the IMU timeline. The downward arrows indicate the timestamps on the corresponding timelines. 1 represents the data acquisition time of both LiDAR and IMU. Their trigger and transmission delays are as follows: and difference This offset is a time-varying time offset, which can be incorporated into the system state variables and jointly optimized using a compensation-based optimization method to achieve convergence. The time consumed from the sensor sending data to the processing unit receiving and processing the data are respectively... and These two time offsets are fixed time offsets, meaning their offsets can be accurately read from the system time synchronized by GNSS, and precise alignment can be achieved using linear interpolation. Data transmission time 1 from the processing unit. ′ That is, the time t is the moment when the data from the two sensors are fused. l and t b These represent the total time offsets of LiDAR and IMU data, with a difference of t.d By processing the time-varying and fixed time offsets as described above, the total time offset can be eliminated, thus achieving precise time synchronization.

[0073] GNSS time synchronization has the advantages of high accuracy and wide coverage. By unifying LiDAR and IMU into GNSS time synchronization and performing linear interpolation, the systematic errors caused by different clock sources are eliminated, and coarse alignment of timestamps between different frequency sensors is achieved.

[0074] Please see Figure 3 The process of linear interpolating the IMU and LiDAR timestamps is as follows: The initial IMU timestamp is... The corresponding LiDAR timestamps before and after are respectively Assumption The corresponding LiDAR data at that time is IMU corresponding time points are obtained through linear interpolation. LiDAR data To achieve coarse synchronization of IMU and LiDAR timestamps, the interpolation formula is as follows:

[0075]

[0076] S2: Construct the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, add this difference as a system state variable, compensate for this difference through IMU pre-integration, and construct the IMU pre-integration residual containing the time-varying time offset difference.

[0077] Sensors often experience trigger and transmission delays during data acquisition, leading to misalignment of sensor timestamps and resulting in time offsets. These time offsets change with the sampling time; this is known as time-varying time offset. During IMU and LiDAR time synchronization, the difference between the IMU's and LiDAR's time-varying time offsets needs to be compensated. Optimizing this compensation amount along with the IMU's state variables as system state variables allows for iterative optimization to minimize the time-varying time offset.

[0078] Please refer to the following: Figure 2 and Figure 4 i l ′ and i ′ b These are the data acquisition times for LiDAR and IMU, respectively. l j l i represents the time interval between two consecutive LiDAR data transmissions. b j bThe timestamps are the data transmission times of two consecutive IMUs, respectively. The time delay between the IMU data acquisition time and the IMU data transmission time is the time-varying time offset of the IMU, denoted as . The time delay between the LiDAR data acquisition time and the LiDAR data transmission time is the time-varying time offset of LiDAR, denoted as . Right now: Because IMU data frequencies are typically higher than LiDAR data frequencies, Greater than At this point, the difference between the two is It is a positive value.

[0079] The state variables of the system are represented as follows:

[0080]

[0081] In the formula, X(t) represents the system state variable at time t; X b (t) represents the IMU state variables at time t, including translations relative to the world coordinate system. speed and rotation and accelerometer zero bias b a and gyroscope zero bias b g ;X bl The external parameters of the LiDAR coordinate system and the IMU coordinate system include the relative translation p. bl and relative rotation q bl T represents the matrix transpose operation.

[0082] The IMU measurement model is represented as follows:

[0083]

[0084]

[0085] In the formula, These represent the measured values ​​of acceleration and angular velocity at time t, respectively. These represent the true values ​​of acceleration and angular velocity at time t, respectively. For rotation from the world coordinate system to the IMU coordinate system; g w The acceleration due to gravity in the world coordinate system. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. The values ​​are Gaussian white noise for the accelerometer and gyroscope, respectively.

[0086] For measured values and Pre-integration is performed in two time periods, one of which is from the IMU data transmission time i. b up to LiDAR data transmission time i l The other segment is from the LiDAR data transmission time i l To the next LiDAR data transmission time k adjacent to this time l i b Time to k l At any given moment, the process of updating the system's translation, velocity, and rotation is as follows:

[0087]

[0088]

[0089]

[0090] In the formula, They represent j respectively l Translation, velocity, and rotation of the time system; Representing i respectively b Translation, velocity, and rotation of the time system; Δt represents the translation, velocity, and rotation of two adjacent LiDAR data transmission times i. l and j l The time interval, Δt=j l -i l ; Indicates the current time relative to i b The rotation of time;

[0091] j b Time to j l At any given time, the translational, velocity, and rotational pre-integral quantities, which are only related to the IMU measurements, are expressed as:

[0092]

[0093]

[0094]

[0095] In the formula, These represent the translational, velocity, and rotational pre-integral values ​​associated with the IMU measurements, respectively.

[0096] pre-integral quantity As a measurement value pair i b to j l Constraining the state variables between time points to construct a time difference containing time-varying time offsets The IMU pre-integration residual is expressed as:

[0097]

[0098] In the formula This represents the IMU pre-integration residual. Indicates the measured or estimated value of noise: and These are the IMU translation residual, velocity residual, rotation residual, and zero-bias residual of the accelerometer and gyroscope, respectively; among which,

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] In the formula, Indicates the world coordinate system to i b Rotation in the IMU coordinate system at any given moment; Indicate i l Translation, velocity, and rotation of the time system; Indicates the world coordinate system to i l Rotation in the IMU coordinate system at any given moment; Indicate i l Rotation of the time system; [·] xyz This refers to the operation of taking the imaginary part of a quaternion; Indicate j l Accelerometer bias at any given moment; Indicate i b Accelerometer bias at any given moment; Indicate j l The gyroscope bias at any given moment; Indicate i b The gyroscope bias at any given moment.

[0105] In the process of calculating the IMU pre-integration residual, the impact of time-varying time offset on the overall positioning system is fully considered to improve the accuracy of IMU pre-integration.

[0106] S3: Calculate the curvature of the point cloud in the LiDAR data, extract feature points that conform to line and surface features from the point cloud based on the curvature, project the feature points of the current frame onto the previous frame, find the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and construct the point-line residual and the point-surface residual. The point-line residual and the point-surface residual together constitute the LiDAR residual.

[0107] Step S3 specifically includes the following steps:

[0108] S31: Integrate the angular velocity and acceleration information obtained from the IMU measurement to calculate the relative change in pose between two cloud frames. The calculation process is expressed as follows:

[0109]

[0110]

[0111] In the formula, Let be the relative translational variable between the two cloud frames at time points; This represents the relative velocity variable between two cloud frames at different times; t represents the relative change in rotation between two cloud frames at time points; k t k+1 These represent the k-th and k+1-th timestamps, respectively; b k Indicates t k The IMU coordinate system at time t, where t represents [t k ,t k+1 At any time in ]; express n w Indicates gyroscope noise. This represents the measured angular velocity value. This represents the measured acceleration value. This indicates the current time t relative to t. k The rotational change in the IMU coordinate system at any given time. This represents the relative change in pose between two cloud frames;

[0112] S32: Based on a uniform motion model, according to the timestamp t i ∈[t k ,t k+1 and relative change in pose For each point cloud coordinate P in the current frame k Linear interpolation is performed to obtain the distortion-free point cloud coordinates. The formula is:

[0113]

[0114] S33: Divide the current frame point cloud into several equal subsets, and then process each point P in each subset. i Select 5 points on each side to form a point set S, and calculate the curvature using the following formula:

[0115]

[0116] After calculating the curvature, the curvature of the point is compared with a preset curvature threshold. If it is greater than the threshold, the point is determined to be a line point; otherwise, the point is a plane point.

[0117] S34: Project the feature points whose categories have been determined onto the previous frame, find the nearest line corresponding to the line point, and the nearest plane corresponding to the plane point. Construct the LiDAR residual based on the distance relationship between the line point and the plane point. The calculation process is represented as follows:

[0118]

[0119] In the formula, r l e r l s represents the LiDAR residuals constructed from line points and planar points, respectively; m is the feature line direction vector constructed from the point cloud; Point clouds for the current frame and the previous frame, respectively; e i To find points on the feature line; They are respectively arrive The rotation and translation transformations; n is the normal vector of the characteristic surface, and D is the constant term in the general equation of the characteristic surface.

[0120] IMU measurements were used to distort LiDAR point clouds and provide initial pose values ​​for LiDAR, thus improving the robustness of LiDAR.

[0121] S4: Based on the tight coupling principle, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

[0122] Joint optimization obtains the maximum a posteriori estimate by minimizing the sum of the prior and Mahalanobis norms of the IMU pre-integration residual and the LiDAR residual, thus eliminating the additional error they cause to the positioning results. This iteratively minimizes the time-varying time offset, improving the accuracy and robustness of LiDAR and IMU fusion positioning. The joint optimization process is expressed as follows:

[0123]

[0124] In the formula, and These are the LiDAR residuals and the IMU pre-integration residuals, respectively. Indicates r l e r l s sum.

[0125] This application also provides a compensation-optimized multi-sensor time synchronization system for autonomous driving, comprising:

[0126] Timestamp coarse alignment module: Uses GNSS to provide unified time synchronization for IMU and LiDAR, and performs linear interpolation on the timestamps of LiDAR and IMU to achieve coarse alignment of the timestamps of LiDAR and IMU;

[0127] IMU pre-integration residual calculation module: Constructs the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, adds this difference as a system state variable, compensates for this difference through IMU pre-integration, and constructs the IMU pre-integration residual containing the time-varying time offset difference;

[0128] The LiDAR residual calculation module calculates the curvature of the point cloud in the LiDAR data, extracts feature points that conform to line and surface features from the point cloud based on the curvature, projects the feature points of the current frame onto the previous frame, finds the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and constructs point-line residuals and point-surface residuals. The point-line residuals and point-surface residuals together constitute the LiDAR residuals.

[0129] Joint optimization module: Based on the principle of tight coupling, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

[0130] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A compensation-optimized multi-sensor time synchronization method for autonomous driving, characterized in that, Includes the following steps: S1: Use GNSS to provide unified time synchronization for IMU and LiDAR, and perform linear interpolation on the timestamps of LiDAR and IMU to achieve coarse alignment of the timestamps of LiDAR and IMU; S2: Construct the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, add this difference as a system state variable, compensate for this difference through IMU pre-integration, and construct the IMU pre-integration residual containing the time-varying time offset difference; S3: Calculate the curvature of the point cloud in the LiDAR data, extract feature points that conform to line and surface features from the point cloud based on the curvature, project the feature points of the current frame to the previous frame, find the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and construct the point-line residual and the point-surface residual. The point-line residual and the point-surface residual together constitute the LiDAR residual. S4: Based on the tight coupling principle, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

2. The method for multi-sensor time synchronization in autonomous driving based on compensation optimization according to claim 1, characterized in that, The process of "linearly interpolating the IMU and LiDAR timestamps" in step S1 is as follows: the initial IMU timestamp is... The corresponding LiDAR timestamps before and after are respectively The corresponding LiDAR data at that time is IMU corresponding time points are obtained through linear interpolation. LiDAR data To achieve coarse synchronization of IMU and LiDAR timestamps, the interpolation formula is as follows:

3. The method for multi-sensor time synchronization in autonomous driving based on compensation optimization according to claim 1, characterized in that, The state variables of the system are represented as follows: In the formula, X(t) represents the system state variable at time t; X b (t) represents the IMU state variables at time t, including translations relative to the world coordinate system. speed and rotation and accelerometer zero bias b a and gyroscope zero bias b g ;X bl The external parameters of the LiDAR coordinate system and the IMU coordinate system include the relative translation p. bl and relative rotation q bl ; The difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR; T is the matrix transpose operation; The IMU measurement model is represented as follows: In the formula, These represent the measured values ​​of acceleration and angular velocity at time t, respectively. These represent the true values ​​of acceleration and angular velocity at time t, respectively. For rotation from the world coordinate system to the IMU coordinate system; g w The acceleration due to gravity in the world coordinate system. and These are the zero bias values ​​for the accelerometer and gyroscope, respectively. The values ​​are Gaussian white noise for the accelerometer and gyroscope, respectively.

4. The multi-sensor time synchronization method for autonomous driving based on compensation optimization according to claim 3, characterized in that, i′ l and i′ b These are the data acquisition times for LiDAR and IMU, respectively. l j l i represents the time interval between two consecutive LiDAR data transmissions. b j b i represents the data transmission times of two consecutive IMUs. b Time to j l At time, then from i b Time to j l At any given moment, the process of updating the system's translation, velocity, and rotation is as follows: In the formula, They represent j respectively l Translation, velocity, and rotation of the time system; Representing i respectively b Translation, velocity, and rotation of the time system; Δt represents the translation, velocity, and rotation of two adjacent LiDAR data transmission times i. l and j l The time interval, Δt = j l -i l ; Indicates the current time relative to i b The rotation of time; i n Time to j l At any given time, the translational, velocity, and rotational pre-integral quantities, which are only related to the IMU measurements, are expressed as: In the formula, These represent the translational, velocity, and rotational pre-integral values ​​related to the IMU measurements, respectively. pre-integral quantity As a measurement value pair i b to j l Constraining the state variables between time points to construct a time difference containing time-varying time offsets The IMU pre-integration residual is expressed as: In the formula This represents the IMU pre-integration residual. Indicates the measured or estimated value of noise: and These are the translational residuals, velocity residuals, rotational residuals, and zero-bias residuals of the accelerometer and gyroscope, respectively. In the formula, Indicates the world coordinate system to i b Rotation in the IMU coordinate system at any given moment; Indicate i l Translation, velocity, and rotation of the time system; Indicates the world coordinate system to i l Rotation in the IMU coordinate system at any given moment; Indicate i l Rotation of the time system; [·] xyz This refers to the operation of taking the imaginary part of a quaternion; Indicate j l Accelerometer bias at any given moment; Indicate i b Accelerometer bias at any given moment; Indicate j l The gyroscope bias at any given moment; Indicate i b The gyroscope bias at any given moment.

5. The multi-sensor time synchronization method for autonomous driving based on compensation optimization according to claim 4, characterized in that, Step S3 specifically includes the following steps: S31: Integrate the angular velocity and acceleration information obtained from the IMU measurement to calculate the relative change in pose between two cloud frames. The calculation process is expressed as follows: In the formula, Let be the relative translational variable between the two cloud frames at time points; This represents the relative velocity variable between two cloud frames at different times; t represents the relative change in rotation between two cloud frames at time points; k t k+1 These represent the k-th and k+1-th timestamps, respectively; b k Indicates t k The IMU coordinate system at time t, where t represents [t k ,t k+1 At any time in ]; express n w Indicates gyroscope noise. This represents the measured angular velocity value. This represents the measured acceleration value. This indicates the current time t relative to t. k The rotational change in the IMU coordinate system at any given time. This represents the relative change in pose between two cloud frames; S32: Based on timestamp t i ∈[t k ,t k+1 and relative change in pose For each point cloud coordinate P in the current frame k Linear interpolation is performed to obtain the distortion-free point cloud coordinates. The formula is: S33: Divide the current frame point cloud into several equal subsets, and then process each point P in each subset. i Select 5 points on each side to form a point set S, and calculate the curvature using the following formula: After calculating the curvature, the curvature of the point is compared with a preset curvature threshold. If it is greater than the threshold, the point is determined to be a line point; otherwise, the point is a plane point. S34: Project the feature points whose categories have been determined onto the previous frame. For line points, find the two nearest points; for planar points, find the three nearest non-collinear points. Construct the LiDAR residual based on the distance relationship between line and planar points. The calculation process is represented as follows: In the formula, represents the LiDAR residuals constructed from line points and planar points, respectively; m is the feature line direction vector constructed from the point cloud; These are the feature points of the current frame and the previous frame, respectively; e i To find points on the feature line; They are respectively arrive The rotation and translation transformations; n is the normal vector of the characteristic surface, and D is the constant term in the general equation of the characteristic surface.

6. The multi-sensor time synchronization method for autonomous driving based on compensation optimization according to claim 5, characterized in that, Step S4 specifically includes the following steps: Joint optimization obtains the maximum a posteriori estimate by minimizing the sum of the prior and Mahalanobis norms of the IMU pre-integration residuals and the LiDAR residuals. The joint optimization process is expressed as follows: In the formula, and These are the LiDAR residuals and the IMU pre-integration residuals, respectively. express sum.

7. A multi-sensor time synchronization system for autonomous driving based on compensation optimization, characterized in that, include: Timestamp coarse alignment module: Uses GNSS to provide unified time synchronization for IMU and LiDAR, and performs linear interpolation on the timestamps of LiDAR and IMU to achieve coarse alignment of the timestamps of LiDAR and IMU; IMU pre-integration residual calculation module: Constructs the difference between the time-varying time offset of the IMU and the time-varying time offset of the LiDAR, adds this difference as a system state variable, compensates for this difference through IMU pre-integration, and constructs the IMU pre-integration residual containing the time-varying time offset difference; The LiDAR residual calculation module calculates the curvature of the point cloud in the LiDAR data, extracts feature points that conform to line and surface features from the point cloud based on the curvature, projects the feature points of the current frame onto the previous frame, finds the nearest line corresponding to the line point and the nearest plane corresponding to the plane point, and constructs point-line residuals and point-surface residuals. The point-line residuals and point-surface residuals together constitute the LiDAR residuals. Joint optimization module: Based on the principle of tight coupling, the Ceres optimization solver is used to jointly optimize the IMU pre-integration residual and the LiDAR residual to obtain the vehicle positioning result after minimizing the time offset and achieving precise time synchronization.

Images