Data synchronization and drift calibration method for experimental vehicle, electronic device and storage medium

Abstract

This invention discloses a data synchronization and drift calibration method, electronic device, and storage medium for experimental vehicles, relating to the field of vehicle perception data processing technology. The method includes: acquiring visual image data, lidar point cloud data, and motion state data of the experimental vehicle within the current time window; resampling the visual image data based on a preset data sampling weight to obtain synchronized image data aligned with the timestamp of the point cloud data; extracting the semantic center coordinates of a predetermined target object in the synchronized image data and the geometric center coordinates in the lidar point cloud data; and projecting the semantic center coordinates onto the three-dimensional coordinate system of the lidar point cloud data based on a preset projection mapping matrix to obtain the three-dimensional coordinates of the target object inferred from the synchronized image data. Its beneficial effect is that it prevents data distortion caused by accidental calibration triggers, thereby ensuring the authenticity and continuity of the experimental vehicle's perception analysis results.

Description

Technical Field

[0001] This invention relates to the field of vehicle perception data processing technology, and in particular to a data synchronization and drift calibration method, electronic device and storage medium for experimental vehicles. Background Technology

[0002] In autonomous driving and vehicle dynamics experiments, experimental vehicles are usually equipped with a variety of heterogeneous sensors such as vision sensors, lidar, and inertial measurement units. In order to obtain accurate information about the vehicle's surrounding environment, it is necessary to perform strict temporal and spatial alignment on the multi-source perception data to ensure that the perception data is fused and replayed under a unified spatiotemporal reference.

[0003] However, existing technologies for sensor drift calibration primarily rely on periodic offline calibration or static correction based on pre-defined physical models. When dealing with the complex, long-term dynamic conditions of experimental vehicles, this approach often only compensates for isolated deviations in a single sensor, making it difficult to establish a mapping relationship between multi-source sensing data observations and underlying operational state deviations. The drawback of existing technologies lies in their primarily open-loop calibration logic, lacking the ability to continuously and dynamically optimize underlying synchronization parameters based on real-time sensing differences. This makes the synchronization accuracy of experimental data in the spatiotemporal dimensions highly susceptible to cumulative deviations with increasing mileage, leading to a continuous decline in synchronization performance and ultimately affecting the reliability and realism of the experimental vehicle's perception analysis. Summary of the Invention

[0004] In view of the above-mentioned prior art, this application is hereby filed. Embodiments of this application provide a data synchronization and drift calibration method, electronic equipment, and storage medium for experimental vehicles, which can ensure the purity of the data source participating in the reverse computation, prevent data distortion caused by accidental calibration, and thus guarantee the authenticity and continuity of the experimental vehicle's perception analysis results.

[0005] According to one aspect of this application, a method for data synchronization and drift calibration for experimental vehicles is provided, including:

[0006] The system acquires visual image data, lidar point cloud data, and motion state data of the experimental vehicle in the current time window. Based on a preset data sampling weight, it resamples the visual image data to obtain synchronized image data that is aligned with the timestamp of the point cloud data. The system then extracts the semantic center coordinates of the predetermined target object in the synchronized image data and the geometric center coordinates in the lidar point cloud data.

[0007] Based on a preset projection mapping matrix, the semantic center coordinates are projected onto the three-dimensional coordinate system where the lidar point cloud data is located, to obtain the three-dimensional coordinates of the target object inferred from the synchronized image data;

[0008] Based on the semantic tags of the target object, a cross-modal semantic association matrix of the semantic center coordinates and geometric center coordinates of the target object is established. Based on the spatial vector displacement between each set of associated coordinates in the cross-modal semantic association matrix, the semantic consistency residual is quantified.

[0009] Based on the semantic consistency residuals at each time point of the current time window, a dynamic residual field is constructed, which is formed by the evolution of the semantic consistency residuals over time. Each semantic consistency residual is associated with the motion state data at the corresponding time point.

[0010] Based on the spatial distribution directionality and temporal evolution phase characteristics of the semantically consistent residuals in the dynamic residual field, the directional deviation component characterizing the spatial installation offset of the sensor and the temporal offset component characterizing the time asynchrony of the sensor are extracted respectively. Based on the directional deviation component and the temporal offset component, the dynamic drift parameters including clock drift increment and external parameter deformation displacement are obtained.

[0011] The data sampling weights are corrected and the projection mapping matrix is ​​updated based on the clock drift increment and the extrinsic deformation displacement.

[0012] The spatial and temporal drift between the visual image data and the lidar point cloud data is eliminated by using the corrected data sampling weights and the updated projection mapping matrix to obtain the synchronously calibrated experimental vehicle dataset.

[0013] According to another aspect of this application, an electronic device is provided, including a memory and a processor, the memory being used to store computer-executable instructions, and the processor being used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method described above.

[0014] According to another aspect of this application, a computer storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0015] Compared with existing technologies, the data synchronization and drift calibration method, electronic equipment and storage medium for experimental vehicles according to the embodiments of this application precisely decouple the mixed deviation into physical extrinsic parameter deformation and timing clock drift, eliminating data representation drift from the root cause and greatly improving the synchronization accuracy of experimental data. By directly using the semantic association matrix of the visual center and the geometric center as a benchmark, this mechanism of mutual verification and calibration between sensors enables the experimental vehicle to still have strong self-repair capabilities in unknown and non-standard scenarios, ensuring the purity of the data source participating in the reverse calculation, preventing data distortion caused by accidental calibration, and thus ensuring the authenticity and continuity of the perception analysis results of the experimental vehicle. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 This is a schematic diagram of the overall process of the data synchronization and drift calibration method for experimental vehicles according to the present invention.

[0018] Figure 2 This is a schematic diagram of the hierarchical calibration method for data synchronization and drift calibration of experimental vehicles according to the present invention.

[0019] Figure 3 This is a schematic diagram illustrating the structural confidence assessment of the data synchronization and drift calibration method for experimental vehicles according to the present invention. Detailed Implementation

[0020] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0021] In existing technologies, multi-sensor fusion technology for experimental vehicles is evolving towards higher precision and long-term stability. However, data synchronization and drift calibration under complex dynamic conditions still face significant challenges. For experimental vehicles, long-term driving or vibrations in complex road conditions often cause sub-millisecond jitter in the hardware clock and slight deformation of the sensor mounting brackets. Traditional offline calibration or static compensation methods can only handle fixed deviations and are unable to perceive and correct the cumulative drift of data in the spatiotemporal dimension in real time. This results in inconsistencies between visual and point cloud data representations, affecting the authenticity of autonomous driving algorithm verification and vehicle dynamics analysis.

[0022] To address the aforementioned issues, the applicant discovered that the core of the perception data synchronization failure lies in the lack of a deep mapping between observation bias and the underlying hardware drift state. Analysis revealed that the semantic representation of the same target by different modal sensors should possess spatial consistency, and its deviation vector can be quantified as a systematic residual. Furthermore, this residual exhibits a regular spatiotemporal evolution with the vehicle's motion state. Based on this, a technical approach combining semantic residual field construction, cross-modal parameter reverse calculation, and closed-loop dynamic correction is proposed: First, the semantic / geometric center coordinates of visual and point cloud data are extracted and spatial vector displacements are quantified to construct a dynamic residual field that evolves over time. Then, the displacement pressure generated by the residual field is transformed into a calibration driving force, and the dynamic drift parameters, including clock drift increments and extrinsic parameter deformation displacements, are obtained through reverse calculation. Finally, closed-loop self-repair of the perception data stream is achieved by real-time correction of sampling weights and updating the projection mapping matrix.

[0023] This application can effectively eliminate the spatiotemporal cumulative deviation caused by long-term operation of experimental vehicles, and achieve sub-millisecond-level synchronization and centimeter-level calibration of perception data without the need for external high-precision benchmarks. It significantly improves the confidence of experimental datasets and the robustness of system operation, and provides a precise data foundation for the development and testing of advanced autonomous driving technologies.

[0024] Example 1:

[0025] Reference Figures 1-3 As an embodiment of the present invention, a data synchronization and drift calibration method for experimental vehicles is provided:

[0026] Figure 1 The illustration shows a data synchronization and drift calibration method for experimental vehicles according to an embodiment of this application, including:

[0027] The system acquires visual image data, lidar point cloud data, and motion state data of the experimental vehicle within the current time window. Based on a preset data sampling weight, it resamples the visual image data to obtain synchronized image data aligned with the timestamp of the point cloud data. It then extracts the semantic center coordinates of the predetermined target object in the synchronized image data and the geometric center coordinates in the lidar point cloud data. Based on a preset projection mapping matrix, it projects the semantic center coordinates onto the three-dimensional coordinate system of the lidar point cloud data to obtain the three-dimensional coordinates of the target object inferred from the synchronized image data.

[0028] Specifically, the experiment acquires visual image data collected by the vehicle's onboard vision sensor during operation, as well as point cloud data synchronously collected by the LiDAR sensor. After acquiring the data, the visual image data is resampled using data sampling weights to align it with the trigger pulse of the LiDAR point cloud on the timestamp axis, thereby generating data pairs with the same temporal attributes. Based on a preset projection mapping matrix, the three-dimensional spatial coordinates in the LiDAR point cloud data are projected onto the two-dimensional pixel coordinate system of the visual image. After completing the initial spatial alignment, semantic feature recognition is performed on the visual image data to extract the pixel center points of preset targets such as traffic signs, light poles, or curbs in the image, obtaining the semantic center coordinates of the targets. Simultaneously, a clustering algorithm is used to segment the corresponding targets in the point cloud data into point cloud clusters, calculate the geometric centroid of the point cloud cluster in three-dimensional space, and use it as the geometric center coordinates of the corresponding targets in the LiDAR point cloud data. Through the above steps, core positional features with the same semantic attributes are extracted from multi-source heterogeneous data, providing basic data support for the subsequent construction of semantic consistency residuals.

[0029] Based on the semantic tags of the target object, a cross-modal semantic association matrix of the semantic center coordinates and geometric center coordinates of the target object is established. Based on the spatial vector displacement between each set of associated coordinates in the cross-modal semantic association matrix, the semantic consistency residual is quantified.

[0030] Specifically, firstly, the semantic attribute labels carried by the semantic center coordinates and geometric center coordinates of the target object are extracted. Based on the consistency of the semantic attribute label categories, visual feature points belonging to the same traffic target are logically paired with lidar feature points, and a cross-modal semantic association matrix is ​​constructed accordingly. For each pair of associated coordinates in the cross-modal semantic association matrix, cross-modal spatial coordinate alignment and difference operations are performed to quantify the spatial vector displacement between visual perception and radar perception. The specific formula is as follows:

[0031] ;

[0032] in, The semantic consistency residual represents the vector displacement in a three-dimensional coordinate system. Its magnitude represents the magnitude of the perception bias, and its direction represents the spatial orientation of the perception misalignment. It reflects the degree of inconsistency in the spatial dimension of the experimental vehicle's perception system under the current calibration parameters. This represents the geometric center coordinates of the target extracted from the lidar point cloud data, and represents the target position vector observed by the radar in three-dimensional physical space. This represents the semantic center coordinates of the target object extracted from visual image data, with its initial dimension being pixel coordinates. This represents the preset projection mapping matrix, which serves as the spatial projection operator in this formula. It is used to map the semantic center coordinates on the two-dimensional pixel plane to the three-dimensional physical space to which the lidar belongs, based on the camera's intrinsic and initial extrinsic parameters.

[0033] Based on the semantic consistency residuals at each time point in the current time window, a dynamic residual field is constructed, which is formed by the evolution of semantic consistency residuals over time. Each semantic consistency residual is associated with the motion state data at the corresponding time point.

[0034] Specifically, real-time motion state data of the experimental vehicle is acquired synchronously. This data includes at least the vehicle's speed, steering angle vector, and acceleration at the current sampling moment. Using this data, a relative motion mapping relationship is established between the spatial coordinates of the semantic consistency residuals at each moment and the vehicle's trajectory. Subsequently, the quantized semantic consistency residuals are sequentially arranged according to their timestamps. Combined with the relative motion mapping relationship, the discrete residual vectors are projected onto a unified local vehicle coordinate system. By accumulating all residual vectors within a preset time period along the temporal dimension, a dynamic residual field reflecting the evolution of perception bias over time is constructed, as shown below:

[0035] ;

[0036] in, The dynamic residual field is essentially a spatiotemporal data set containing residual vector features and vehicle motion features. This represents the semantic consistency residual obtained by quantization at time t, reflecting the magnitude and direction of the spatial perception bias at that time. To represent the real-time motion state vector of the experimental vehicle at time t, serving as the motion constraint background for residual evolution, This indicates the preset sliding window time interval, defining the observation span of the dynamic residual field in the time dimension;

[0037] In the process of constructing the dynamic residual field F, the core key is to achieve a deep logical binding between the semantically consistent residual R and the motion state of the experimental vehicle. That is, to ensure that each semantically consistent residual R is associated with motion state data at a corresponding time point. In specific implementation, the system provides a synchronization reference for the visual sensor, LiDAR and vehicle motion sensor through a unified hardware triggering mechanism or a high-precision external timestamp server, so that at any sampling time t, while quantizing the semantically consistent residual R, the system can simultaneously capture the physical operation characteristics of the experimental vehicle at that time.

[0038] The physical significance of the correlation operation lies in that it gives each set of perception deviations a clear kinematic background. Due to the regular differences in the residual characteristics exhibited by the sensor's small jitter on the time axis and physical displacement in the spatial dimension under different vehicle motion conditions, by temporally coupling the semantically consistent residual R with the real-time motion state vector V(t) of the experimental vehicle at the corresponding time t, a deterministic mapping relationship of the deviation evolving with the time series can be established. At the data structure level, this correlation is reflected in the formation of a series of spatiotemporal data sets containing {R(t), V(t)} in the dynamic residual field F. This allows the system to not only record the instantaneous state of the perception deviation, but also the motion constraint background when the deviation occurs. In this way, the discrete residual vector is transformed into a trajectory field with an evolutionary trend, thereby providing a complete spatiotemporal observation sample for subsequently extracting the systematic drift parameters through reverse calculation.

[0039] The semantic consistency residual obtained by quantization at time t The real-time motion state vector of the experimental vehicle at time t By performing temporal coupling, the residual field is no longer an isolated static deviation, but forms a trajectory field with an evolutionary trend. Using this dynamic residual field, the distribution law of the residual under different motion conditions can be observed (e.g., whether the residual increases linearly with the increase of speed, or changes in direction with the change of steering angle), thus providing a complete spatiotemporal observation sample for the subsequent extraction of systematic drift parameters through inverse calculation.

[0040] Based on the spatial distribution directionality and temporal evolution phase characteristics of semantically consistent residuals in the dynamic residual field, the directional deviation component characterizing the spatial installation offset of the sensor and the temporal offset component characterizing the time asynchrony of the sensor are extracted respectively. Based on the directional deviation component and the temporal offset component, the dynamic drift parameters, including clock drift increment and external parameter deformation displacement, are obtained by inverse calculation.

[0041] Specifically, the extraction of the directional deviation component includes:

[0042] The dynamic residual field is decomposed into sub-residual sets corresponding to different motion modes using motion state data. The motion modes include at least uniform linear motion and nonlinear directional motion modes. The mean projection of each sub-residual set onto each axis of the three-dimensional coordinate system is extracted, and the cooperative variation trend of the mean projection with the velocity vector changes in the motion state data is calculated. The formula for calculating the mean projection is as follows:

[0043] ;

[0044] in, This represents the mean of the projection of the residual onto the i-th axis, where i can take the values ​​x, y, and z. This represents the semantic consistency residual obtained by quantization at time t within the observation period. The total amount of deviation accumulated in a specific spatial direction. The unit direction vector along this axis, and N represents the total number of sampling points involved in the calculation in the dynamic residual field, used to normalize the cumulative deviation in the time dimension;

[0045] Based on the cooperative change trend, systematic biases modulated by the motion state of the experimental vehicle are separated from the semantic consistency residuals, and these systematic biases are identified as directional deviation components. Specifically, the projected mean values ​​of each axis are identified. The real-time motion state vector of the experimental vehicle at time t The coordinated change trend, if the projected mean If a continuous and stable offset is observed along the spatial axis, it is defined as the directional offset component and used as the external parameter deformation displacement. ;

[0046] Obtaining external parameters deformation displacement Next, long-term spatial biases caused by physical factors such as sensor bracket deformation need to be removed from the dynamic residual field to extract a pure feature sequence that evolves over time. The specific formula is as follows:

[0047] ;

[0048] in, The residual sequence obtained after stripping represents the instantaneous position deviation vector caused solely by time asynchrony, after removing static installation deviations. This represents the projected component of the predicted spatial deviation caused by the deformation of the external parameters, and quantifies the position offset mapping caused by the sensor installation angle deviation during vehicle movement.

[0049] It should be noted that the extraction of the time offset component includes:

[0050] The observation signal of semantic consistency residuals evolving over time is extracted from the dynamic residual field, and the characteristic motion vector signal of the experimental vehicle is extracted from the motion state data.

[0051] Specifically, these semantic consistency residuals reflect the spatiotemporal differences between visual image data and lidar point cloud data, particularly the deviations caused by hardware triggering delays, sensor installation errors, and other dynamic factors. These semantic consistency residuals, as observation signals, are crucial for understanding temporal offsets in dynamic systems. The dynamic residual field consists of semantic consistency residuals from multiple time points, exhibiting certain evolutionary characteristics along the time axis. Specifically, the residual at each time point is affected by the vehicle's current motion state (e.g., speed, acceleration). Therefore, it is necessary to correlate the residuals at different time points to form a dynamic residual sequence. To compare and register with the semantic consistency residuals, it is necessary to extract the characteristic motion vector signal of the experimental vehicle from the motion state data. This characteristic motion vector signal typically includes physical quantities such as the vehicle's speed, acceleration, and steering angle. This information helps the system understand the vehicle's trajectory and state changes. By processing this motion state data, the characteristic motion vector of the vehicle at each time point can be obtained.

[0052] Calculate the cross-correlation power spectral density between the observed signal and the characteristic motion vector signal in the frequency domain;

[0053] Specifically, the extracted observation signal needs to be compared with the feature motion vector signal through frequency domain analysis. By calculating the cross-correlation power spectral density of the two signals, their similarity and correlation can be evaluated. In particular, the cross-correlation function can quantify the time delay or phase difference between the two signals, which is crucial for understanding the timing deviations caused by hardware delays, clock drift, etc. It should be noted that cross-correlation is a function that measures the degree of time alignment between two signals. By calculating the frequency domain cross-correlation of the signals, the hysteresis phase angle between the observation signal and the motion state signal can be obtained.

[0054] Identify the hysteresis phase angle of the observed signal relative to the characteristic motion vector signal;

[0055] In frequency domain analysis, the key to identifying the hysteresis phase angle is to detect the phase difference between two signals. This difference reflects timing asynchrony caused by clock drift or hardware trigger delay. The hysteresis phase angle is usually expressed as the phase difference between two signals in the frequency domain, which can clearly reveal the time deviation. The hysteresis phase angle can be calculated from the cross-correlation results in the frequency domain, and the specific formula is as follows:

[0056] ;

[0057] in, The lag phase angle, Here, represents the frequency domain representation of the observed signal, specifically the frequency domain expression of the time signal x(t) after its Fourier transform. Let be the frequency domain representation of the characteristic motion vector signal, and let be the frequency domain expression of the time signal y(t) after Fourier transform. for The conjugate of the complex number is typically used to process the phase information of a signal. To return the complex value of the phase angle, i.e. the phase difference between signals, This indicates that in the frequency domain, the signal X(f) and Integration is performed, typically over the entire frequency range or a specified frequency band.

[0058] Based on the consistency of the distribution of the hysteresis phase angle within the preset observation period, the timing offset component is quantified to characterize the timing asynchrony caused by hardware trigger delay or system clock drift.

[0059] Specifically, the consistency of the lag phase angle distribution is crucial for determining the stability of timing offsets. If the lag phase angle remains consistent or exhibits a regular change over multiple observation periods, the timing offset of the system can be considered stable, thus providing a basis for further clock drift calibration. The clock drift increment is calculated using the following formula:

[0060] ;

[0061] in, The clock drift increment represents the amount of time lag between heterogeneous sensors (unit: seconds). The preset sampling frequency of the sensor. It is the radian period coefficient;

[0062] By inversely solving based on directional deviation components and temporal offset components, dynamic drift parameters, including clock drift increments and extrinsic deformation displacements, can be effectively obtained. The calculated dynamic drift parameters provide spatiotemporal precision calibration for the experimental vehicle, ensuring the accuracy and consistency of multi-sensor data. This process, while ensuring high-precision synchronization, helps to improve the reliability and stability of the autonomous driving system.

[0063] The data sampling weights are corrected and the projection mapping matrix is ​​updated based on the clock drift increment and the extrinsic parameter deformation displacement.

[0064] Specifically, the data sampling weights are corrected and the projection mapping matrix is ​​updated, including:

[0065] The sampling step compensation amount is determined based on the clock drift increment, using the following formula:

[0066] ;

[0067] in, The sampling step compensation amount represents the hardware count value or number of sampling pulses (dimensionless) that needs to be offset. For clock drift increment, This indicates the current hardware interrupt frequency of the sensor, that is, the number of interrupts triggered by the hardware per unit time.

[0068] Determine the sampling step compensation amount Then, considering the current hardware interrupt frequency of the sensor. The data sampling weights are smoothly adjusted. By adjusting the weight distribution, the logical sampling time of the visual image frame is brought closer to the triggering time of the lidar, thereby compensating for the clock drift increment.

[0069] The corrected gradient of the projection mapping matrix is ​​calculated based on the extrinsic deformation displacement, using the following formula:

[0070] ;

[0071] in, The corrected gradient of the projection mapping matrix represents the rate of change of the mapping matrix relative to the extrinsic parameters. This represents the current projection mapping matrix;

[0072] Obtaining the corrected gradient Then, within a preset confidence period, the gradient is corrected using a preset weight decay mechanism. The projection is gradually integrated into the current projection mapping matrix. This incremental update avoids abrupt changes in matrix parameters and ensures the smooth evolution of projection correlation during the dynamic calibration process.

[0073] It should be noted that the fusion of smoothing adjustment and gradient correction includes:

[0074] Calculate the variance volatility of the semantically consistent residuals within the dynamic residual field, and determine whether the variance volatility exceeds the preset dynamic stability threshold.

[0075] Specifically, at the current observation time t, a preset time sliding window is obtained. The sequence of modulus values ​​of all semantically consistent residuals within. ,in Calculate the arithmetic mean of the modulus sequence within the current sliding window. The specific formula is as follows:

[0076] ;

[0077] in, The denominator is the sum of the magnitudes of all residual vectors within a preset time window. The total number of samples within the sliding window, based on the average value. Calculate the variance volatility of the current residual distribution. The specific formula is as follows:

[0078] ;

[0079] Variance volatility describes the degree of dispersion of the residuals from the mean, and is used to characterize the intensity of transient noise interference on the sensing system. This represents the sum of squared differences between the magnitude of each semantic consistency residual and the average magnitude of the semantic consistency residual within a preset time window. It reflects the total temporal fluctuation energy of the semantic consistency residual magnitude sequence. This represents the total number of samples included in the calculation within the preset time window.

[0080] If the judgment result is yes, it is determined that the current experimental vehicle is under severe operating conditions (such as bumpy road surface or sudden sensor vibration). At this time, in order to suppress calibration oscillations caused by instantaneous environmental noise, the following adjustment actions are performed: increase the adjustment span L of the time envelope and decrease the attenuation coefficient of the weight attenuation mechanism. The adjusted parameter logic is as follows:

[0081] ;

[0082] ;

[0083] in, The adjusted span. Adjust the span based on the base. The adjusted attenuation coefficient, The preset base attenuation coefficient;

[0084] If the judgment result is negative, the current experimental vehicle is determined to be in a stable operating condition. This is achieved using variance volatility. With dynamic stability threshold The proportional relationship between them is used to perform real-time reduction of the time envelope adjustment span and increase the attenuation coefficient of the weight attenuation mechanism. The adjustment formula is as follows:

[0085] ;

[0086] ;

[0087] in, This represents the adjustment span of the time envelope after real-time reduction under steady operating conditions. This represents the attenuation coefficient of the weight attenuation mechanism after real-time increase under stable operating conditions. This indicates the proportional relationship between the variance volatility and the dynamic stability threshold, since under this operating condition... This ratio is less than or equal to 1, thereby achieving a proportional reduction in the adjustment span and an inverse increase in the attenuation coefficient.

[0088] By eliminating the spatiotemporal drift between visual image data and lidar point cloud data using the corrected data sampling weights and the updated projection mapping matrix, the experimental vehicle dataset after synchronous calibration is obtained.

[0089] Specifically, the original acquired visual image stream is re-timestamped using the corrected data sampling weights, and the sampling step is compensated accordingly. Adjust the shutter trigger logic or read sequence of image frames to achieve sub-millisecond alignment between each visual image frame and its corresponding LiDAR point cloud frame on the time axis; then, use the updated projection mapping matrix. Spatial reprojection is performed on the lidar point cloud data. This reprojection operation is used to correct the spatial offset caused by the deformation of sensor extrinsic parameters. The calculation formula is as follows:

[0090] ;

[0091] in, This represents the calibration coordinates of the target object in the image coordinate system after synchronous calibration. This represents the updated projection mapping matrix, which contains the latest pose compensation parameters fused via the modified gradient G. Represents the original three-dimensional spatial coordinates of points in the lidar point cloud data;

[0092] The visual image frames remapped by timestamps and the LiDAR point cloud frames reprojected by spatial domain operations are encapsulated to obtain the experimental vehicle dataset after synchronous calibration. In this dataset, the pixel distribution of the same semantic target in the visual image and the geometric distribution in the point cloud data completely overlap in the spatiotemporal dimension. This step eliminates the representation bias caused by hardware clock drift and bracket deformation, providing high-confidence basic data for the training and verification of subsequent autonomous driving algorithms.

[0093] The scheme to eliminate representation drift by correcting data sampling weights and updating the projection mapping matrix is ​​based on the core logic of achieving decoupling and reconstruction of heterogeneous sensing data in the spatiotemporal dimensions. Since the experimental vehicle is in dynamic motion, the deviation between the visual image and the LiDAR point cloud is often a nonlinear coupling caused by time asynchrony leading to motion displacement and geometric deviation caused by support deformation. While traditional linear interpolation can approximate the point cloud time using adjacent image frames, it cannot compensate for hardware-level trigger jitter and is powerless against spatial misalignment caused by external parameter deformation. Static matrix correction alone cannot address the dynamic pointing deviation caused by minute deformation of the support under long-term operating conditions. This scheme corrects data sampling weights based on clock drift increments and dynamically updates the projection mapping matrix by combining external parameter deformation displacement. This allows for simultaneous intervention in the temporal and spatial domains from the underlying source of data generation, ensuring the convergence and stability of calibration parameters during dynamic fluctuations. This two-dimensional closed-loop correction operation, compared to open-loop static compensation, can significantly cut off the path of deviation accumulation, enabling the sensing data stream to achieve high-precision self-healing alignment without the need for an external high-precision benchmark.

[0094] For example, consider a set of coherent dynamic data: Suppose the experimental vehicle... The clock drift increment is obtained by inverse calculation of the dynamic residual field while the device is moving at a constant speed. External parameters, deformation and displacement (such as pitch angle direction). In this state, firstly based on the clock drift increment Combined with hardware interrupt frequency The sampling step compensation amount is calculated. One pulse; then, the sampling step compensation amount is used to smooth the data sampling weight of the visual image based on the time envelope, so that the logical sampling time of the image frame is closer to the triggering time of the lidar, thereby offsetting the clock drift increment. Delay in The 0.2m displacement deviation generated at the speed; synchronously, based on the deformation displacement of the external parameters. Calculate the corrected gradient G of the projection mapping matrix, and gradually fuse it into the current projection mapping matrix within a preset confidence period using a weight decay mechanism. In this process, the updated projection mapping matrix is ​​obtained. This is used to correct the approximately 0.3m projection mapping offset caused by the support deformation; finally, timestamp remapping is performed using the corrected weights, and the updated weights are used... Perform spatial reprojection operation This ensures that the pixel distribution of the same semantic target in the visual image and its geometric distribution in the point cloud data completely overlap in the spatiotemporal dimension, eliminating cumulative representation bias.

[0095] Figure 2This is a schematic diagram of the hierarchical calibration method for data synchronization and drift calibration of experimental vehicles according to the present invention.

[0096] To further improve the accuracy of the experimental vehicle dataset under extreme dynamic conditions, the method provided in this invention also includes a process of in-depth analysis and hierarchical calibration of the dynamic residual field. The specific implementation process is as follows:

[0097] By analyzing the dynamic residual field, we can identify the dominant mode that characterizes systematic drift and the secondary mode that characterizes random perturbation.

[0098] Specifically, spectral analysis and principal component extraction are performed on the dynamic residual field to identify the dominant mode that characterizes the systematic drift and the secondary mode that characterizes the random disturbance. The dominant mode corresponds to the high-energy, low-frequency component in the residual field, which is usually caused by the linear drift of the hardware clock or the quasi-static deformation of the support. The secondary mode corresponds to the high-frequency, random fluctuation component, which is usually caused by the instantaneous vibration of the road surface or the electromagnetic interference of the sensor.

[0099] Based on the identification results of the dominant and secondary modes, the calculation process of dynamic drift parameters is calibrated in layers.

[0100] Among them, the hierarchical calibration includes, for the dominant mode, calling the global calibration strategy to update the clock drift increment and extrinsic parameter deformation displacement; for the secondary mode, calling the local calibration strategy to optimize the semantic consistency residual.

[0101] Specifically, for the dominant mode, the reverse calculation process described above is executed to update the clock drift increment and extrinsic parameter deformation displacement, eliminating long-term deviations from the system bottom layer; for the secondary mode, real-time optimization of semantic consistency residuals is performed to eliminate transient disturbances.

[0102] It should be noted that semantic consistency residual optimization includes:

[0103] Based on semantically consistent residuals, and according to their spatial distribution and time-series data, the propagation path and attenuation characteristics of their perturbations between consecutive frames are extracted. The specific formula is as follows:

[0104] ;

[0105] ;

[0106] in, The disturbance propagation path is represented by the spatial displacement vector of the secondary modes of random disturbances across consecutive frames. In the secondary patterns representing random perturbations, the vector difference between adjacent sampling times corresponds to the semantic consistency residuals. Indicates the time span between adjacent sampling times. For attenuation characteristics, The damping coefficient;

[0107] Based on the propagation path and attenuation characteristics, the residual evolution state of the secondary mode in the current frame is predicted using the following formula:

[0108] ;

[0109] in, Indicates the residual evolution state. This represents the semantic consistency residual corresponding to the secondary pattern representing random perturbation at the previous time step;

[0110] Determine whether semantic consistency residuals exhibit regular spatiotemporal correlations;

[0111] If the judgment result is yes, it indicates that the current semantic consistency residual is affected by regular secondary pattern perturbations. In this case, the predicted residual evolution state is used. Vector compensation is applied to the current semantic consistency residuals to obtain the optimized residual vector, calculated using the following formula:

[0112] ;

[0113] in, The optimized residual vector, This represents the original semantic consistency residual observed at the current sampling time.

[0114] If the judgment result is negative, the current semantic consistency residual is identified as random noise to prevent the noise sample from contaminating the calculation accuracy of the dynamic drift parameter, and an isolation operation is performed. Specifically, in the cross-modal semantic association matrix, the system assigns the weight of each set of associated coordinate rows that are identified as random noise at the current sampling time to zero, or removes the set of coordinate pairs from the current cost function accumulation through a Boolean mask. Through this isolation operation, random noise no longer participates in the subsequent process of "reverse calculation based on the calibration driving force generated by the dynamic residual field", ensuring the robustness of the dynamic drift parameter calculation.

[0115] By performing hierarchical calibration and residual optimization on the dynamic residual field, the decoupling processing of heterogeneous interference in complex environments is achieved. As the experimental vehicle is in extreme dynamic conditions, the deviation faced by its sensing system is often the superposition of long-term systematic drift and instantaneous random disturbance. Traditional calibration methods often treat all residuals as homogeneous data and perform weighted processing. When faced with high-frequency vibration or electromagnetic interference, this can easily lead to oscillations in the solution process and make it impossible to distinguish between underlying physical deformation and surface environmental noise.

[0116] The underlying logic of this scheme is to leverage the differences in spectral characteristics and energy distribution to deeply peel away the residual field. The dominant mode, with its low-frequency and high-energy characteristics, can stably map the linear clock drift and quasi-static deformation of the support at the hardware level. Through a global calibration strategy, long-term compensation can be achieved from the bottom layer. The secondary mode, by extracting the propagation path and attenuation characteristics of disturbances, constructs a dynamic prediction framework with spatiotemporal correlation. Its advantage is that it can identify regular instantaneous disturbances such as periodic swaying of the suspension and perform vector compensation using the predicted evolution state, avoiding erroneous triggering of global parameter updates.

[0117] For example, in the actual processing logic, taking a set of continuous observation data affected by road bumps as an example: the system first identifies the dominant mode component and the secondary mode component through spectrum analysis; let the original semantic consistency residual observed at the current sampling time be... If the analysis reveals that the original semantic consistency residual has a clear propagation path W between consecutive frames and conforms to the damping coefficient... Modulation attenuation characteristics If the secondary mode component is determined to have a regular spatiotemporal correlation, then based on the residual from the previous time step... The residual evolution state of the current frame is predicted. By performing vector compensation operations It can accurately extract the optimized residual vector to maintain the high-frequency stability of the sensing data. If the residual does not have the above correlation, its weight is set to zero through the Boolean mask operation of the cross-modal semantic correlation matrix, which completely isolates the interference of random noise on the dynamic drift parameter calculation. This hierarchical processing method replaces the conventional simple low-pass filtering technology, which significantly enhances the anti-interference ability of the system under non-steady-state conditions while retaining the system calibration sensitivity.

[0118] Figure 3 This is a schematic diagram illustrating the structural confidence assessment of the data synchronization and drift calibration method for experimental vehicles according to the present invention.

[0119] Building upon the aforementioned basic calibration process, the method provided by this invention further includes a multi-hypothesis dynamic calibration enhancement mechanism based on structural confidence assessment, aiming to improve the robustness of the system under ambiguous operating conditions. The specific implementation process is as follows:

[0120] Based on the consistency between the spatiotemporal distribution entropy value and the gradient of the dynamic residual field, its structural confidence is calculated using the following formula:

[0121] ;

[0122] in, The structural confidence score is used to quantify the enrichment of effective calibration information in the dynamic residual field. This represents the spatiotemporal distribution entropy value of the dynamic residual field. The gradient consistency of the dynamic residual field reflects the stability of the residual vector in spatial orientation as the experimental vehicle moves. It is obtained by calculating the mean cosine similarity of the residual gradient direction vectors between adjacent sampling times. and This represents the preset weighting coefficients used to balance the contribution ratio of spatiotemporal distribution entropy and gradient consistency to confidence.

[0123] Determine whether the structural confidence level is lower than a preset threshold;

[0124] If the judgment result is yes, it is determined that the current dynamic residual field is ambiguous. Multiple candidate drift parameter hypotheses compatible with the local features of the dynamic residual field are generated in parallel. The smoothness of the historical motion trajectory of the experimental vehicle is introduced as a constraint condition. The optimal dynamic drift parameter is selected from multiple candidate drift parameter hypotheses as the dynamic drift parameter.

[0125] Specifically, the original data is pre-calibrated using each candidate drift parameter assumption, and the observation trajectory of the experimental vehicle within a preset period is reconstructed. Subsequently, the reconstructed trajectory is calculated using the smoothness evaluation operator S. The calculation logic of the smoothness evaluation operator S is as follows:

[0126] ;

[0127] in, This represents the sum of the spurious acceleration vector magnitudes among the sampling points in the reconstructed trajectory. To reconstruct the spurious acceleration vector of the sampling points in the trajectory, the hypothesis that maximizes the smoothness evaluation operator S is selected from multiple candidate drift parameter hypotheses and determined as the optimal dynamic drift parameter.

[0128] If the judgment result is negative, a single solution is performed based on the dynamic residual field to directly obtain the dynamic drift parameters. The specific implementation is as follows:

[0129] A global reprojection error loss function J is constructed based on the dynamic residual field. The loss function J is used to quantify the sum of residuals after correction by the drift parameters to be solved at all sampling times. Its goal is to find a set of optimal dynamic drift parameters that minimize the loss function. The calculation formula is as follows:

[0130] ;

[0131] in, Indicated by clock drift increment With external parameters deformation and displacement Let be the global cost function of the variable to be optimized. This represents the sum of the squared L2 norms of the residual sequences at all sampling times t within the observation period of the dynamic residual field, where N is the total number of samples. The loss function J is solved using the Gauss-Newton iterative method, and the loss function J is calculated relative to the clock drift increment. and extrinsic deformation displacement The partial derivatives are used to determine the descent direction of the loss function in the current parameter space. The extreme point that minimizes the loss function J is found along the gradient descent direction. When the iteration converges or reaches the preset number of iterations, the value of the optimization variable at this time is extracted, and the dynamic drift parameter is obtained directly.

[0132] By introducing a multi-hypothesis dynamic calibration enhancement mechanism based on structural confidence assessment, highly reliable positioning and alignment of experimental vehicles under complex and ambiguous conditions were achieved. The design logic of this scheme is to address the technical pain point that a single solution is prone to getting trapped in local optima when the dynamic residual field is feature-scarce or subject to severe environmental interference. Compared with single-point solution technology that relies solely on data-driven approaches, this scheme can quantify the quality of current calibration information in real time by integrating the structured assessment of spatiotemporal distribution entropy and gradient consistency. When the structural confidence is low, the system no longer forcibly outputs a single possible result, but instead covers the potential parameter space through a multi-hypothesis generation strategy and uses the inherent smoothness of the historical motion trajectory of vehicle dynamics as a physical constraint for secondary screening. The advantage of this operation is that it transforms pure geometric mathematical calculation into an optimization process that conforms to the laws of physical motion, effectively replacing the traditional least squares method that is prone to non-physical jumps, and ensuring the authenticity of drift parameters under extreme environments.

[0133] Example 2:

[0134] In one embodiment of the present invention, which differs from the previous embodiment, the electronic device includes one or more processors and a memory.

[0135] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0136] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0137] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). In addition, depending on the specific application, the electronic device may include any other suitable components.

[0138] Example 3:

[0139] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps described in the "Exemplary Methods" section above according to the various embodiments of this application.

[0140] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not restrict the application from being implemented using the specific details described above.

[0142] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0143] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0144] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0145] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A data synchronization and drift calibration method for experimental vehicles, characterized in that, include: The system acquires visual image data, lidar point cloud data, and motion state data of the experimental vehicle in the current time window. Based on a preset data sampling weight, it resamples the visual image data to obtain synchronized image data that is aligned with the timestamp of the point cloud data. The system then extracts the semantic center coordinates of the predetermined target object in the synchronized image data and the geometric center coordinates in the lidar point cloud data. Based on a preset projection mapping matrix, the semantic center coordinates are projected onto the three-dimensional coordinate system where the lidar point cloud data is located, to obtain the three-dimensional coordinates of the target object inferred from the synchronized image data; Based on the semantic tags of the target object, a cross-modal semantic association matrix of the semantic center coordinates and geometric center coordinates of the target object is established. Based on the spatial vector displacement between each set of associated coordinates in the cross-modal semantic association matrix, the semantic consistency residual is quantified. Based on the semantic consistency residuals at each time point of the current time window, a dynamic residual field is constructed, which is formed by the evolution of the semantic consistency residuals over time. Each semantic consistency residual is associated with the motion state data at the corresponding time point. Based on the spatial distribution directionality and temporal evolution phase characteristics of the semantically consistent residuals in the dynamic residual field, the directional deviation component characterizing the spatial installation offset of the sensor and the temporal offset component characterizing the time asynchrony of the sensor are extracted respectively. Based on the directional deviation component and the temporal offset component, the dynamic drift parameters including clock drift increment and external parameter deformation displacement are obtained. The data sampling weights are corrected and the projection mapping matrix is ​​updated based on the clock drift increment and the extrinsic deformation displacement. The spatial and temporal drift between the visual image data and the lidar point cloud data is eliminated by using the corrected data sampling weights and the updated projection mapping matrix to obtain the synchronously calibrated experimental vehicle dataset.

2. The data synchronization and drift calibration method for experimental vehicles according to claim 1, characterized in that, The method further includes: By analyzing the dynamic residual field, the dominant mode representing systematic drift and the secondary mode representing random perturbation are identified. Based on the identification results of the dominant and secondary modes, the calculation process of the dynamic drift parameters is calibrated in layers. The hierarchical calibration includes, for the dominant mode, calling a global calibration strategy to update the clock drift increment and extrinsic deformation displacement; and for the secondary mode, calling a local calibration strategy to optimize the semantic consistency residual.

3. The data synchronization and drift calibration method for experimental vehicles according to claim 2, characterized in that, The optimization of semantic consistency residuals includes: Based on the semantic consistency residual, the propagation path and attenuation characteristics of its disturbance between consecutive frames are extracted according to its spatial distribution and time series data. Based on the propagation path and attenuation characteristics, predict the residual evolution state of the secondary mode in the current frame, and determine whether the semantic consistency residuals exhibit regular spatiotemporal correlation. If the judgment result is yes, the semantic consistency residual is compensated using the predicted residual evolution state; If the judgment result is negative, it is determined to be random noise and isolated from the current calculation of the cross-modal semantic association matrix.

4. The data synchronization and drift calibration method for experimental vehicles according to claim 1, characterized in that, The method further includes: Calculate the structural confidence level based on the consistency between the spatiotemporal distribution entropy value and the gradient of the dynamic residual field; Determine whether the confidence level of the structure is lower than a preset threshold; If the judgment result is yes, it is determined that the current dynamic residual field is ambiguous, and multiple candidate drift parameter hypotheses compatible with the local features of the dynamic residual field are generated in parallel. The smoothness of the historical motion trajectory of the experimental vehicle is introduced as a constraint condition, and the optimal dynamic drift parameter is selected from the multiple candidate drift parameter hypotheses as the dynamic drift parameter. If the judgment result is negative, a single solution is performed based on the dynamic residual field to directly obtain the dynamic drift parameters.

5. The data synchronization and drift calibration method for experimental vehicles according to claim 1, characterized in that, The extraction of the directional deviation component includes: The dynamic residual field is decomposed into sub-residual sets corresponding to different motion modes using the motion state data, wherein the motion modes include at least uniform linear motion mode and nonlinear directional motion mode; Extract the mean projection of each of the sub-residual sets along each axis of the three-dimensional coordinate system, and calculate the cooperative variation trend of the mean projection with the velocity vector change in the motion state data; Based on the cooperative change trend, the systematic deviation modulated by the motion state of the experimental vehicle is separated from the semantic consistency residual and identified as the directional deviation component.

6. The data synchronization and drift calibration method for experimental vehicles according to claim 1, characterized in that, The extraction of the time offset component includes: The observation signal of the semantic consistency residual evolving over time is extracted from the dynamic residual field, and the characteristic motion vector signal of the experimental vehicle is extracted from the motion state data; Calculate the cross-correlation power spectral density between the observed signal and the characteristic motion vector signal in the frequency domain, and identify the hysteresis phase angle of the observed signal relative to the characteristic motion vector signal; Based on the consistency of the distribution of the hysteresis phase angle within a preset observation period, the timing offset component is quantified to characterize the timing asynchrony caused by hardware trigger delay or system clock drift.

7. The data synchronization and drift calibration method for experimental vehicles according to claim 1, characterized in that, The step of correcting the data sampling weights and updating the projection mapping matrix includes: The sampling step compensation amount is determined based on the clock drift increment, and the data sampling weight is smoothly adjusted based on the time envelope in combination with the current hardware interrupt frequency of the sensor. Based on the extrinsic deformation displacement, the corrected gradient of the projection mapping matrix is ​​calculated, and within a preset confidence period, the corrected gradient is gradually fused into the current projection mapping matrix using a preset weight decay mechanism.

8. The data synchronization and drift calibration method for experimental vehicles according to claim 7, characterized in that, The fusion of smoothing adjustment and gradient correction includes: Calculate the variance volatility of the semantically consistent residuals within the dynamic residual field, and determine whether the variance volatility exceeds a preset dynamic stability threshold. If the judgment result is yes, increase the adjustment span of the time envelope and reduce the attenuation coefficient of the weight attenuation mechanism to extend the parameter fusion period; If the judgment result is negative, the adjustment span of the time envelope is reduced in real time and the decay coefficient of the weight decay mechanism is increased by utilizing the proportional relationship between the variance volatility and the dynamic stability threshold, thereby accelerating parameter convergence.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 8.

10. A computer storage medium storing computer-executable instructions thereon, characterized in that: When the computer-executable instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images