Multi-sensor data time synchronization error compensation method and device

Abstract

The application is suitable for the technical field of data processing, and particularly relates to a sensor data time synchronization error compensation method and device. The method comprises the following steps: acquiring multi-sensor original data of a current frame and extracting multi-sensor feature data; obtaining an error estimation vector based on an error estimation network; performing error compensation on the multi-sensor original data to obtain multi-sensor compensation data; constructing a factor graph and optimizing and solving the factor graph to obtain a residual set; calculating a comprehensive quality score according to the residual set; determining that a trigger condition is met according to the comprehensive quality score and a comprehensive quality score of the previous M frames, and triggering fine tuning; and fine tuning model parameters of the error estimation network based on the multi-sensor original data, the multi-sensor feature data and the comprehensive quality score in a time window to obtain a fine-tuned error estimation network. The method realizes more accurate compensation of the time sequence error and data distortion between multi-sensors and continuous adaptive optimization.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a method and device for compensating time synchronization errors of multi-sensor data. Background Technology

[0002] Time synchronization of data from multiple sensors (such as cameras, IMUs, LiDAR, and millimeter-wave radar) is a crucial aspect of robotics, autonomous driving, and the Internet of Things. However, due to factors such as different clock references, sampling frequencies, and clock drift caused by temperature fluctuations among the sensors, it is often difficult to achieve strict simultaneous alignment of sensor data. Therefore, researching and implementing high-precision and robust time error modeling and compensation mechanisms is a key foundation for ensuring the consistency and fusion accuracy of multi-source information.

[0003] In existing technologies, time synchronization errors of multi-sensor data are compensated through pre-calibration or physical models (such as assuming uniform velocity or linear interpolation). However, in real-world applications, the system's motion state is often complex and variable, and the aforementioned simplified assumptions cannot accurately describe the true state. Furthermore, the data processing within the sensors may be affected by environmental conditions or system load, leading to complex, nonlinear, and time-varying timing errors.

[0004] In summary, in the process of compensating for time synchronization errors in multi-sensor data, there is a problem that simple rules cannot accurately compensate for the timing errors and data distortions that change over time between different sensors. Summary of the Invention

[0005] This application provides a method and device for compensating time synchronization errors of multi-sensor data, which can solve the problem that in the process of compensating time synchronization errors of multi-sensor data, it is difficult to accurately compensate for the timing errors and data distortions that change over time between different sensors due to simple rules.

[0006] In a first aspect, embodiments of this application provide a method for compensating for time synchronization errors in multi-sensor data, including:

[0007] The system acquires the raw multi-sensor data of the current frame and performs feature extraction on the multi-sensor data to obtain multi-sensor feature data. The raw multi-sensor data includes image data, LiDAR point cloud data, and IMU data, and the multi-sensor feature data includes image features, point cloud features, and IMU temporal tensors.

[0008] An error estimation vector is obtained based on multi-sensor feature data and an error estimation network; wherein, the error estimation network includes an input encoding layer, a cross-modal feature fusion layer and an error regression output layer, and the error estimation vector includes relative time delay, distortion parameters, scan angular velocity residual, and confidence level;

[0009] Based on the error estimation vector, error compensation is performed on the original multi-sensor data to obtain multi-sensor compensated data; wherein, the multi-sensor compensated data includes compensated image data, LiDAR point cloud data, and IMU data;

[0010] Based on the multi-sensor compensation data, a factor map is constructed, and the factor map is optimized to obtain a residual set; based on the residual set, a comprehensive quality score is calculated; wherein, the residual set includes image residuals, LiDAR residuals, and IMU residuals;

[0011] Based on the overall quality score and the overall quality score of the previous M frames, it is determined that the triggering condition is met, triggering fine-tuning, and acquiring the multi-sensor raw data, multi-sensor feature data, and overall quality score within the time window; wherein, the triggering condition includes that both the overall quality score and the overall quality score of the previous M frames are greater than the triggering threshold, and the time window includes a period of time before the first frame in the previous M frames and the time from the first frame in the previous M frames to the current frame;

[0012] Based on the multi-sensor raw data, multi-sensor feature data, and comprehensive quality score within the time window, the model parameters of the error estimation network are fine-tuned to obtain the fine-tuned error estimation network.

[0013] The technical solutions described in this application embodiment have at least the following technical effects:

[0014] The multi-sensor data time synchronization error compensation method provided in this application first acquires the original multi-sensor data of the current frame and extracts features from the multi-sensor data to obtain multi-sensor feature data. Then, based on the multi-sensor feature data and an error estimation network, an error estimation vector is obtained. Next, based on the error estimation vector, error compensation is performed on the original multi-sensor data to obtain multi-sensor compensated data. Then, based on the multi-sensor compensated data, a factor graph is constructed and optimized to obtain a residual set. Then, based on the residual set, a comprehensive quality score is calculated. Then, based on the comprehensive quality score and the comprehensive quality scores of the previous M frames, a trigger condition is determined to trigger fine-tuning, and the original multi-sensor data, multi-sensor feature data, and comprehensive quality score within the time window are acquired. Finally, based on the original multi-sensor data, multi-sensor feature data, and comprehensive quality score within the time window, the model parameters of the error estimation network are fine-tuned to obtain a fine-tuned error estimation network. This method can improve the alignment accuracy between data from different sensors and reduce information inconsistencies caused by time misalignment and acquisition differences. This method achieves more accurate compensation for complex, time-varying, and nonlinear temporal errors and data distortions among multiple sensors, and continuously adaptively optimizes during system operation, thereby improving the alignment accuracy of multi-sensor data and the stability and reliability of fused sensing, localization, and mapping results. Through continuous evaluation and online adjustment, this method enhances the robustness and sustainable operation of the system over long-term operation.

[0015] In a second aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the embodiments of the first aspect.

[0016] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a multi-sensor data time synchronization error compensation method provided in an embodiment of this application. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] In related technologies, such as autonomous driving, robotic perception, and intelligent devices, systems typically require the simultaneous use of multiple types of sensors, including vision devices, laser rangefinders, and inertial measurement units. These sensors operate through different hardware interfaces, drivers, and data processing links, and their data acquisition and output processes often exhibit unavoidable time differences and processing delays. Furthermore, the working mechanisms of different sensors themselves differ; for example, some devices acquire data gradually over a period of time rather than instantaneously, a characteristic that can easily introduce data distortion or temporal shifts during device movement.

[0023] Existing technologies compensate for these errors through pre-calibration or methods based on simplified physical assumptions, such as assuming the device maintains a uniform speed for a short period or estimating state changes between different time points through linear interpolation. However, in real-world applications, system motion states are often complex and variable, including sudden acceleration, turning, and vibration, making it difficult for the aforementioned simplified assumptions to accurately describe the true state. Furthermore, the data processing within the sensor may be affected by environmental conditions or system load; factors such as variations in exposure time, internal processing delays, or communication buffers can all cause timing errors to exhibit complex, nonlinear, and time-varying characteristics.

[0024] Because these errors are dynamic and difficult to model accurately, traditional methods based on fixed models or offline calibration struggle to maintain accuracy over the long term. Once errors accumulate, spatial or temporal inconsistencies arise between data from different sensors, affecting the system's ability to perceive the environment and further reducing the stability and reliability of subsequent tasks such as localization, mapping, or target recognition.

[0025] To address the aforementioned issues, this application provides a method and apparatus for compensating time synchronization errors in multi-sensor data. The method first acquires raw multi-sensor data for the current frame and extracts features from the multi-sensor data to obtain multi-sensor feature data. Then, based on the multi-sensor feature data and an error estimation network, an error estimation vector is obtained. Next, based on the error estimation vector, error compensation is performed on the raw multi-sensor data to obtain compensated multi-sensor data. Then, based on the compensated multi-sensor data, a factor graph is constructed, and the factor graph is optimized to obtain a residual set. Based on the residual set, a comprehensive quality score is calculated. Then, based on the comprehensive quality score and the comprehensive quality scores of the previous M frames, a trigger condition is determined, triggering fine-tuning. The raw multi-sensor data, multi-sensor feature data, and comprehensive quality score within the time window are acquired. Finally, based on the raw multi-sensor data, multi-sensor feature data, and comprehensive quality score within the time window, the model parameters of the error estimation network are fine-tuned to obtain a fine-tuned error estimation network. This method can improve the alignment accuracy between data from different sensors and reduce information inconsistencies caused by time misalignment and acquisition differences. This method achieves more accurate compensation for complex, time-varying, and nonlinear temporal errors and data distortions among multiple sensors, and continuously adaptively optimizes during system operation, thereby improving the alignment accuracy of multi-sensor data and the stability and reliability of fused sensing, localization, and mapping results. Through continuous evaluation and online adjustment, this method enhances the robustness and sustainable operation of the system over long-term operation.

[0026] The multi-sensor data time synchronization error compensation method provided in this application embodiment can be applied to a multi-sensor data time synchronization error compensation device. In this case, the multi-sensor data time synchronization error compensation device is the execution subject of the multi-sensor data time synchronization error compensation method provided in this application embodiment. This application embodiment does not impose any restrictions on the specific type of multi-sensor data time synchronization error compensation device.

[0027] For example, a multi-sensor data time synchronization error compensation device may include an image acquisition device, a point cloud acquisition device, an IMU acquisition device, and a control device communicatively connected to the image acquisition device, point cloud acquisition device, and IMU acquisition device. The image acquisition device is a device capable of acquiring image data, and may be an industrial camera, image acquisition card, etc.; the point cloud acquisition device is a device capable of acquiring LiDAR point cloud data, and may be a mechanical LiDAR, semi-solid-state LiDAR, etc.; the IMU acquisition device is a device capable of acquiring IMU data, and may be an inertial measurement unit, which includes an accelerometer and a gyroscope; the control device is a device capable of controlling the image acquisition device, point cloud acquisition device, and IMU acquisition device, and performing data processing, and may be a laptop computer, ultra-mobile personal computer (UMPC), netbook, desktop computer, computer, laptop computer, etc.

[0028] To better understand the multi-sensor data time synchronization error compensation method provided in the embodiments of this application, the specific implementation process of the multi-sensor data time synchronization error compensation method provided in the embodiments of this application will be described by way of example below.

[0029] Figure 1 This paper presents a schematic flowchart of a multi-sensor data time synchronization error compensation method provided in an embodiment of this application. The multi-sensor data time synchronization error compensation method includes:

[0030] S100: Acquire the raw multi-sensor data of the current frame and extract features from the multi-sensor data to obtain multi-sensor feature data. The raw multi-sensor data includes image data, LiDAR point cloud data, and IMU data, while the multi-sensor feature data includes image features, point cloud features, and IMU temporal tensors.

[0031] Image data is understood to be two-dimensional image information acquired by a camera or visual acquisition device. It consists of pixels and is used to reflect visual content such as color, texture, brightness, edges, and target appearance in a scene.

[0032] LiDAR point cloud data is a collection of spatial points acquired by a lidar system. LiDAR emits lasers and receives reflected signals to obtain distance information from surrounding objects to the sensor, thus forming a large number of discrete spatial points. These points together constitute the three-dimensional geometry of the target scene.

[0033] IMU data is motion-related data collected by an inertial measurement unit, which includes accelerometers and gyroscopes. It measures the acceleration and angular velocity of the equipment during motion. IMU data can reflect changes in the equipment's motion, such as whether it is accelerating, rotating, or undergoing attitude changes.

[0034] For example, a Global Navigation Satellite System (GNSS) receiver can be used as the absolute time reference. The GNSS module not only outputs position information, but also outputs high-precision pulse-of-seconds (PPS) signals and GPRMC data frames containing UTC (Coordinated Universal Time) time information.

[0035] The PPS signal can be simultaneously distributed to all sensors (such as cameras, LiDAR, and inertial measurement units) and the industrial control computer that supports hardware synchronization via hardware circuitry. The rising edge of the PPS signal has extremely high steepness and nanosecond-level accuracy.

[0036] For sensors connected via Ethernet (such as industrial cameras and smart cameras), the PTP protocol can be used. With the support of network hardware (switches, network cards), the PTP protocol can synchronize the clocks of all nodes in the network with the master clock (which can be the industrial control computer clock, which has been tamed by PPS) to sub-microsecond accuracy by exchanging synchronization messages containing timestamps.

[0037] For lidar that receives PPS signals, its internal clock is hard-calibrated to the whole second on the rising edge of each second pulse. For cameras that support PTP, their internal clock continuously follows the master clock, ensuring high accuracy of data packet timestamps.

[0038] Through the above process, a high-precision hardware time base can be established. When the raw data from multiple sensors in the current frame leaves the sensor or arrives at the acquisition network card, it is stamped with a hardware timestamp relative to UTC time, with an error in the microsecond range or even lower.

[0039] Raw data from multiple sensors over a period of time can be cached to support subsequent error estimation and model fine-tuning. A thread-safe, fixed-size circular buffer can be maintained for each sensor thread. The buffer stores a structure that can include the sensor ID, a hardware timestamp (a uint64_t in nanoseconds), and a pointer to the raw data (or the data itself). The size N of the circular buffer can be set based on the maximum expected latency and algorithm requirements. For example, if the system needs to handle a cumulative latency of up to 500ms and the total sensor frame rate is high, N can be set to accommodate data from the most recent 2 seconds.

[0040] Due to the uncertainty of network and operating system scheduling, the arrival order of data packets may not be consistent with the actual order in which they occur. When pushing data into the circular buffer, the system performs approximate sorting based on hardware timestamps or maintains a separate index to ensure that the data is strictly arranged in chronological order when reading window data.

[0041] The raw data from multiple sensors is high in dimensionality and has a large amount of information redundancy. Directly feeding it into an error estimation network would result in excessive computation and make training difficult. Therefore, feature extraction can be performed to transform the raw data into a compact feature representation that includes motion and temporal information.

[0042] Image feature extraction: Two or three consecutive frames of images can be extracted from a circular buffer (e.g., ... , , , (Representing the current frame), using GPU-accelerated dense optical flow algorithms (such as lightweight versions of Farneback, FlowNet, or RAFT) to calculate the pixel motion field between adjacent frames, resulting in an optical flow map for several consecutive frames. It is a two-dimensional vector field, representing each pixel from Time's up Displacement at any given moment. Stacking the optical flow maps of several consecutive frames forms a multi-channel optical flow tensor, which is the image feature (e.g., 3 frames can generate 2 optical flow maps, with a total of 4 channels: x-direction displacement and y-direction displacement). Alternatively, the feature point method can be used to extract corner points or ORB features from the image, and then track them using the LK optical flow method to obtain the motion trajectories of a series of discrete feature points. Image features can capture motion information in image sequences, providing a basis for estimating inter-frame camera motion and rolling shutter distortion.

[0043] Point cloud feature extraction: For LiDAR, the LiDAR point cloud of the current frame can be projected into a distance map according to its horizontal and vertical angular resolution. The distance map is a two-dimensional image, where pixel values ​​represent the nearest distance at that angle. The distance map preserves the neighborhood relationships of the point cloud, facilitating subsequent error estimation network processing. When each point is projected onto the distance map, in addition to recording the distance value, its normalized acquisition time (e.g., the relative time of the point within the current frame's scanning cycle, 0 indicating the start and 1 indicating the end) can be used as another channel to obtain a two-channel distance-time map, which is the point cloud feature. Point cloud features can capture motion distortion within the point cloud frame.

[0044] IMU temporal tensor extraction: Based on the timestamp of the current LiDAR frame, extract all IMU measurements (three-axis acceleration) within a small time window (e.g., 50ms) preceding that frame. , , and triaxial angular velocity , , Since IMU data is uniformly sampled, a fixed-length IMU temporal tensor can be directly formed, with a shape of [time step, 6], where 6 corresponds to the triaxial acceleration and triaxial angular velocity. If there are insufficient data points within the time window, interpolation or zero-padding is performed; if there are too many, downsampling is performed. The IMU temporal tensor can provide high-frequency, accurate motion information, serving as a reference skeleton for continuous motion.

[0045] This step provides high-quality input to the subsequent error estimation network. The multi-sensor feature data not only contains the raw motion information, but also provides an initial time alignment basis for subsequent error estimation through hardware synchronization.

[0046] S200, based on multi-sensor feature data and an error estimation network, obtains an error estimation vector. The error estimation network includes an input encoding layer, a cross-modal feature fusion layer, and an error regression output layer. The error estimation vector includes relative time delay, distortion parameters, scan angular velocity residuals, and confidence level.

[0047] It can be understood that the input encoding layer is used to extract high-level representation information from the features of each sensor; the cross-modal feature fusion layer is used to establish the correlation between different sensor features and strengthen the joint expression of information such as time deviation, motion inconsistency and structural mismatch; and the error regression output layer is used to output the error estimation vector.

[0048] For example, multi-sensor feature data can be used to construct input data. ,in, The shape is ( , , Optical current tensor This represents the number of optical flow tensor channels. , Indicates the width and height of the image; The shape is ( , , Distance-time graph of ) This indicates the number of channels in the distance-time plot (e.g., 2). , This represents the width and height of the distance-time plot. The shape is ( The IMU temporal tensor of ,6) This indicates the time step, and 6 corresponds to the triaxial acceleration and triaxial angular velocity.

[0049] The input data X can be input into the error estimation network. The error estimation network performs one forward inference and outputs the relative time delay (X). Scalar, representing the time offset of the camera relative to LiDAR in the current frame), distortion parameters ( Two-dimensional vector ( , ), representing the average speed of the rolling shutter camera's inter-row movement (used for distortion compensation) and the scan angular velocity residual ( , scalar, representing the residual of the lidar angular velocity (used for motion compensation) , confidence level (c, scalar, range [0,1], representing the network's confidence in the estimation results).

[0050] Training process of the error estimation network: The sample dataset can be collected through a multi-sensor acquisition platform equipped with cameras, LiDAR, and inertial measurement units. The collected data includes rich dynamic scene sample data, such as rapid acceleration, rapid deceleration, steering, continuous attitude changes, and traversing bumpy roads, to improve the network's adaptability to complex motion states and dynamic scenes. High-precision hardware synchronization devices and timing modules can be used to uniformly synchronize the data from each sensor, thereby obtaining raw data that closely approximates real-world synchronization.

[0051] Artificially set errors can be applied to a portion of the sensor data in the sample dataset to generate labeled sample data. For example, a preset overall time offset can be applied to image data to simulate the time delay of the camera relative to the LiDAR; deformation related to line-by-line exposure can be applied to image data by combining known motion trajectories and imaging models to simulate distortion during the imaging process; and perturbations can be introduced into the motion parameters during the LiDAR scanning process. Through this process, the original sample data can be transformed into sample data with known error labels, thereby providing supervision signals for network training.

[0052] The features of each sensor data in the sample dataset can be extracted using the method in step S100. The features of each sensor data in the sample dataset and their corresponding labels are then divided into a training set and a validation set. Parameters are updated on the training set, and the changes in the loss function and the predictive performance of each error output term are monitored on the validation set.

[0053] In each training round, a batch of training samples is selected from the training set and input into the error estimation network. Forward propagation is performed to obtain the error estimation result. The loss value between the error estimation result and the corresponding label is calculated using the loss function. Backpropagation is performed based on the loss value to calculate the gradient of the network parameters. The network parameters are updated using an optimizer (such as the AdamW optimizer) based on the network parameter gradient. This process can be repeated in multiple training batches and multiple training rounds to continuously optimize the network parameters in the direction of reducing loss until the convergence condition is met. The convergence condition can be that the number of training rounds reaches a preset upper limit, the verification loss no longer decreases significantly in several consecutive training rounds, or the model performance reaches a preset requirement.

[0054] After each training round, the validation loss can be calculated using the validation set to monitor the model's generalization ability and prevent overfitting. When the validation loss reaches an optimal value, the corresponding network parameters are saved as an offline model. The error estimation network trained offline can learn the error distribution patterns among multi-sensor data and has the ability to make initial estimates of temporal biases and distortion parameters under complex dynamic conditions, thus providing a foundation for subsequent online error compensation and model fine-tuning.

[0055] The loss function can be Huber loss, which balances smooth optimization within a small error range with robustness to outliers, thereby improving the stability of the training process.

[0056] In one possible implementation, S200, based on multi-sensor feature data and an error estimation network, obtains an error estimation vector, including:

[0057] S210 utilizes the input encoding layer of the error estimation network to extract high-dimensional features from multi-sensor feature data, obtaining multi-modal high-dimensional feature vectors. The input encoding layer includes an image encoder, a LiDAR encoder, and an IMU encoder. The image encoder and LiDAR encoder are convolutional neural networks, while the IMU encoder is a temporal convolutional network or a gated recurrent unit.

[0058] For example, the multi-sensor feature data can be constructed as input data using the method in step S200.

[0059] Image encoders can employ two-dimensional convolutional neural networks (2D CNNs), such as lightweight ResNet-18 or EfficientNet-B0. Since the input is optical flow rather than an RGB image, the number of input channels in the first convolutional layer of the image encoder can be adjusted to... (e.g. 4).

[0060] Optical Flow Tensor The network employs multiple convolutional blocks, each containing a convolutional layer, batch normalization (BatchNorm), and a ReLU activation function. As the network deepens through convolutional operations, the spatial resolution of the feature map gradually decreases, while the number of channels gradually increases. The image encoder ultimately outputs the feature map. ,in, This represents the number of feature channels (e.g., 256). , This represents the spatial dimensions after downsampling.

[0061] Can be Applying global average pooling, we average along the spatial dimensions to obtain a fixed-length feature vector. This vector encodes the motion information of the entire image region, providing a unified representation for subsequent cross-modal fusion.

[0062] LiDAR encoders can employ two-dimensional convolutional neural networks, similar in structure to image encoders, and the number of input channels can be adjusted. (e.g., 2). Distance-Time Graph Local spatial structure and temporal distribution features are extracted through multiple convolutional layers. After obtaining feature maps through deep convolution, global average pooling is used to generate fixed-length feature vectors. This vector encodes the spatial geometry of the point cloud and the temporal distribution information of intra-frame sampling.

[0063] IMU encoders can employ either Temporal Convolutional Networks (TCNs) or Gated Recurrent Units (GRUs). TCNs support parallel computation and can capture long-term dependencies by expanding the receptive field through dilated convolutions.

[0064] IMU temporal tensor The original 6-dimensional input is mapped to a higher-dimensional space (e.g., 64-dimensional) through 1x1 convolutions. Multiple dilated convolutional layers (or GRU layers) are stacked to gradually expand the receptive field, capturing motion patterns at different time scales. Global average pooling is then performed on the time dimension to generate a fixed-length feature vector. This vector encodes the dynamic information of inertial motion throughout the entire time window.

[0065] S220 utilizes the cross-modal feature fusion layer of the error estimation network to fuse multimodal high-dimensional feature vectors, obtaining a fused feature vector. The cross-modal feature fusion layer employs a multi-head cross-attention mechanism to capture complementary information and correlations between different modalities.

[0066] For example, the feature vectors output by three modality encoders can be stacked into a feature sequence. Where D is the unified feature dimension. Since the order of the feature sequence implies the modality type, a learnable positional encoding can be added to each modality in the feature sequence. ,Right now .Will The input is a Transformer encoder layer (i.e., a cross-modal feature fusion layer). This layer calculates attention weights between each modal feature vector and other modal feature vectors, achieving weighted aggregation of information. After one or more Transformer encoder layers, a new feature sequence is output. Each These three fused feature vectors, which incorporate information from other modalities, can be concatenated to obtain a fused feature vector. .

[0067] Optionally, in step S220, the multimodal high-dimensional feature vectors are fused using the cross-modal feature fusion layer of the error estimation network to obtain a fused feature vector, including:

[0068] It is understandable that the cross-modal feature fusion layer can adopt a one- or multi-layer Transformer encoder structure, with each layer including a multi-head self-attention (MHA) sublayer and a feedforward network (FFN) sublayer, and residual connections and normalization are used to ensure stable gradient propagation.

[0069] S221, construct an initial feature sequence from the multimodal high-dimensional feature vectors, and add positional encoding to each feature vector in the initial feature sequence.

[0070] For example, since the three modalities may have different dimensions, an independent linear projection layer (fully connected layer) can be introduced for each modality to map them to a common embedding space of a unified dimension D, i.e. , , ,in, , , , representing the modal eigenvectors after linear projection. , , This represents the linear projection weight matrix (weights of the fully connected layer) for each modality, with dimensions of... , The dimension of the original feature vector for each modality (e.g.) ), , , This represents the linear projection bias vector (bias term of the fully connected layer) for each modality, with dimension D. This process facilitates subsequent attention calculations by providing a unified feature dimension basis, while also enabling different modalities to perform similarity measurements and information exchange within the same vector space.

[0071] The three embedding vectors can be stacked in a fixed order to construct a modal token sequence of length 3. And add modal type encoding to each token. (Also known as positional encoding / modal encoding), used to explicitly inform the cross-modal feature fusion layer that the 1st, 2nd, and 3rd tokens correspond to the camera, LiDAR, and IMU, respectively. ,in, These are learnable parameters (updated during training along with the rest of the network parameters). This process prevents the attention layer from treating tokens from different modalities as homogeneous sequences, thereby improving the discriminability of cross-modal interactions.

[0072] S222 utilizes the attention layer of the cross-modal feature fusion layer to calculate and aggregate the attention weights between each feature vector in the initial feature sequence to obtain the second feature sequence.

[0073] For example, the attention layer will input sequence (No. The layer input generates the attention query, key, and value respectively. , , ,in, , , This represents a linear mapping matrix. Q, K, and V are divided into h subspaces based on the number of attention heads h (a hyperparameter, such as 4, 8, or 16 heads) (each head has a dimension d = D / h). , , ,in, , , .

[0074] For the first Each attention head is used to calculate attention weights. Attention weight matrix This indicates the level of interest that one modal token has in another modal token. For example, [1,3] can be understood as the attention weight of the camera token to the IMU token. This can be represented using a weight matrix. The output feature matrix of the attention head is obtained by performing a weighted summation on the Value. Concatenate the output feature matrices of all attention heads along their feature dimensions. And linearly projected back to D dimension , Represents the linear projection matrix. This represents the output of all attention heads on the input sequence. With input sequence Sum the residuals and normalize them (either Pre-LN or Post-LN is acceptable), that is... , ,in, This represents the second feature sequence. Dropout is used to improve generalization and noise resistance, and to prevent a certain modality from being locked into excessive dependence in the early stages of training.

[0075] S223, using the feedforward network of the cross-modal feature fusion layer, the residual normalization of the second feature sequence is performed to obtain the fused feature sequence.

[0076] For example, a feedforward network (FFN) consists of two fully connected layers with nonlinear activations: ,in, You can use GELU or ReLU. , These represent the weight matrices of the first and second layers of the FFN, respectively. , These represent the bias vectors of the first and second fully connected layers, respectively.

[0077] For the second characteristic sequence Each token is independently applied FFN to obtain Similarly, residual connections and normalization are used to obtain the fused feature sequence, i.e. .

[0078] Steps S222 and S223 represent the fusion process of a single-layer Transformer. If a multi-layer Transformer structure is configured, it can... Repeat steps S222 and S223 as input to the next layer, thereby forming a deeper cross-modal interaction capability.

[0079] S224, concatenate each feature vector in the fused feature sequence to obtain the fused feature vector.

[0080] For example, after L layers of Transformer fusion, the final fused feature sequence is obtained. The three modalities can be concatenated to form a fused feature vector. This approach preserves the independent representation of each modality, making it suitable for multi-task output heads to utilize different modal dominant information.

[0081] The fused feature vector takes into account both the local information and global correlation of each modality, and can more accurately represent the joint state of multi-sensor data. It provides a richer, more stable and reliable input representation for subsequent error estimation, compensation and multi-modal fusion processing, thereby improving the system's ability to perceive and compensate for cross-modal temporal deviations and motion distortions.

[0082] S230 utilizes the error regression output layer of the error estimation network to regress the fused feature vector, thereby obtaining the error estimation vector.

[0083] For example, the error regression output layer can adopt a multi-task learning architecture, sharing the fused feature vectors from the underlying layer. It also branches out into multiple task-specific regression heads, each responsible for predicting one or more specific error parameters.

[0084] The delay regression head can consist of 2-3 fully connected layers (FC layers). Input delay regression header, output relative delay header The activation function can be linear (unrestricted) or Tanh activation when the delay range is known, which restricts the output to [-1, 1] or other scaled ranges.

[0085] The distortion parameter and scan residual regression head are also composed of fully connected layers. Input distortion parameters and scan residual regression head, which outputs distortion parameters. and scan angular velocity residual The activation function can be linear activation.

[0086] The confidence regression head can consist of a fully connected layer and a sigmoid activation function. The confidence regression head outputs a confidence score c between [0,1]. The confidence regression head quantifies the network's confidence in its own error estimation; the confidence score c is used for weight adjustment or compensation mode selection during subsequent compensation execution and online learning fine-tuning.

[0087] The reliability of confidence estimation can be improved by employing deep learning techniques based on evidence. Treating the error prediction as a Gaussian distribution, where the delayed regression head outputs the mean μ, a new branch outputs the variance. (The positive value is ensured by the logarithmic join function), and the confidence level c can be correlated with the variance. During inference, the network outputs ( , ), and by The transformation yields a confidence level c. The larger the variance, the lower the confidence level, reflecting the uncertainty of the prediction.

[0088] The S300 performs error compensation on the raw data from multiple sensors based on the error estimation vector, obtaining multi-sensor compensated data. This multi-sensor compensated data includes compensated image data, LiDAR point cloud data, and IMU data.

[0089] For example, the timestamp of the current LiDAR frame can be used as a reference time, based on the relative delay. The image data is shifted or interpolated and resampled over time to obtain image data that is more consistent with the LiDAR scanning time. Based on this image data, distortion parameters are then used to... Geometric correction is performed on each row or pixel position of the image to reduce image distortion caused by line-by-line exposure, thereby obtaining compensated image data.

[0090] Based on the scan angular velocity residual The angular velocity parameters in the lidar scanning model are corrected to recalculate or correct the position of each point in the LiDAR point cloud data, thereby reducing the point cloud distortion caused by the scanning process and motion, and obtaining the compensated LiDAR point cloud data.

[0091] Based on relative delay The time series of IMU data is aligned by means of time interpolation or resampling to make the IMU data consistent with the compensated image data and LiDAR point cloud data under a unified time reference, thus obtaining the compensated IMU data.

[0092] The confidence level 'c' can be used to weight or constrain the above compensation process. When the confidence level is high, the error estimation vector can be used directly for compensation; when the confidence level is low, the influence of the error estimation vector on data correction can be reduced, or smoothing can be performed by combining historical estimation results to improve the overall stability of the system.

[0093] This step can effectively reduce data mismatch problems caused by timing inconsistencies and motion distortion, thereby improving the alignment accuracy between data from different sensors.

[0094] In one possible implementation, step S300 involves performing error compensation on the original multi-sensor data based on the error estimation vector to obtain multi-sensor compensated data, including:

[0095] S310, determine the compensation mode based on the confidence level of the error estimation vector. The compensation mode can be a fine-grained compensation mode, a hybrid compensation mode, or a basic compensation mode.

[0096] For example, a high confidence threshold can be predefined. (e.g., 0.8) and low confidence threshold (e.g., 0.3), based on the confidence level, the compensation mode is divided into three modes: Fine-grained compensation mode: the trigger condition is c. Using all parameters in the error estimation vector, the specific compensation method is described in steps S301-304 below.

[0097] Hybrid compensation mode: The trigger condition is Using some parameters from the error estimation vector: Time offset compensation: relative time delay can be used. The original camera exposure time in the image data (After the hardware synchronization is established (step S100), the camera can generate a high-precision hardware timestamp at or near the exposure center moment) for correction, that is... No image remapping is performed.

[0098] Roller shutter distortion compensation: Extracting the first line of exposure time from image data and line exposure interval And calculate the exposure timestamp for each row of pixels, i.e. ,in, Indicates the first Row pixel exposure timestamp. This can be determined based on... =( , ), calculate the row offset, i.e. , ,in, Indicates the first row relative to reference row Time difference, Indicates the reference row index, ( , ) indicates the first The pixel offset of the row. Based on the row offset, each row is linearly shifted, i.e. , ,in, Indicates the first The original position of the line, Indicates the distortion correction result of the first The position of the line. Image distortion correction is not performed using IMU data, only distortion parameters are used. .

[0099] LiDAR point cloud motion compensation: The center or end time of the current frame of the LiDAR sensor can be selected as the reference time. Based on IMU data, a continuous pose trajectory from the start of a frame to the end of a frame is constructed using IMU pre-integration. And calculate the continuous pose trajectory of LiDAR. ,in, This represents the pose transformation of the LiDAR coordinate system relative to the IMU coordinate system, obtained through offline calibration. The offset time of each point relative to the start of the frame can be extracted from the LiDAR point cloud data. and the coordinates of each point And calculate the absolute sampling time for each point. ,in, This represents the hardware timestamp indicating the start of the LiDAR scan. The sampling time can be calculated for each point. to reference time pose transformation, and the coordinates of each point Transform to the coordinate system of the reference time, i.e. Only IMU data is used; scan angular velocity residuals are not used. .

[0100] Basic compensation mode: The trigger condition is c The error estimation network's current output can be considered unreliable, so the error estimation vector is not used. Time offset compensation uses only hardware timestamps and nearest neighbor matching or linear interpolation for time alignment; no rolling shutter distortion compensation is performed, only the original image is used; only simple point cloud stitching is performed, without motion compensation; IMU data retains its original timestamps without additional compensation.

[0101] S320 performs error compensation on the raw data from multiple sensors based on the compensation mode and error estimation vector to obtain multi-sensor compensated data.

[0102] For example, the original data from multiple sensors is compensated for errors according to the compensation method in the compensation mode selected in the previous step to obtain multi-sensor compensated data.

[0103] These steps enable adaptive adjustment of the error compensation strategy for multi-sensor data, which can reduce the risk of erroneous compensation while ensuring compensation accuracy, improve the alignment consistency and fusion reliability of multi-sensor data, and thus enhance the perception stability and robustness of the system in complex dynamic environments.

[0104] In one possible implementation, the fine-grained compensation pattern includes:

[0105] S301, based on the relative time delay, the image data is remapped to the reference time to obtain the initial compensated image. The reference time is either the center time or the end time of the current frame of the LiDAR sensor.

[0106] For example, a continuous pose trajectory from the start of a frame to the end of a frame can be constructed based on IMU data using IMU pre-integration or visual-inertial odometry (VIO). And calculate the continuous pose trajectory of the camera. ,in, This represents the pose transformation of the camera coordinate system relative to the IMU coordinate system, obtained through offline calibration. The calculation starts from the camera's original exposure time. to reference time Relative pose transformation between .

[0107] Based on relative pose Based on the scene depth (which can be obtained from approximate depth or LiDAR projection), the original image is remapped, the new position of each pixel in the new image is calculated, and interpolation padding is performed to generate an initial compensated image.

[0108] because For small (millisecond-level) images with limited motion speed and minimal changes in pixel position, image remapping can be avoided; instead, the image's timestamp can be adjusted. This aligns it with the timestamps of the LiDAR point cloud.

[0109] S302: Calculate the row-level pose based on IMU data, reference time, the first row exposure time and row exposure interval in the image data; calculate the row offset based on the distortion parameters, row exposure interval and row-level pose; remap the initial compensated image based on the row offset to obtain the compensated image data.

[0110] For example, based on the results obtained through IMU pre-integration Calculate the continuous pose trajectory of the camera ,in, This represents the pose transformation of the camera coordinate system relative to the IMU coordinate system, obtained through offline calibration. The camera pose at each exposure time can be obtained through interpolation. and reference time Camera pose It can be based on and Calculate the relative pose transformation from each row to the reference time. This transformation represents the first row pixels at exposure time to reference time The movement of the camera between them That is, the first Row-level pose of row pixels.

[0111] In step S310, the line exposure interval is used. Calculate ,according to and It can calculate the motion residual of each row in the image plane. And calculate the row offset for each row. ,in, The pixel displacement predicted by the IMU is derived from the row-level pose. What can be obtained by pushing: can Expand into a homogeneous transformation matrix , Indicates the first The rotation matrix of the row, Indicates the first The translation vector of the row. We can assume the original coordinates of a pixel are... The camera intrinsic parameter matrix is Convert pixel coordinates to normalized camera coordinates , The corresponding unit ray is Transform P, that is... Assuming the scene depth is Z, then After transformation .Will Reprojecting onto the image, the projection formula is: , Based on the original and transformed coordinates of the pixel, calculate... ,Right now If there is no scene depth for each pixel, then set an average depth, or simply use a rotation matrix, i.e. Reprojection can avoid deep dependencies.

[0112] The coordinates of each pixel can be extracted from the initial compensated image. And perform coordinate transformation (compensation) on the coordinates of each row of pixels: , This represents the compensated pixel coordinates. Since the transformed coordinates are floating-point numbers, bilinear interpolation can be used to generate the final pixel values.

[0113] This step can improve the spatiotemporal consistency between images, LiDAR point clouds, and IMU data, reduce the impact of image deformation and cross-modal misalignment on subsequent feature extraction, factor construction, and fusion optimization, thereby improving the accuracy of multi-sensor compensation and the accuracy and stability of system perception, localization, and mapping.

[0114] S303 calculates the LiDAR sampling time based on the scan angular velocity residual, the offset time in the LiDAR point cloud data, and the scan start time. Then, based on the LiDAR sampling time, the reference time, and the IMU data, coordinate compensation is performed on the LiDAR point cloud data to obtain the compensated LiDAR point cloud data.

[0115] For example, using the offset time of each point and scan start time Calculate the sampling time for each point And using the scan angular velocity residual The sampling time for each point is corrected, i.e. This yields the corrected sampling time (LiDAR sampling time) for each point.

[0116] For each point, the corrected sampling time can be calculated. to reference time pose transformation, and the coordinates of each point Transform to reference time In the coordinate system, that is .

[0117] This step can improve the geometric consistency and spatial structure accuracy of point cloud data, reduce point cloud stretching, distortion and misalignment, and enhance the spatiotemporal alignment between LiDAR point clouds and image data and IMU data, thereby providing more reliable point cloud input for subsequent feature extraction, matching constraint construction and fusion optimization.

[0118] S304, based on the relative time delay, corrects the sampling time of the IMU data to obtain the compensated IMU data.

[0119] For example, suppose the sampling time in the IMU data is Using relative delay Correct the sampling time: .

[0120] You can also define a confidence gating function, which helps to avoid abrupt changes caused by mode switching: Gating the error estimation vector: , , This process ensures that the compensation gradually decreases as the confidence level c decreases, rather than being suddenly shut down.

[0121] S400 constructs a factor map based on multi-sensor compensation data and optimizes the factor map to obtain a residual set; based on the residual set, it calculates the overall quality score. The residual set includes image residuals, LiDAR residuals, and IMU residuals.

[0122] For example, image factors can be constructed based on compensated image data to characterize the consistency constraints between image observations and corresponding states; LiDAR factors can be constructed based on compensated LiDAR point cloud data to characterize the consistency constraints between point cloud geometric relationships and states; and IMU factors can be constructed based on compensated IMU data to characterize the consistency constraints between motion propagation relationships and states across consecutive time points. Introducing the image factors, LiDAR factors, and IMU factors into the factor graph together forms a joint constraint model for multi-sensor compensated data. States can include state variables such as pose, velocity, and bias.

[0123] A nonlinear least squares optimization method can be used to iteratively solve for each state node in the factor graph, thereby reducing the overall error of various constraint factors. During the optimization process, the residual values ​​corresponding to each type of factor can be calculated separately, thus obtaining the residual set. Image residuals are used to reflect the degree of matching between image observations and optimized states, LiDAR residuals are used to reflect the degree of satisfaction of point cloud geometric constraints, and IMU residuals are used to reflect the degree of satisfaction of inertial motion constraints.

[0124] The image residual, LiDAR residual, and IMU residual can be weighted and summed to obtain a comprehensive quality score. The comprehensive quality score is used to characterize the overall quality of the multi-sensor compensation data in the current frame. The comprehensive quality score can be used as a basis for triggering subsequent model fine-tuning, and can also be used to evaluate whether the current error compensation effect meets the system requirements.

[0125] This step not only allows for a more accurate determination of whether the current multi-sensor data has reached a good time synchronization state, but also provides a reliable basis for subsequent model fine-tuning triggers, thereby improving the system's ability to evaluate the compensation effect and the stability and reliability of multi-sensor fusion processing.

[0126] In one possible implementation, in step S400, a factor map is constructed based on the multi-sensor compensation data, and the factor map is optimized to obtain a residual set, including:

[0127] S410: Extract key point coordinates from the compensated image data; extract feature point set from the compensated LiDAR point cloud data; pre-integrate the compensated IMU data to obtain the pre-integrated quantity.

[0128] For example, feature detection algorithms (such as ORB, FAST, SuperPoint, etc.) can be used to detect salient feature points (key points) in the compensated image data, and corresponding descriptors can be calculated to obtain the image feature set. ,in, Indicates the first The pixel coordinates of key points in the image This represents the corresponding feature descriptor vector. For example, the ORB algorithm is used to detect keypoint coordinates in a compensated image. A local neighborhood is selected centered on the key point, and the gray-level distribution of the neighborhood is encoded to generate a descriptor describing the local appearance features of the key point. This descriptor is essentially a vector used to represent the texture and structural features around the point, facilitating matching with keypoints in historical or adjacent frames. The extracted keypoints can be matched with image features in historical keyframes, thereby establishing cross-frame visual correspondences and generating visual observation constraints.

[0129] The compensated LiDAR point cloud data can be traversed along scan lines or local neighborhoods to calculate the degree of geometric change between each point and its neighbors, such as local curvature, normal variation, and depth gradient. Points with significant geometric changes are identified as edge feature points, while points with gentler geometric changes and approximately coplanar features are identified as planar feature points. All edge and planar feature points are then added to the LiDAR feature point set. ,in, Indicates the first The three-dimensional coordinates of each feature point. By matching this set of feature points with a local map or historical point cloud, geometric constraints between points and planes or lines can be established for subsequent optimization calculations.

[0130] We can assume that the time of the previous frame is The current frame time is For the time interval corresponding to the previous frame and the current frame [ , The triaxial acceleration and triaxial angular velocity data from all compensated IMU data within the range are pre-integrated to obtain the IMU pre-integrated quantity. , , , Indicates within the time interval [ , Displacement increment within ] Indicates the speed increment. This indicates the amount of change in pose.

[0131] S420: Based on the keypoint coordinates, a visual reprojection factor is constructed; based on the feature point set, a LiDAR matching factor is constructed; and based on the pre-integration quantity, an IMU pre-integration factor is constructed. The factor map includes the visual reprojection factor, the LiDAR matching factor, and the IMU pre-integration factor.

[0132] For example, nodes in the factor graph represent the system state to be estimated. The state variables in the factor graph can include the state of the previous frame and the state of the current frame, where the state vector of the current frame is... The state vector of the previous frame is ,in, , , These represent the pose rotation matrix, position vector, and velocity vector, respectively. , These represent the IMU accelerometer bias and gyroscope bias, respectively. Visual reprojection factors, LiDAR matching factors, and IMU pre-integration factors can be constructed based on keypoint coordinates, feature point sets, and pre-integration quantities, and all these factors can be added to the factor graph.

[0133] We can assume that the first The coordinates of the key points are: ,in, Indicates the first The key points are the actual pixel locations in the compensated image. In order to be with the first The three-dimensional spatial points corresponding to each key point The predicted projection position in the current frame is ,in, Let be the camera projection function, representing the projection of 3D points onto the image plane. The residual of the visual reprojection factor (i.e., the image residual) is expressed as: ,in, This reflects the difference between the predicted projection position and the actual observation position of feature points. The visual reprojection factor is used to constrain the consistency between the current frame image observation and the corresponding state of the current frame.

[0134] The set of feature points After transforming the current frame's corresponding state to the map coordinate system, the transformed position of the feature point in the map coordinate system is: The residuals of the LiDAR matching factor (i.e., the LiDAR residuals) are expressed as follows: ,in, This represents the normal vector of the plane that the feature point matches in the local map. Represents a reference point on the plane. The signed distance from a point to the matching plane measures the registration error between the current point cloud and the local map. The LiDAR matching factor constrains the consistency between the current frame's point cloud observations and the local map geometry.

[0135] The residual representation of the IMU pre-integration factor (i.e., the IMU residual) can be constructed based on the pre-integration of the keyframe: ,in, , , , The time interval between the previous frame and the current frame is represented by g, where g represents the gravity vector. This represents rotation combination operations. This represents the vector form that maps the rotation error to the minimum parameter space. The IMU pre-integration factor is used to constrain the motion continuity between the state at the previous reference time and the corresponding state in the current frame.

[0136] This step fully leverages the complementarity between different modes, improves the accuracy of current frame state estimation and observation consistency analysis, enhances the system's robustness under complex motion, texture degradation, and sparse point cloud conditions, and provides a reliable constraint basis for subsequent residual solution and compensation quality quantification.

[0137] S430 constructs an objective function based on the visual reprojection factor, LiDAR matching factor, and IMU pre-integration factor.

[0138] For example, we can assume that the set of state variables to be optimized in the factor graph is ,in, This represents the state vector corresponding to the current frame. This represents the state vector corresponding to the previous frame. Based on the residual representations corresponding to the visual reprojection factor, LiDAR matching factor, and IMU pre-integration factor, the objective function is constructed. ,in, Indicates the first The residuals of each visual reprojection factor This represents the residual of the k-th LiDAR matching factor. denoted by , m represents the residual of the IMU pre-integration factor, m represents the number of keypoints involved in optimization in the previous frame, and n represents the number of feature points involved in matching in the LiDAR point cloud of the current frame.

[0139] S440 uses the nonlinear least squares method to solve the objective function and obtain the residual set.

[0140] For example, since the objective function includes rotation transformation, projection transformation, and pre-integration constraints, and is therefore a nonlinear function overall, iterative optimization can be used to solve it. Given the initial values ​​of the set of state variables X, which can be provided by the estimation results from the previous time step, inertial propagation results, or local map matching results, in the current iteration, the various residual representations are linearized to first order near the current state estimation point, resulting in... ,in, Indicates the state increment. The residual is represented by the Jacobian matrix with respect to the state variables.

[0141] Substitute the linearized residual representation into the objective function. , obtain information about the state increment The problem of quadratic approximation: This linear least squares subproblem can be solved using the Gauss-Newton method or the Levenberg-Marquardt method. Taking the Gauss-Newton method as an example, the normal equations can be constructed. Solve the normal equation to obtain the state increment under the current iteration. Utilize this state increment Update the state variables, that is + .

[0142] After completing one state update, the residuals of various factors are recalculated, and it is determined whether the change in the objective function value or the norm of the state increment satisfies the convergence condition. If not, the linearization, solution, and update steps continue until the preset maximum number of iterations is reached or the convergence condition is met. After iterative optimization, the objective function can be optimized. Minimal optimal state estimation result .

[0143] The optimal state estimation results can be used Substituting back into the various residual representations, we calculate the optimized image residual, LiDAR residual, and IMU residual respectively, to obtain the residual set R={ , , },in, Indicates the first Visual reprojection residuals at key points This represents the matching residual of the k-th LiDAR feature point. This represents the IMU motion constraint residual between the previous frame and the current frame.

[0144] This step not only more accurately reflects the degree of deviation between the compensated image data, point cloud data, and IMU data of the current frame and the system state, but also provides a reliable basis for subsequent comprehensive quality scoring, thereby improving the accuracy, stability, and robustness of the current frame compensation quality assessment.

[0145] In one possible implementation, in step S400, a comprehensive quality score is calculated based on the residual set, including:

[0146] S401, Obtain all residual sets within the preset window prior to the current frame, and obtain a residual set sequence. The residual set sequence includes the residual set of the current frame.

[0147] For example, the preset window W can be the time interval corresponding to the most recent M frames before the current frame, or the time interval from the most recent 100ms to 500ms before the current frame. The residual sets corresponding to each frame within the preset window W can be read and arranged into a residual set sequence in chronological order. ,in, , ,..., This represents the frame times within the preset window W, and That is, the residual set sequence includes the residual set of the current frame.

[0148] S402, calculate the magnitude of all image residuals in the residual set sequence, and calculate the root mean square error based on the magnitude of all image residuals to obtain the first index.

[0149] For example, it can be derived from a sequence of residual sets. Extract all image residuals ,in, Indicates the time interval tn. The image residuals of each key point are calculated, and their moduli are calculated respectively. ,in, Indicates the time interval tn. The magnitude of the image residuals at each key point. The total number of image residuals within a preset window can be counted. The root mean square error is calculated using the modulus of all image residuals, resulting in the first index: ,in, The first indicator is represented by m1, which represents the number of image residuals for all keypoints at time tn.

[0150] S403, calculate the average absolute distance of all LiDAR residuals in the residual set sequence to obtain the second index.

[0151] For example, it can be derived from a sequence of residual sets. Extract all LiDAR residuals ,in, Let represent the LiDAR residual of the k-th LiDAR feature point at time tn.

[0152] Since LiDAR residuals represent signed distances from a point to a plane or from a point to a line, to eliminate the canceling effect of positive and negative signs on the averaging result, the absolute value of each LiDAR residual can be taken to obtain... , This represents the absolute distance of the LiDAR residual at the k-th LiDAR feature point at time tn. It can be used to count the total number of LiDAR residuals within a preset window. The second metric is the average of the absolute distances of all LiDAR residuals. ,in, The second index is represented by m2, which represents the number of LiDAR residuals for all LiDAR feature points at time tn.

[0153] S404: Calculate the Mahalanobis distance of all IMU residuals in the residual set sequence, and calculate the average value based on the Mahalanobis distance of all IMU residuals to obtain the third index.

[0154] For example, it can be derived from a sequence of residual sets. Extract all IMU residuals ,in, This represents the IMU residual at time tn. The Mahalanobis distance for each IMU residual can be calculated, i.e. ,in, The Mahalanobis distance of the IMU residuals at time tn is represented. This represents the inverse of the covariance matrix, used to reflect the uncertainty and correlation of the residual components in each dimension. The average Mahalanobis distance of all IMU residuals is the third indicator. ,in, This indicates the third indicator.

[0155] S405 calculates the weighted sum of the first, second, and third indicators to obtain the overall quality score.

[0156] For example, since the indicators may be affected by transient noise, short-term occlusion, or local abnormal matching, a first-order low-pass filter can be used to smooth each indicator: ,in, This represents the smoothed index corresponding to time t in the current frame. express , , , This represents the smoothed index corresponding to the previous time step. This represents the smoothing coefficient.

[0157] Different indicators have different dimensions and numerical ranges. To facilitate subsequent integration, each indicator can be normalized to a unified range (e.g., [0,1]). ,in, This represents the normalized index. , These represent the mean and standard deviation of the corresponding indicators, respectively, and can be obtained from all values ​​within the time window. , , Calculated.

[0158] The three normalized indicators can be weighted and summed to obtain the overall quality score: ,in, , , These represent the weights corresponding to the first, second, and third indicators, respectively, and can be calibrated experimentally.

[0159] These steps not only reduce the impact of instantaneous noise or local anomalies at a single moment on the evaluation results and improve the stability and robustness of the quality assessment, but also more comprehensively and accurately reflect the compensation effect of the current frame and the spatiotemporal alignment level of multiple sensors, providing a reliable basis for subsequent anomaly detection, compensation effect judgment and model fine-tuning triggering.

[0160] S500 determines whether the triggering conditions are met based on the overall quality score and the overall quality score of the previous M frames, triggers fine-tuning, and acquires the multi-sensor raw data, multi-sensor feature data, and overall quality score within the time window. The triggering conditions include both the overall quality score and the overall quality score of the previous M frames being greater than a trigger threshold. The time window includes a period before the first frame of the previous M frames and the time from the first frame of the previous M frames to the current frame.

[0161] For example, the system can store all comprehensive quality scores within a preset sliding window (such as the last 10 minutes) in the background. and calculate all mean and standard deviation It can be based on and Construct the trigger threshold, i.e. , Indicates the trigger threshold. This represents the sensitivity coefficient (e.g., 3~5). When the current frame... > This can help to initially determine if there is an anomaly; it can also read the first M frames (e.g., 5 frames) from the background. And determine all of the first M frames Are they all greater than If the triggering condition is met, fine-tuning is triggered, and the process proceeds to step S600. The triggering threshold dynamically changes with the average and fluctuation range of the system under normal conditions; it is not a fixed value, making it more suitable for different devices and operating conditions. When the normal quality level of the system changes slowly, the triggering threshold can be adjusted accordingly, making it less prone to frequent false alarms due to environmental changes.

[0162] A time window W can be preset. , indicating the first frame in the first M frames A while ago ( arrive The time period (representing the pre-expansion time), and from Up to the current frame The time period. It can read multi-sensor raw data, multi-sensor feature data, and comprehensive quality scores within the time window W from the circular buffer.

[0163] This step enables fine-tuning to be triggered in cases of continuous anomalies rather than momentary fluctuations, thereby reducing the probability of false triggers and ensuring that the extracted data covers the contextual information before and after the anomaly occurs.

[0164] S600 fine-tunes the model parameters of the error estimation network based on the raw data from multiple sensors, the feature data from multiple sensors, and the comprehensive quality score within the time window, thus obtaining the fine-tuned error estimation network.

[0165] For example, all frames within a time window can be filtered based on the overall quality score within the time window, and frames with an overall quality score greater than the trigger threshold (i.e., abnormal frames) and the corresponding multi-sensor raw data and multi-sensor feature data of the abnormal frames can be filtered out.

[0166] Different delay compensation values ​​can be tried within a certain range (e.g., ±50ms) with small steps (e.g., 1ms). For each The system compensates for image data in abnormal frames, calculates the quality of the compensated image data, and selects the image data with the optimal quality index. Pseudo-labels for relative delay of anomalous frames .

[0167] Anomalous frames and their adjacent frames (or local maps) can be selected, and continuous motion trajectories can be constructed using IMU pre-integration. By optimizing and simultaneously adjusting pose and intra-frame motion parameters, the reprojection error or point cloud registration residual can be minimized, thus obtaining pseudo-labels for the distortion parameters of the anomalous frames. Pseudo-labels and scan angular velocity residuals .

[0168] The overall quality score can be used as a supervision signal or weight signal to construct a fine-tuned loss function related to the error estimation vector. Multi-sensor feature data within a time window can be input into the error estimation network, and the network output... , , The pseudo-labels and network outputs are calculated by fine-tuning the loss function. , , The loss value between the two layers is calculated. Based on the loss value, the gradient of each layer parameter in the error estimation network is calculated through backpropagation, and the network parameters are updated using a preset optimization method. The preset optimization method can be a gradient descent method with a small learning rate or an adaptive optimization method to avoid excessive fluctuations in network parameters during online updates. After the parameter update is completed, a fine-tuned error estimation network is obtained, which can be used for error estimation processing in subsequent frames.

[0169] This step enables the error estimation network to adaptively adjust its parameters based on the recent operating status of the system, thereby improving its adaptability to changes in the current environment, equipment status, and time-varying errors.

[0170] In one possible implementation, step S600 involves fine-tuning the model parameters of the error estimation network based on the raw multi-sensor data, multi-sensor feature data, and comprehensive quality score within the time window, to obtain the fine-tuned error estimation network, including:

[0171] S610 filters normal and abnormal frames from within the time window based on the overall quality score within the time window.

[0172] For example, the overall quality score for each frame within the time window W can be... and corresponding trigger threshold Compare, The frame is marked as a normal frame. The frame is marked as an abnormal frame.

[0173] S620: Based on the original multi-sensor data of the abnormal frame, generate a pseudo-label for the abnormal frame. The pseudo-label is the actual error estimation vector.

[0174] For example, with abnormal frames Construct a search space centered at ±50ms, and try different delay compensation values ​​in small steps (e.g., 1ms). For each The system compensates for image data in abnormal frames and calculates the quality of the compensated image data (e.g., visual reprojection error), selecting the frame with the optimal quality index. Pseudo-labels for relative delay of anomalous frames .

[0175] Abnormal frames and their adjacent frames (or local maps) can be selected, and continuous motion trajectories can be constructed using IMU pre-integration. By optimizing (such as local BA or ICP) and simultaneously adjusting pose and intra-frame motion parameters (such as parameterizing the rolling shutter model or LiDAR rotation model), the reprojection error or point cloud registration residual can be minimized, thus obtaining pseudo-labels for the distortion parameters of the abnormal frames. Pseudo-labels and scan angular velocity residuals

[0176] Optionally, step S620 involves generating pseudo-labels for the abnormal frames based on the multi-sensor raw data of the abnormal frames, including:

[0177] S621, construct a search space based on the relative delay in the error estimation vector of the abnormal frame; calculate the first index of each delay compensation value in the search space based on the exposure time of the first row of the abnormal frame, and determine the delay compensation value corresponding to the smallest first index as the relative delay of the pseudo tag.

[0178] For example, suppose the relative delay in the error estimation vector of the anomalous frame is... This relative time delay can be used Construct a one-dimensional search space centered at the center, and let the search half-width be . Then the delay compensation value satisfy ,in, The time drift range can be set according to the maximum allowable time drift range of the system, such as 20ms to 50ms.

[0179] A two-level search strategy can be adopted, which can balance search accuracy and computational efficiency. That is, a coarse search step size is used first. Discretely sample the search space (e.g., 2ms) to obtain a coarse search candidate set. For each delay compensation value ,Will Applied to the image timestamp of the aberrant frame, i.e., assuming the exposure time of the first line in the aberrant frame image data is... Then in Below, the corresponding corrected first line exposure time is Based on the corrected exposure time of the first row. The compensation process with the image data is re-executed, and the corresponding first index is calculated. Used to characterize the delay compensation value The overall error level of the image residuals.

[0180] After completing the coarse search, select the coarse search candidate value that minimizes the first index. ,by Construct a fine search space around the center, with a fine search step size of . (e.g., 0.5ms), can be Further discrete sampling in the vicinity yields a finer search candidate set. For each delay compensation value Similarly, update the exposure time of the first row, i.e. And recalculate the corresponding first index based on the updated first row exposure time. After the fine search is completed, the delay compensation value that minimizes the first metric can be selected as the relative delay of the pseudo-label for the abnormal frame, i.e. .

[0181] This step uses the evaluation results of the compensation effect of the abnormal frame itself to generate more reliable relative delay pseudo-labels, which helps to improve the accuracy and relevance of the supervision signal during subsequent network fine-tuning.

[0182] S622, robust regression fitting is performed on the image residuals of abnormal frames to obtain pixel offsets; distortion parameters of pseudo-labels are calculated based on line exposure intervals and pixel offsets.

[0183] For example, it can be assumed that there are M valid feature points in the abnormal frame, and the first... The observation row number of each feature point is Reference behavior (You can take the center row of the image), the first row. The image residual of each feature point is ,in, Indicates the first The actual observed pixel coordinates of each feature point Indicates the first The predicted projected pixel coordinates of each feature point (distortion parameters in the error estimation vector of the anomalous frame). ).

[0184] For the horizontal pixel residuals respectively and vertical pixel residual Establish a linear fitting model: , ,in, , These represent the slopes of the systematic pixel shift caused by each additional row along the image row direction. , These represent the corresponding intercept terms. Robust regression methods, such as Huber regression or RANSAC regression, can be used to solve the linear fitting model to obtain robust fitting results. , , , Robust regression methods can reduce the impact of dynamic targets, mismatched points, and local outlier residuals on the fitting results.

[0185] Can , Pixel offset speed converted per unit time , The distortion parameters of the pseudo-labels of the abnormal frames are ,in, Indicates the row exposure interval.

[0186] The distortion parameters of the pseudo-labels generated through this step can reflect the true distortion trend in abnormal frames. At the same time, robust regression is used to suppress the interference of mismatched points and local anomalies, thereby improving the reliability of pseudo-labels and providing more effective supervision information for the fine-tuning of the subsequent error estimation network.

[0187] S623, construct candidate intervals based on the scan angular velocity residuals in the error estimation vector of the abnormal frames; calculate the second index of each candidate in the candidate interval based on the offset time and LiDAR residuals of the abnormal frames; and determine the candidate corresponding to the smallest second index as the scan angular velocity residual of the pseudo-tag.

[0188] For example, suppose the scan angular velocity residual in the error estimation vector of the anomalous frame is The residual of the scanning angular velocity can be used. Construct candidate intervals centered at the x-axis, and let the search half-width be y. Then the candidate scan angular velocity residual satisfy ,in, It can be preset according to the system's allowed range of scanning angular velocity drift.

[0189] Candidate intervals can be discretized and sampled at a preset step size to obtain a candidate set. For each candidate scan angular velocity residual The point cloud scanning time axis can be corrected based on the offset time of each LiDAR point in the abnormal frame, i.e. ,in, Indicates the first Offset time corrected for each LiDAR point Indicates the first The original offset time of each LiDAR point. This can be used to determine the corrected offset time. Lightweight deskew compensation is performed on the LiDAR point cloud of the anomalous frame to obtain the candidate scan angular velocity residuals. The compensation point cloud below .

[0190] We can assume that the set of LiDAR feature points of the anomalous frame has K values, then the candidate scan angular velocity residuals The compensation point cloud of the kth node is Compensated point cloud based on all LiDAR feature points ,pass Calculate candidate scan angular velocity residuals LiDAR residuals at each LiDAR feature point .

[0191] according to Calculate the candidate scan angular velocity residual. The corresponding second indicator , Used to characterize the current candidate scan angular velocity residual The matching error level between the LiDAR point cloud and the local map is determined. The candidate set is traversed using the method described above. The candidate scan angular velocity residual that minimizes the second criterion can be selected as the scan angular velocity residual of the pseudo-label for the abnormal frame: .

[0192] This step can generate relatively reliable pseudo-label scanning angular velocity residuals from the registration results of the abnormal frames themselves, which helps to improve the accuracy and relevance of the supervision information during subsequent network fine-tuning, and further improves the accuracy of LiDAR point cloud distortion compensation and multi-sensor spatiotemporal alignment.

[0193] S630 combines multi-sensor feature data and pseudo-labels from all frames within the time window into training sample pairs. In the pseudo-labels for normal frames, the relative delay, distortion parameters, and scan angular velocity residuals are all 0, with a confidence level of 1.

[0194] For example, the multi-sensor feature data and pseudo-labels of all anomalous frames are combined into training sample pairs, i.e., (X, ), where X represents the multi-sensor feature data of the abnormal frame, Pseudo-labels representing anomalous frames, including , , c.

[0195] The multi-sensor feature data and pseudo-labels of all normal frames are combined into training sample pairs, i.e. ( , ),in, Multi-sensor feature data representing normal frames A pseudo-tag representing a normal frame.

[0196] S640: Based on the training sample pairs and the loss function, the error estimation network is iteratively trained, the model parameters of the error estimation network are updated, and the fine-tuned error estimation network is obtained.

[0197] For example, the training sample pairs of abnormal frames can be divided into a training set and a validation set. This can be done chronologically, with the last 20% of samples within the window used as the validation set to monitor overfitting. Since the number of samples is limited (possibly only tens to hundreds of frames), mini-batch training is used, with the batch size set to B=8 or 16 to ensure stable gradient estimation and controllable memory usage.

[0198] You can load the parameters of the error estimation network currently in use online. The learned parameters are then copied and used as initial parameters for fine-tuning and as reference parameters for subsequent regularization constraints. The AdamW optimizer, which combines Adam's adaptive learning rate with decoupled weight decay, is suitable for fine-tuning scenarios. SGD with momentum can also be used, but requires more precise tuning of the learning rate. Extremely low learning rates can be set. ,like or And set the L2 regularization coefficient. = ~ Normal sample supervised loss weights =0.1~0.3, these coefficients can be adjusted through a small number of experiments or based on the performance on the validation set. At the same time, a gradient clipping threshold (such as 1.0) can be set to limit the gradient norm and avoid abnormal gradient increases during fine-tuning.

[0199] Due to the small sample size, training is performed for 5-20 epochs (one epoch refers to going through all training samples once), and the loss is evaluated on the validation set after each epoch. During training, the training set is iterated in epochs. For each training batch, the training samples X and... The input is an error estimation network, which outputs a predicted error estimation vector. and The abnormal sample prediction output, the normal sample prediction output, and the corresponding pseudo-labels are substituted into the loss function to calculate the loss function value for the current batch. Based on this function value, backpropagation is performed on the network parameters to calculate the gradient of the loss with respect to each network parameter. The optimizer updates the network parameters based on this gradient and performs gradient pruning during the update process.

[0200] Repeat the forward propagation, loss calculation, backpropagation, and parameter update process until all batches of the current epoch have been trained. After each epoch, the supervised loss can be calculated on the validation set, and the network can be judged to have overfitted based on the change in the validation loss. If the validation loss no longer decreases for several consecutive epochs, training can be stopped early. After a preset number of epochs or when the early stopping condition is met, the updated model parameters are obtained. .

[0201] After fine-tuning, the updated network parameters can be... Load the error estimation network to obtain the fine-tuned error estimation network. To avoid affecting the stability of the online service, a dual-model buffer mechanism can be adopted. That is, the parameters are updated in the background using the training network. After the fine-tuning is completed and verified, the updated network parameters atomically replace the current online network parameters, while the old network parameters are retained for rollback when necessary.

[0202] Loss functions can include supervised loss and regularization loss: supervised loss can use regression loss (such as Huber loss) to measure the network's current output. and The gap between them, and to incorporate normal supervision, that is The regularization loss uses L2 regularization, which can constrain the changes in network weights, preventing them from deviating too far from the original pre-trained model. The loss function is .

[0203] The above process enables the error estimation network to continuously update and adapt to changes in sensor state and environment, improving the accuracy and robustness of error estimation results, thereby further enhancing the overall performance of multi-sensor data compensation and fusion.

[0204] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0205] Corresponding to the multi-sensor data time synchronization error compensation method described in the above embodiments, this application also provides a multi-sensor data time synchronization error compensation device. The multi-sensor data time synchronization error compensation device includes an image acquisition device, a point cloud acquisition device, an IMU acquisition device, and a control device communicatively connected to the image acquisition device, point cloud acquisition device, and IMU acquisition device. The control device of this embodiment includes: at least one processor, at least one memory, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, it causes the multi-sensor data time synchronization error compensation device to implement the steps in any of the above embodiments of the multi-sensor data time synchronization error compensation method, or to implement the functions of each unit in the above embodiments of the device.

[0206] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the control device of the multi-sensor data time synchronization error compensation device.

[0207] The control device for the multi-sensor data time synchronization error compensation device can be a computing device such as a desktop computer, laptop, or handheld computer. This multi-sensor data time synchronization error compensation device may include, but is not limited to, a processor and memory; for example, it may also include input / output devices, network access devices, and buses.

[0208] The processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or other programmable logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0209] In some embodiments, the memory may be an internal storage unit of the control device of the multi-sensor data time synchronization error compensation device, such as the hard disk or memory of the multi-sensor data time synchronization error compensation device. In other embodiments, the memory may be an external storage device of the multi-sensor data time synchronization error compensation device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) card, flash card, etc., equipped on the multi-sensor data time synchronization error compensation device. Further, the memory may include both internal storage units and external storage devices of the multi-sensor data time synchronization error compensation device. The memory is used to store the operating system, application programs, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0210] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0211] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for compensating for time synchronization errors in multi-sensor data, characterized in that, include: The system acquires the raw multi-sensor data of the current frame and performs feature extraction on the multi-sensor data to obtain multi-sensor feature data. The raw multi-sensor data includes image data, LiDAR point cloud data, and IMU data, and the multi-sensor feature data includes image features, point cloud features, and IMU temporal tensors. An error estimation vector is obtained based on multi-sensor feature data and an error estimation network; wherein, the error estimation network includes an input encoding layer, a cross-modal feature fusion layer and an error regression output layer, and the error estimation vector includes relative time delay, distortion parameters, scan angular velocity residual, and confidence level; Based on the error estimation vector, error compensation is performed on the original data from the multi-sensor system to obtain multi-sensor compensated data; wherein, the multi-sensor compensated data includes compensated image data, LiDAR point cloud data and IMU data, and the compensation mode is fine compensation mode, hybrid compensation mode or basic compensation mode. Based on the multi-sensor compensation data, a factor map is constructed, and the factor map is optimized to obtain a residual set; based on the residual set, a comprehensive quality score is calculated; wherein, the residual set includes image residuals, LiDAR residuals, and IMU residuals; Based on the overall quality score and the overall quality score of the previous M frames, it is determined that the triggering condition is met, triggering fine-tuning, and acquiring the multi-sensor raw data, multi-sensor feature data, and overall quality score within the time window; wherein, the triggering condition includes that both the overall quality score and the overall quality score of the previous M frames are greater than the triggering threshold, and the time window includes a period of time before the first frame in the previous M frames and the time from the first frame in the previous M frames to the current frame; Based on the multi-sensor raw data, multi-sensor feature data, and comprehensive quality score within the time window, the model parameters of the error estimation network are fine-tuned to obtain the fine-tuned error estimation network. The fine compensation mode includes: Based on the relative time delay, the image data is remapped to a reference time to obtain an initial compensated image; wherein, the reference time is the center time or end time of the current frame of the LiDAR sensor; Based on the IMU data, the reference time, the first row exposure time and the row exposure interval in the image data, the row-level pose is calculated; based on the distortion parameters, the row exposure interval and the row-level pose, the row offset is calculated; based on the row offset, the initial compensated image is remapped to obtain the compensated image data; The LiDAR sampling time is calculated based on the scanning angular velocity residual, the offset time in the LiDAR point cloud data, and the scanning start time. Then, coordinate compensation is performed on the LiDAR point cloud data based on the LiDAR sampling time, the reference time, and the IMU data to obtain the compensated LiDAR point cloud data. Based on the relative time delay, the sampling time of the IMU data is corrected to obtain compensated IMU data.

2. The multi-sensor data time synchronization error compensation method as described in claim 1, characterized in that, The error estimation vector obtained based on multi-sensor feature data and the error estimation network includes: Using the input encoding layer of the error estimation network, high-dimensional features are extracted from the multi-sensor feature data to obtain multimodal high-dimensional feature vectors; wherein, the input encoding layer includes an image encoder, a LiDAR encoder, and an IMU encoder, the image encoder and the LiDAR encoder are convolutional neural networks, and the IMU encoder is a temporal convolutional network or a gated recurrent unit; The multimodal high-dimensional feature vectors are fused using the cross-modal feature fusion layer of the error estimation network to obtain a fused feature vector; wherein, the cross-modal feature fusion layer is a multi-head cross-attention mechanism used to capture complementary information and correlations between modalities; The error estimation vector is obtained by regressing the fused feature vector using the error regression output layer of the error estimation network.

3. The multi-sensor data time synchronization error compensation method as described in claim 1, characterized in that, The step of performing error compensation on the original multi-sensor data based on the error estimation vector to obtain multi-sensor compensated data includes: The compensation mode is determined based on the confidence level of the error estimation vector; wherein the compensation mode is a fine compensation mode, a hybrid compensation mode, or a basic compensation mode. Based on the compensation mode and the error estimation vector, error compensation is performed on the original data from the multi-sensor system to obtain the multi-sensor compensated data.

4. The multi-sensor data time synchronization error compensation method as described in claim 1, characterized in that, The step of constructing a factor graph based on the multi-sensor compensation data and optimizing the factor graph to obtain a residual set includes: Extract key point coordinates from the compensated image data; extract feature point set from the compensated LiDAR point cloud data; pre-integrate the compensated IMU data to obtain the pre-integrated quantity; Based on the key point coordinates, a visual reprojection factor is constructed; based on the feature point set, a LiDAR matching factor is constructed; based on the pre-integration quantity, an IMU pre-integration factor is constructed; wherein, the factor map includes the visual reprojection factor, the LiDAR matching factor, and the IMU pre-integration factor; Based on the visual reprojection factor, the LiDAR matching factor, and the IMU pre-integration factor, an objective function is constructed; The objective function is solved using the nonlinear least squares method to obtain the residual set.

5. The multi-sensor data time synchronization error compensation method as described in claim 1, characterized in that, The step of calculating the comprehensive quality score based on the residual set includes: Obtain all residual sets within a preset window prior to the current frame to obtain a residual set sequence; wherein, the residual set sequence includes the residual set of the current frame; Calculate the magnitude of all image residuals in the residual set sequence, and calculate the root mean square error based on the magnitude of all image residuals to obtain the first index; The average absolute distance of all LiDAR residuals in the residual set sequence is calculated to obtain the second index; Calculate the Mahalanobis distance of all IMU residuals in the residual set sequence, and calculate the average value based on the Mahalanobis distance of all IMU residuals to obtain the third index; The first indicator, the second indicator, and the third indicator are weighted and summed to obtain the comprehensive quality score.

6. The multi-sensor data time synchronization error compensation method as described in claim 5, characterized in that, The process of fine-tuning the model parameters of the error estimation network based on the multi-sensor raw data, multi-sensor feature data, and comprehensive quality score within the time window to obtain the fine-tuned error estimation network includes: Based on the comprehensive quality score within the time window, normal and abnormal frames are filtered out from the time window. Based on the multi-sensor raw data of the abnormal frame, a pseudo-label for the abnormal frame is generated; wherein, the pseudo-label is the actual error estimation vector; The multi-sensor feature data and pseudo-labels of all frames within the time window are combined into training sample pairs; wherein, in the pseudo-labels of normal frames, the relative delay, distortion parameters, and scanning angular velocity residuals are all 0, and the confidence level is 1; Based on the training sample pairs and the loss function, the error estimation network is iteratively trained, and the model parameters of the error estimation network are updated to obtain the fine-tuned error estimation network.

7. The multi-sensor data time synchronization error compensation method as described in claim 6, characterized in that, The step of generating pseudo-labels for the abnormal frames based on the multi-sensor raw data of the abnormal frames includes: A search space is constructed based on the relative delay in the error estimation vector of the abnormal frame; based on the exposure time of the first row of the abnormal frame, a first index of each delay compensation value in the search space is calculated, and the delay compensation value corresponding to the smallest first index is determined as the relative delay of the pseudo tag. Robust regression fitting is performed on the image residuals of the abnormal frames to obtain the pixel offset; the distortion parameters of the pseudo-label are calculated based on the row exposure interval and the pixel offset. Candidate intervals are constructed based on the scan angular velocity residuals in the error estimation vector of the abnormal frames; a second index is calculated for each candidate in the candidate interval based on the offset time and LiDAR residuals of the abnormal frames; the candidate corresponding to the smallest second index is determined as the scan angular velocity residual of the pseudo-tag.

8. The multi-sensor data time synchronization error compensation method as described in claim 2, characterized in that, The method of fusing the multimodal high-dimensional feature vectors using the cross-modal feature fusion layer of the error estimation network to obtain a fused feature vector includes: The multimodal high-dimensional feature vectors are used to construct an initial feature sequence, and positional encoding is added to each feature vector in the initial feature sequence. Using the attention layer of the cross-modal feature fusion layer, the attention weights between each feature vector in the initial feature sequence are calculated and aggregated to obtain the second feature sequence; Using the feedforward network of the cross-modal feature fusion layer, the second feature sequence is residual normalized to obtain the fused feature sequence; Each feature vector in the fused feature sequence is concatenated to obtain the fused feature vector.

9. A multi-sensor data time synchronization error compensation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images