An online time alignment method and device and a storage medium

Abstract

This application discloses an online time alignment method, apparatus, and storage medium, belonging to the technical field of multi-sensor fusion and calibration. It collects information from a camera, inertial measurement unit (IMU), and odometry, using the time offset between the camera and IMU as a system state variable. Based on the currently estimated time offset, linear velocity, and angular velocity, it performs first-order time compensation on the reference pose to obtain the compensated pose. Using the compensated pose, it constructs a residual model including a time offset Jacobian term, making the residual differentiable with respect to the time offset. A nonlinear optimization method is executed within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity. Finally, it outputs the currently estimated time offset and a timestamp-compensated image stream. This method does not rely on front-end feature velocity, is applicable to various VINS frameworks, and improves the accuracy and robustness of multi-sensor fusion.

Description

Technical Field

[0001] This application belongs to the technical field of multi-sensor fusion and calibration, and particularly relates to an online time alignment method, device and storage medium. Background Technology

[0002] VINS achieves 6DoF state estimation by fusing observations from cameras and IMUs. However, the two types of sensors inherently differ in sampling frequency, time reference, and link delay. If the time offset (denoted as...) If the model is not accurately modeled, it will significantly reduce accuracy and robustness, and may even lead to divergence.

[0003] However, the existing technology has the following technical defects: 1. Some existing technologies use optimized online calibration, which relies on optical flow pixel speed for time compensation, but it is sensitive to front-end quality and has poor versatility; 2. Some existing technologies use offline calibration (such as Kalibr), which relies on specific calibration boards and processes, and is not suitable for scenarios where the offset drifts with temperature drift / drive changes during operation. Summary of the Invention

[0004] To address the time misalignment issue between the camera and the IMU, this invention provides an online time alignment method, apparatus, and storage medium. The technical solution is as follows: On the one hand, an online time alignment method is provided, including: The multi-sensor data fusion framework is used to acquire image data from cameras, inertial measurement units (IMUs), and odometry. The multi-sensor data fusion framework is implemented in the form of ROS2 nodes. Time offset between the camera and the inertial measurement unit (IMU) As a system state variable; Based on the currently estimated time offset linear velocity angular velocity First-order time compensation is performed on the reference pose to obtain the compensated pose, wherein the linear velocity... and angular velocity Independent of image feature point speed; Using the compensated pose, construct a residual model that includes a time-shift Jacobian term, such that the residuals are sensitive to time shift. Differentiable; A nonlinear optimization method is performed within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU); Output the final estimated time offset And image streams with timestamps compensated.

[0005] Furthermore, the time offset between the camera and the inertial measurement unit (IMU) As a system state variable, the time alignment algorithm is used, and the calculation formula is as follows: ; In the formula, Represents the camera timestamp aligned with the IMU timeline; Indicates the camera's original timestamp; This indicates the time offset between the camera and the inertial measurement unit (IMU).

[0006] Furthermore, first-order time compensation is performed on the reference pose to obtain the time-compensated pose, calculated using the following formula: First-order pose compensation: ; In the formula, This indicates the rotation after time compensation. Represents rotation and position in the world system. It is by The constructed antisymmetric matrix, Indicates angular velocity, Indicates linear velocity. Indicates the position after time compensation. Indicates the position within the world system. This indicates the time update amount in the current iteration. Indicates the first j The time offset of the next iteration; Reprojection residuals, i.e., poses with time compensation: ; In the formula, For reprojection residuals, For the first Frame to point Pixel observation, For the first The world frame of the inertial measurement unit (IMU) rotates. This refers to the rotation from the inertial measurement unit (IMU) coordinate system to the camera coordinate system. The first in the World System Coordinates of feature points For the first The time-compensated position of each inertial measurement unit (IMU) state. This is the translation vector from the origin of the inertial measurement unit (IMU) to the optical center of the camera.

[0007] Furthermore, a residual model including a time-off Jacobian term is constructed using the compensated pose, calculated using the following formula: Jacobian of residual with respect to time offset : ; ; ; In the formula, J Represents the Jacobian matrix. For the Jacobian of the residual with respect to position, Linear velocity, For the Jacobian of the residual with respect to rotation, Rotation and position in the world system It is by The resulting antisymmetric matrix.

[0008] Furthermore, a nonlinear optimization method is performed within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU) using the following formula: Sliding window least squares update: ; ; ; ; In the formula, Indicates the time offset update amount. Represents the information matrix. Represent the gradient term; construct alignment sensitivity within the window using the velocity norm and angular velocity norm. ,form( , And perform least squares update. This represents the Jacobian of position with respect to the residual. The Jacobian representing rotation with respect to time shift. Represents the residual; Joint optimization objective: ; in, Represents the full state, including pose, velocity, bias, feature points, and time offset. , For the IMU pre-integration factor set, Represents the set of visual reprojection factors. This indicates that the a priori residuals are comparable to Jacobi. This represents the Jacobian of the residual with respect to position. Represents the covariance matrix. Represents the information matrix.

[0009] Furthermore, the residual includes at least one of reprojection error, pose consistency error, or direct photometric error.

[0010] Furthermore, the linear velocity angular velocity It originates from at least one of VIO state estimation, IMU pre-integration, or external odometer.

[0011] Furthermore, the nonlinear optimization method uses any one of the solvers: g2o, Ceres, or iSAM2.

[0012] On the other hand, an online time alignment device is provided, comprising: The data input module is used to receive camera images, IMU data, and odometer information; The time compensation module is used to perform first-order time compensation on the reference pose based on the currently estimated time offset, linear velocity, and angular velocity. The residual construction module is used to build residual models that include time-shifted Jacobian terms; The sliding window optimization module performs nonlinear optimization within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU); Output module, used to output time offset Image streams with timestamps already compensated.

[0013] On the other hand, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the online time alignment method described above.

[0014] The technical solution includes at least the following technical effects: 1. Achieved universal online time synchronization: Instead of relying on pixel-level inter-frame velocity, pose and time compensation is driven by system linear velocity and IMU angular velocity, so that the residuals naturally contain time offset. Jacobi can be directly migrated to various optimized VINS / VI-SLAM / LIO.

[0015] 2. Robust and efficient: Insensitive to visual noise; experiments show faster and more accurate convergence under multiple datasets and noisy conditions.

[0016] 3. Easy to implement: Only the residual structure needs to be modified, and the optimization dimensions and overhead are comparable to those of conventional VINS; compatible with g2o and Ceres.

[0017] 4. From "Unable to run" to "Stable and reliable": It solves problems such as missing dependencies, chaotic interfaces, and unstable values ​​in real systems, and provides an industrial product module that can be directly integrated and run for a long time.

[0018] 5. From "fixed parameters" to "flexible adaptation": Through parametric design, the same set of code can be flexibly adapted to sensors of different brands and specifications, which greatly reduces deployment costs.

[0019] 6. From “academic prototype” to “industrial component”: Provided in the form of ROS2 nodes, it has standard communication interfaces and complete lifecycle management, and can be quickly embedded as a standard component into various robots, VR / AR or autonomous driving systems, greatly improving development efficiency.

[0020] 7. Directly replace the pose with the time-compensated pose in the reprojection / geometric factor to offset the time. Jacobian sub-blocks emerge naturally and are jointly optimized with pose / velocity / bias states; the output includes a scalar time offset. Image streams with re-tagged timestamps.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart of an online time alignment method provided in a preferred embodiment of this application; Figure 2 This is a structural block diagram of an online time alignment device provided in a preferred embodiment of the present application. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] Explanation of proper nouns: VINS: Visual-Inertial Navigation System.

[0026] IMU: Inertial Measurement Unit, is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

[0027] Temporal Offset / Calibration: The constant / gradual offset between the camera and the IMU time axis, denoted as... .

[0028] Sliding-Window Optimization: Nonlinear optimization and edge detection within a fixed keyframe window.

[0029] Reprojection Error: The error between the projection of 3D points onto an image and the actual pixels.

[0030] Extrinsics: The pose extrinsic parameters of the camera and inertial measurement unit (IMU).

[0031] As attached Figure 1 As shown: In some embodiments, an online time alignment method is provided that uses time offset as a system state parameter. Pose-time compensation is performed using system linear and angular velocities in the optimized residual model, making the residuals differentiable with respect to time offset. This allows for joint estimation of time offset and other states during sliding window optimization. This method is independent of feature point velocities, thus robust to front-end noise and applicable to various vision-inertial navigation frameworks. The output includes the estimated time offset and a compensated timestamped image stream for downstream use.

[0032] like Figure 1 As shown, an online time alignment method includes the following steps: Step S1, Subscribe to data: Collect image data from the camera, inertial measurement unit (IMU), and odometry based on a multi-sensor fusion framework, wherein the multi-sensor data fusion framework is implemented in the form of ROS2 nodes; Subscribe to and publish ROS2 topics from cameras, inertial measurement units (IMUs), and odometry devices, and receive image data from these devices.

[0033] In an optional embodiment, the method also supports multi-camera systems, maintaining time offsets independently for each camera. .

[0034] In an alternative embodiment, the method is also applicable to rolling shutter cameras, with additional compensation by introducing a line exposure time delay term.

[0035] In an alternative embodiment, the time offset Step size control or boundary constraints are applied during the estimation process to improve numerical stability.

[0036] The multi-sensor data fusion framework defines multiple subscriber classes corresponding to multiple sensors, and also defines a data parsing interface. As a specific embodiment, the system defines multiple subscriber classes corresponding to multiple sensors, such as the CameraSubscriber class and the Inertial Measurement Unit (IMUSubscriber) class, and configures a data parsing interface.

[0037] An online time alignment device is implemented as a ROS2 node, supporting multi-topic subscription and publishing, and allowing dynamic configuration of operating parameters via a parameter server. The ROS2 node configuration includes modules such as a time compensation module, a residual construction module, and a sliding window optimization module, capable of estimating and outputting timestamp-compensated image streams in real time. The multi-sensor data fusion framework, based on the second-generation robot operating system ROS2, supports the functional implementation of each module. Provided as a ROS2 node, it features standard communication interfaces and complete lifecycle management, allowing for rapid embedding as a standard component into various robot, VR / AR, or autonomous driving systems, significantly improving development efficiency.

[0038] Step S2, caching and time sorting: Sort and cache data from different sensors by timestamp to ensure consistent processing order.

[0039] Step S3, Position and Time Compensation: Adjust the time offset between the camera and the inertial measurement unit (IMU). As a system state variable; based on the currently estimated time offset linear velocity angular velocity Perform first-order time compensation on the reference pose to obtain the compensated pose, and then use the currently estimated time offset. t d and speed v angular velocity ω Extrapolated pose to the actual image acquisition time. It should be noted that the time offset... It is an offset that changes with each iteration, getting closer to the true value after each iteration. The linear velocity... and angular velocity It is independent of image feature point velocity. In some embodiments, linear velocity... angular velocity It originates from at least one of VIO state estimation, IMU pre-integration, or external odometer.

[0040] Step S4, Constructing Residuals and Jacobian: Using the compensated pose, construct a residual model that includes a time-shifted Jacobian term, such that the residuals are correlated with the time shift. Differentiable. Construct the reprojection / geometric residual using the compensated pose, and calculate... , The residual represents the Jacobian of the difference with respect to time offset. In some embodiments, the residual includes at least one of reprojection error, pose consistency error, or direct photometric error.

[0041] Step S5, Sliding Window Optimization: Perform a nonlinear optimization method within the sliding window to jointly estimate the time offset. linear velocity angular velocity Periodically use the latest time offset Align the camera images with the time axis of the inertial measurement unit (IMU).

[0042] In some embodiments, the nonlinear optimization method uses any of the solvers g2o, Ceres, or iSAM2.

[0043] Step S6, Publish alignment results: Output the final estimated time offset And image streams with timestamps compensated.

[0044] The calculation process involved is as follows: Time alignment calculation formula: ; In the formula, For camera timestamps aligned with the IMU timeline; This is the camera's original timestamp; This represents the time offset between the camera and the inertial measurement unit (IMU). Pose time compensation uses mathematical methods to "correct" or "predict" a more accurate camera pose, ensuring that this pose precisely corresponds to the instant the image is actually captured. This solves the problem of time stamp asynchrony between the camera and IMU. The calculation formula is as follows: ; In the formula, This indicates the rotation after time compensation. Represents rotation and position in the world system. It is by The constructed antisymmetric matrix, Indicates angular velocity, Indicates linear velocity. Indicates the position after time compensation. Indicates the position within the world system. Indicates the time update amount in the current iteration; At the state estimation level, the sensor pose itself used for calculation is corrected directly through a kinematic model. This allows for time offset... The gradient can naturally enter the optimization problem, thus it can be stably estimated, and it is not dependent on the quality of front-end feature tracking, thus making it more general and robust.

[0045] Reprojection residuals, i.e., poses with time compensation: ; In the formula, Indicates the reprojection residual. Indicates the first Frame to point Pixel observation, Indicates the first The world frame of the inertial measurement unit (IMU) rotates. This indicates the rotation from the inertial measurement unit (IMU) coordinate system to the camera coordinate system. Represents the first world system Coordinates of feature points To indicate the first Position of each IMU state after time compensation This represents the translation vector from the origin of the inertial measurement unit (IMU) to the optical center of the camera.

[0046] Jacobian of residual with respect to time offset : ; ; ; In the formula, This represents the Jacobian of the residual with respect to position. Linear velocity, Represents the Jacobian of the residual with respect to rotation. Represents rotation and position in the world system. It is by The antisymmetric matrix formed; Sliding window least squares update: ; ; ; ; In the formula, Indicates the time offset update amount. Represents the information matrix (least squares). This represents the gradient term (least squares), and step size and boundary constraints are added in the implementation to improve numerical stability. Indicates position / translation in the world system. R Represents the rotation matrix. This indicates the time offset, specifically the time difference between the camera and the IMU. Indicates angular velocity, Indicates linear velocity. The Jacobian matrix representing the position versus the residual. The Jacobian matrix representing the rotation with respect to time shift. Represents the residual. J This represents the Jacobian matrix.

[0047] Joint optimization objectives (prior / inertial / visual factors): ; in, Represents the full state, including pose, velocity, bias, feature points, and time offset. , For the IMU pre-integration factor set, Represents the set of visual reprojection factors. This indicates that the a priori residuals are comparable to Jacobi. The Jacobian matrix representing the residuals with respect to position. Represents the covariance matrix. Represents the information matrix (least squares).

[0048] It should be noted that the online time alignment method provided in this application is mainly characterized by the use of multiple iterative calculations, which results in higher reliability, stronger stability, and more accurate precision.

[0049] Enhanced robustness and faster convergence: Accurate estimation of time offsets even in noisy images / with sparse features And improve 6DoF accuracy.

[0050] Versatility: Decoupled from the front end, pixel speed is not required; compatible with multiple frameworks such as VINS, VI-SLAM, and LIO.

[0051] like Figure 2 As shown, in another specific embodiment, an online time alignment device is provided, comprising: The data input module is used to receive camera images, IMU data, and odometer information; in a specific embodiment, the inputs are: / camera / image_raw (image), / imu (IMU), / vio / odom (odometer / velocity / angular velocity).

[0052] The time compensation module is used to perform first-order time compensation on the reference pose based on the currently estimated time offset, linear velocity, and angular velocity. In one specific embodiment, a time compensator is used based on the current time offset. and( ， Extrapolate reference pose to + .

[0053] The residual construction module is used to build a residual model that includes a time-off Jacobian term. In one specific embodiment, the residual constructor is used to apply the compensated pose to the reprojection / pose consistency residual, so that... It appears explicitly.

[0054] The sliding window optimization module performs nonlinear optimization within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU); in one specific embodiment, a sliding window optimizer is used to update the time offset iteratively using Gaussian-Newton / least squares within a fixed window. And apply step size and boundary constraints.

[0055] Output module, used to output time offset And a timestamp-compensated image stream. In one specific embodiment, the output is: / temporal_calib / time_offset (currently estimated) ), / camera / image_compensated (Image stream with timestamps compensated for, making it easy for downstream users to consume directly).

[0056] The terminal uses ROS2 nodes to implement: message subscription → caching → triggering sliding window linearization and time offset once per frame. Update → Release new time offset Compensated image.

[0057] Window strategy: Construct { using the image / pose information of the most recent NNN frames} , } and iteratively update the time offset The loop continues in the next frame.

[0058] Project structure (C++ / ROS2ament_cmake) include / temporal_calibrator / temporal_calibrator.hpp (core class declaration); src / temporal_calibrator.cpp (Core implementation: caching, nearest neighbor interpolation, sliding window LS update) (Publish compensated images); src / temporal_calibrator_node.cpp (node ​​entry point); src / demo_publisher.cpp (Example data source: 30Hz image + 100–200Hz IMU + odometry, simulating a real 30ms image latency); launch / demo.launch.py ​​(one-click run); CMakeLists.txt, package.xml (depends on rclcpp / sensor_msgs / nav_msgs / std_msgs / Eigen3).

[0059] Experimental data results: In the demonstration simulating a 30 ms image delay, the estimated time offset... It stabilized at approximately 0.03 seconds within a few seconds.

[0060] Key parameters and interfaces: Parameters: init_td (initial value, default 0), window_size (window size, default 50–60), td_min / max (boundary), use_imu_omega (angular velocity source strategy).

[0061] Run: `ros2launchtemporal_calibrator_cppdemo.launch.py`; or `ros2runtemporal_calibrator_cpptemporal_calibrator_node --ros-args`... Residual forms: reprojection, pose consistency (current implementation), direct photometric error (time-compensated version), LIO point-to-surface / line residuals (time-compensated version).

[0062] Velocity / angular velocity source: from VIO state estimation, IMU pre-integration, or external odometry; IMU raw ω / omega ω is preferred to improve stability.

[0063] Optimizers: g2o / Ceres / iSAM2 are all acceptable; this project implements a lightweight LS, which can be smoothly replaced with a full BA.

[0064] In another specific embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the online time alignment method described above.

[0065] Compared with unengineered theoretical code, the online time alignment method and apparatus of this invention, used as a storage medium, achieve the following technical effects: 1. It solves problems such as missing dependencies, confusing interfaces, and unstable values ​​in real systems, and provides an industrial product module that can be directly integrated and run for a long time.

[0066] 2. Through parametric design, the same set of code can be flexibly adapted to sensors of different brands and specifications, which greatly reduces deployment costs.

[0067] 3. The time calibration function is encapsulated as an independent ROS2 node, with clearly defined input / output interface names (e.g., / camera / image_raw, / imu, / temporal_calib / time_offset), achieving decoupling from the main navigation algorithm. Provided as a ROS2 node, it features standard communication interfaces and complete lifecycle management, allowing for rapid embedding as a standard component into various robots, VR / AR, or autonomous driving systems, greatly improving development efficiency.

[0068] This invention relates to the time alignment problem in a visual-inertial navigation system (VINS), where a time offset exists between the camera and the IMU. This leads to a decrease in fusion accuracy. In existing technologies, some methods rely on optical flow calculation of feature velocity for time compensation, but this method is sensitive to the accuracy of front-end tracking and performs poorly in noisy or texture-sparse scenes. Furthermore, some methods cannot estimate the time offset online. Alternatively, additional hardware synchronization may be required.

[0069] This invention relates to a vision-inertial navigation system that incorporates time offset. Incorporating the system state variables into the nonlinear optimization framework, and introducing pose-time compensation based on the system's linear and angular velocities, the residual model is made more accurate and accurate. Differentiable, thus enabling joint estimation during sliding window optimization. This method, along with other states (such as pose, velocity, bias, etc.), is independent of front-end feature velocity, is applicable to various VINS frameworks, and can output timestamp-aligned image streams, improving the accuracy and robustness of multi-sensor fusion.

[0070] This invention also provides a ROS2 implementation of the method, including modules such as a time compensator, a residual constructor, and a sliding window optimizer, which can estimate in real time online. It outputs a timestamp-compensated image stream. The time offset is... Incorporating the system state, when constructing the visual / fusion residual, the reference pose used for this frame is calculated according to ( ， ， Time compensation is performed to make the residuals adapt to the time shift. Differentiable; in sliding window optimization, the latest time offset is used. Align the image with the IMU timeline and iterate continuously to obtain online time synchronization results; this strategy only modifies the residual structure and can be applied to various optimized VINS frameworks.

[0071] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the implementation conditions of this application. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size should still fall within the scope of the technical content disclosed in this application, provided that they do not affect the effects and purposes that this application can produce.

[0072] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0078] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An online time alignment method, characterized by, include: The multi-sensor data fusion framework is used to acquire image data from cameras, inertial measurement units (IMUs), and odometry. The multi-sensor data fusion framework is implemented in the form of ROS2 nodes. Time offset between camera and inertial measurement unit, imu as a system state variable; based on the current estimated time offset linear velocity angular velocity first-order time compensation is performed on the reference pose to obtain a compensated pose, wherein the linear velocity and the angular velocity are independent of the image feature point velocity; Using the compensated pose, construct a residual model that includes a time-shift Jacobian term, such that the residuals are sensitive to time shift. Differentiable; A nonlinear optimization method is performed within a sliding window to jointly estimate the time offset. Linear velocity, angular velocity, periodically using the latest time offset Align the camera images with the time axis of the inertial measurement unit (IMU); Output the final estimated time offset And image streams with timestamps compensated.

2. The online time alignment method according to claim 1, characterized in that, The time offset between the camera and the inertial measurement unit (IMU) As a system state variable, the time alignment algorithm is used, and the calculation formula is as follows: ； In the formula, Represents the camera timestamp aligned with the IMU timeline; Indicates the camera's original timestamp; This indicates the time offset between the camera and the inertial measurement unit (IMU).

3. The online time alignment method according to claim 1, characterized in that, First-order time compensation is applied to the reference pose to obtain the time-compensated pose, calculated using the following formula: First-order pose compensation: ； In the formula, This indicates the rotation after time compensation. Represents rotation and position in the world system. It is by The constructed antisymmetric matrix, Indicates angular velocity, Indicates linear velocity. Indicates the position after time compensation. Indicates position / translation in the world system. This indicates the time update amount in the current iteration. Indicates the first j The time offset of the next iteration; Reprojection residuals, i.e., poses with time compensation: ； In the formula, For reprojection residuals, For the first Frame to point Pixel observation, For the first The world frame of the inertial measurement unit (IMU) rotates. This refers to the rotation from the inertial measurement unit (IMU) coordinate system to the camera coordinate system. The first in the World System Coordinates of feature points For the first The time-compensated position of each inertial measurement unit (IMU) state. This is the translation vector from the origin of the inertial measurement unit (IMU) to the optical center of the camera.

4. The online time alignment method according to claim 1, characterized in that, The residual model, which includes a time-off Jacobian term, is constructed using the compensated pose, and is calculated using the following formula: Jacobian of residual with respect to time offset : ； ； ； In the formula, The Jacobian matrix representing the residuals with respect to position. Indicates linear velocity. Represents the Jacobian matrix of the residuals with respect to rotation. Rotation and position in the world system It is by The resulting antisymmetric matrix.

5. The online time alignment method according to claim 1, characterized in that, A nonlinear optimization method is performed within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU) using the following formula: Sliding window least squares update: ； ； ； ； In the formula, Indicates the time offset update amount. Represents the information matrix. Represent the gradient term; construct alignment sensitivity within the window using the velocity norm and angular velocity norm. ,form( , And perform least squares update. This represents the Jacobian of position with respect to the residual. The Jacobian representing rotation with respect to time shift. Represents the residual; Joint optimization objective: ； in, Represents the full state, including pose, velocity, bias, feature points, and time offset. , For the IMU pre-integration factor set, Represents the set of visual reprojection factors. This indicates that the a priori residuals are comparable to Jacobi. This represents the Jacobian of the residual with respect to position. Represents the covariance matrix. Represents the information matrix.

6. The online time alignment method according to any one of claims 1-5, characterized in that, The residual includes at least one of reprojection error, pose consistency error, or direct photometric error.

7. The online time alignment method according to any one of claims 1-5, characterized in that, The linear velocity angular velocity It originates from at least one of VIO state estimation, IMU pre-integration, or external odometer.

8. The online time alignment method according to claim 1, characterized in that, The nonlinear optimization method uses any one of the solvers: g2o, Ceres, or iSAM2.

9. An online time alignment device, employing the online time alignment method according to any one of claims 1 to 8, characterized in that, include: The data input module is used to receive camera images, IMU data, and odometer information; The time compensation module is used to perform first-order time compensation on the reference pose based on the currently estimated time offset, linear velocity, and angular velocity. The residual construction module is used to build residual models that include time-shifted Jacobian terms; The sliding window optimization module performs nonlinear optimization within a sliding window, jointly estimating the time offset, linear velocity, and angular velocity, and periodically using the latest time offset. Align the camera images with the time axis of the inertial measurement unit (IMU); Output module, used to output time offset Image streams with timestamps already compensated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the online time alignment method as described in any one of claims 1 to 8.

Images