A block coordinate descent joint space-time calibration method and system

Abstract

This application discloses a block coordinate descent joint spatiotemporal calibration method and system, belonging to the field of sensor fusion and calibration technology. The method involves acquiring camera image sequences and inertial measurement unit (IMU) data; performing a first-stage coarse temporal alignment, estimating the temporal mapping parameters between the camera and IMU based on the camera image sequences and IMU data; performing a second-stage spatiotemporal joint optimization, jointly optimizing spatial extrinsic parameters, camera intrinsic parameters, and IMU data under the constraints of the temporal mapping parameters; iterating alternately between the first-stage coarse temporal alignment and the second-stage spatiotemporal joint optimization until a preset convergence condition is met; and outputting calibration results, including spatial extrinsic parameters, temporal mapping parameters, a noise model, and a calibration quality report. The method employs an alternating strategy of first performing coarse temporal alignment and then optimizing spatial extrinsic parameters, significantly improving the algorithm's convergence stability and calibration reliability.

Description

Technical Field

[0001] This application belongs to the field of sensor fusion and calibration technology, and in particular relates to a block coordinate descent joint spatiotemporal calibration method and system. Background Technology

[0002] The fusion of vision and inertial measurement unit (IMU) systems is crucial for achieving autonomous localization and navigation in fields such as robotics, drones, and autonomous driving. However, as heterogeneous sensors, cameras and IMUs inherently suffer from spatiotemporal inconsistencies: firstly, spatial extrinsic parameters, i.e., the rigid transformation relationship between their coordinate systems; and secondly, temporal errors, including fixed trigger / transmission delays and dynamic clock drift. These uncalibrated errors directly introduce into systems such as visual inertial odometry (VIO), leading to a significant decrease in trajectory estimation accuracy.

[0003] Existing calibration techniques mainly fall into two categories: offline batch calibration (such as the Kalibr tool), which optimizes all parameters at once by collecting specific motion data; and online joint estimation, which optimizes time parameters as state variables in VIO in real time. However, these methods have the following technical drawbacks: Insufficient accuracy and stability: The time parameter is strongly coupled with extrinsic parameters and IMU bias, making the optimization process sensitive to initial values ​​and prone to getting trapped in local optima. Furthermore, most methods assume a fixed time delay, failing to effectively handle real clock drift, and the rolling shutter effect can be confused with time differences, leading to distorted parameter estimation.

[0004] Poor robustness: It lacks effective handling mechanisms for common anomalies such as data frame loss, timestamp jumps, and sensor saturation, which makes the calibration process prone to failure.

[0005] Low efficiency and practicality: Large-scale joint optimization has a heavy computational burden and relies on high-quality, highly motivated data acquisition. The process is complex, the degree of automation is low, and the engineering deployment cost is high. Summary of the Invention

[0006] In order to solve the related technical problems in the background art, this application proposes a block coordinate descent joint spatiotemporal calibration method and system.

[0007] The technical solution is as follows: On the one hand, a block coordinate descent joint spatiotemporal calibration method is provided, including the following steps: Acquire camera image sequences and inertial measurement unit (IMU) data, wherein the camera image sequences include image frames and their timestamps, and the inertial measurement unit (IMU) data includes angular velocity and angular acceleration; The first stage of coarse temporal alignment is performed, and the temporal mapping parameters between the camera and the inertial measurement unit (IMU) are estimated based on the camera image sequence and the IMU data. The temporal mapping parameters include time offset and time drift parameters. The second stage of spatiotemporal joint optimization is performed. Under the constraints of the time mapping parameters, the spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) parameters are jointly optimized. The joint optimization is based on the image reprojection residual, the IMU pre-integration residual, and the temporal residual to construct the objective function. Alternately iterate through the first stage of temporal coarse alignment and the second stage of spatiotemporal joint optimization until the preset convergence condition is met; The calibration results are output, including spatial extrinsic parameters, time mapping parameters, noise model, and calibration quality report.

[0008] A further technical solution, wherein the execution of the first stage of coarse time alignment includes: Visual motion features are extracted from the camera image sequence to construct a visual angular velocity sequence; Angular velocity features are extracted from the inertial measurement unit (IMU) data, and the IMU angular velocity sequence is filtered. By using a normalized cross-correlation coarse search, the correlation between the visual angular velocity sequence and the filtered angular velocity sequence of the inertial measurement unit (IMU) is calculated within a preset time offset search range to estimate the initial time offset.

[0009] A further technical solution involves estimating the time drift parameters using a time model; The time model is as follows: in, a Indicates the time drift parameter. b Indicates time offset.

[0010] A further technical solution is that the overall objective function of the spatiotemporal joint optimization is: in, For robust loss function; , , These are the weighting coefficients; For image reprojection residuals; For IMU pre-integrated residuals; For time residuals; This refers to the residual value of the rolling shutter.

[0011] In a further technical solution, the method also includes an anomaly data processing step: Detecting anomalous data during temporal coarse alignment and / or spatiotemporal joint optimization; Abnormal data segments are weighted down or segmented for alignment to filter data quality.

[0012] A further technical solution is that the method also includes a rolling shutter separation modeling step: For rolling shutter cameras, the exposure time of each line of images is modeled, and the exposure time is calculated based on the frame reference time and the total line scan time; In the spatiotemporal joint optimization, the rolling shutter residual is used as part of the objective function, wherein the rolling shutter residual is calculated based on row-level time mapping and image feature point reprojection error. In a further technical solution, the convergence condition includes at least one of the following: The change in time mapping parameters is less than the threshold; The variation range of spatial extrinsic parameters is less than the threshold. Image reprojection error and / or IMU pre-integration residual stabilize within a preset range; The maximum number of iterations has been reached.

[0013] A further technical solution is that the method also supports multi-sensor calibration, including: Set a unified master timeline and set independent time mapping parameters for each camera or inertial measurement unit (IMU); In the spatiotemporal joint optimization, consistent time mapping and joint evaluation are performed on multi-sensor data. On the other hand, a block coordinate descent joint spatiotemporal calibration system is provided, including: The data access module is used to acquire camera image sequences and inertial measurement unit (IMU) data; The coarse time alignment module is used to estimate the time mapping parameters between the camera and the inertial measurement unit (IMU). The spatiotemporal joint optimization module is used to optimize spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) data under time mapping parameter constraints. The alternating iteration control module is used to control the alternating iteration of temporal coarse alignment and spatiotemporal joint optimization, and to determine convergence. The quality assessment and reporting module is used to assess calibration quality and generate calibration quality reports. The results output module is used to output calibration results, which include spatial extrinsic parameters, time mapping parameters, noise model, and quality assessment report.

[0014] In a further technical solution, the system also includes a rolling shutter modeling module, which is used to correct the exposure time of each line based on the image line number and readout time.

[0015] The technical solution includes at least the following technical effects: A block coordinate descent joint spatiotemporal calibration method separates and iterates time coarse alignment and spatiotemporal joint optimization alternately through a block coordinate descent strategy. This gradually refines time mapping parameters (such as time offset and time drift parameters) and spatial parameters (such as spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) data), thereby improving the accuracy, stability, and robustness of the calibration.

[0016] A block coordinate descent joint spatiotemporal calibration system further enhances its engineering practicality and applicability through anomaly detection, rolling shutter separation modeling, and multi-sensor unified time frame. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] Figure 1 A flowchart of a block coordinate descent joint spatiotemporal calibration method provided in a preferred embodiment of this application; Figure 2 This is a block diagram of a block coordinate descent joint spatiotemporal calibration system provided in a preferred embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Explanation of related terms: IMU (Inertial Measurement Unit): An inertial measurement unit is a combination of motion sensors, typically including a three-axis accelerometer and a three-axis gyroscope, used to measure the linear acceleration and angular velocity of an object in three-dimensional space. In this application, the IMU is one of the calibration objects, and its measurement data is used for spatiotemporal alignment with camera information.

[0022] VIO (Visual-Inertial Odometry) is a technique that combines camera image information and IMU inertial information to estimate the trajectory of a device. The calibration results in this application are commonly used to improve the accuracy and stability of VIO systems.

[0023] UI (User Interface): User interface.

[0024] API (Application Programming Interface): Application programming interface.

[0025] CSV (Comma-Separated Values): A comma-separated value data file.

[0026] SDK (Software Development Kit): Software development toolkit.

[0027] FPS (Frames Per Second): Number of frames per second.

[0028] Extrinsic parameters are rigid transformations describing the relative position and orientation between two sensors (such as a camera and an IMU), typically including three-dimensional rotation and translation. This application optimizes these parameters to ensure consistency in the coordinate systems of the two sensors.

[0029] Intrinsic Parameters: These are the internal optical parameters in the camera's imaging model, such as focal length, principal point, and distortion coefficients, used to project three-dimensional points onto the imaging plane.

[0030] Time Offset: Time offset refers to the fixed delay between data timestamps from different sensors. For example, a camera frame time may lag behind or precede IMU readings by tens of milliseconds. This application supports the parsing and correction of this offset.

[0031] Time drift or clock drift refers to the gradual accumulation of time errors caused by inconsistent internal clock frequencies of different devices over time. This application improves the synchronization stability of long-duration x-series data by modeling time drift.

[0032] Provide a block coordinate descent joint spatiotemporal calibration system, such as Figure 2 As shown, the system includes the following modules: The data access module is used to acquire camera image sequences and IMU data, parse the data format, check the timestamp monotonicity, and output standardized data.

[0033] Input: Supports multiple data sources, including camera data files, inertial measurement unit (IMU) data files, or real-time data streams. The IMU data includes angular velocity and angular acceleration. Camera image sequences include image frames and their timestamps.

[0034] The coarse temporal alignment module estimates the temporal mapping parameters between the camera and the inertial measurement unit (IMU) based on the camera image sequence and IMU data. The temporal mapping parameters include time offset and time drift parameters.

[0035] Spatiotemporal joint optimization module: Under the constraints of the time mapping parameters, it performs joint optimization on spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) data.

[0036] Among them, spatial extrinsic parameters refer to the rigid transformation relationship of the relative position and attitude between the camera and the inertial measurement unit (IMU), which includes three-dimensional rotation and translation.

[0037] Representation method: A 3D translation vector: (The position of the origin of the camera coordinate system in IMU coordinates, or vice versa); A 3D rotation (which can be achieved using Euler angles, quaternions, rotation vectors, etc.): .

[0038] The transformation from the IMU coordinate system to the camera coordinate system is written as: ; in, P_cam represents the coordinates of a point in the inertial measurement unit (IMU) coordinate system, and P_cam represents the corresponding coordinates in the camera coordinate system. This entire ( , The spatial extrinsic parameter is an important parameter that this application aims to jointly optimize.

[0039] Camera intrinsic parameters refer to the internal optical parameters in the camera imaging model, such as focal length, principal point, and distortion coefficient, which are used to project three-dimensional points onto the imaging plane.

[0040] The inertial measurement unit (IMU) data includes angular velocity and angular acceleration.

[0041] Rolling shutter modeling module: Used to correct the exposure time of each line based on the image line number and readout time. In some embodiments, the rolling shutter modeling module 104 is present if a rolling shutter is used; otherwise, this module may not be required.

[0042] Alternating Iteration Control Module: Used to control the alternating execution of temporal coarse alignment and spatiotemporal joint optimization, and to determine convergence.

[0043] Function Overview: The time coarse alignment module and the spatiotemporal joint optimization module are called alternately in the outer loop to gradually improve the accuracy of time and space parameters.

[0044] Mathematically, this can be understood as: The time module solves sub-problems: ; The spatial module solves sub-problems: Where k is the number of outer iteration rounds.

[0045] Pseudocode form: initialization x_0 (External and internal parameters of space) a_0 = 1.0, b_0 = 0, For k = 0 to K_max : 1. Call the time coarse alignment module: 2. Write the time map and update the timestamps of all cameras (S4). 3. Invoke the spatiotemporal joint optimization module: 4. Calculate the change in parameters: ; ; The percentage decrease.

[0046] 5) The loop terminates if any of the following conditions are met: Δt_change <0.0001 seconds and ext_change <threshold; error_change <2%; k The preset K_max has been reached.

[0047] Output the final ( a_final , b_final, x_final ).

[0048] The results output module is used to statistically analyze residuals, fluctuations, and uncertainties, generate a calibration quality report, and output calibration results, which include spatial extrinsic parameters, time mapping parameters, noise models, and calibration quality reports.

[0049] 1) Spatial extrinsic parameters: refer to the fixed spatial transformation relationship between different sensor coordinate systems (such as camera and IMU), including rotation matrices. Translation vector This is used to describe the rigid body transformation relationship from the IMU coordinate system to the camera coordinate system.

[0050] 2) Time mapping parameters: These refer to the linear mapping relationship parameters between the time axes of different sensors, and are generally represented by the following linear model: , in, This is the time drift parameter (i.e., the clock scaling factor). This is a fixed time offset parameter.

[0051] 3) Noise Model: This refers to the statistical characteristics of the noise measured by the inertial measurement unit (IMU), including the standard deviation of angular velocity noise σ_gyro, the standard deviation of acceleration noise σ_acc, and the bias random walk intensity σ_bias. This model is used to construct the weight matrix in the joint optimization objective.

[0052] 4) Calibration Quality Report: Quantitative evaluation information of calibration results, including indicators such as root mean square reprojection error (RMSE), time offset residual, external parameter convergence change and outlier ratio, used to verify calibration accuracy and stability.

[0053] The noise model is a statistical model used to describe the random errors in sensor measurements and serves as the basis for weighting in optimization calculations. Taking an inertial measurement unit (IMU) as an example, it includes: 1. Accelerometer noise (accelerometer measurement noise): Modeled as zero-mean Gaussian white noise, with standard deviation denoted as... Unit: m / s^2 (meters per second squared).

[0054] 2. Gyroscope noise (angular velocity measurement noise): Modeled as zero-mean Gaussian white noise, with standard deviation denoted as... Units: rad / s (radians per second) or deg / s (degrees per second).

[0055] 3. Bias random walk: The acceleration bias changes slowly over time, and the noise intensity is denoted as... The gyroscope bias changes slowly over time, and the noise intensity is denoted as... .

[0056] These parameters together determine the covariance matrix of the inertial pre-integral.

[0057] Distortion model: A geometric correction model is established for camera lens imaging distortion (such as radial distortion and tangential distortion) and is used to correct image projection errors.

[0058] As shown in Figure 1, a method for joint spatiotemporal calibration of block coordinate descent is provided, including the following steps: Step S1: Data collection and import, the specific steps are as follows: Step S101: Acquire camera image sequence and inertial measurement unit (IMU) data, wherein the camera image sequence includes image frames and their timestamps, and the inertial measurement unit (IMU) data includes angular velocity and angular acceleration.

[0059] Step S102, data quality inspection and preprocessing (including frame loss detection and timestamp inspection), that is, performing preprocessing operations such as data format parsing, timestamp monotonicity check, and data standardization.

[0060] Step S2: Perform the first stage of coarse temporal alignment, outputting the time offset based on the camera image sequence and IMU data. Time drift parameters .

[0061] enter: Camera image frame sequence: frame timestamp t_c(i), i = 1…N_c; The camera pose estimation R_c(i) comes from simple visual odometry or target pose estimation; IMU angular velocity sequence: Measurements and timestamp , k = 1…N_i.

[0062] Output: The time offset Δt, considering only a fixed delay, satisfies the following relationship between IMU time and camera time: ; When clock drift needs to be considered, output time drift parameter a and time offset parameter b; When there is a clock frequency deviation, the IMU time and the camera time satisfy: ; in, This is the time drift parameter, used to describe the linear drift of the clock frequency. Let a be a fixed time offset parameter. When the drift effect is negligible, we can let a = 1. In this case, b is equivalent to Δt, and we only need to estimate the fixed time offset.

[0063] The specific steps are as follows: S201: Calculate the camera pose difference between adjacent frames and construct a visual angular velocity sequence; S202: Filtering and smoothing the IMU angular velocity sequence; S203: Perform a coarse search within a preset time offset range and calculate the normalized correlation; S204: Perform a fine search or least squares optimization near the optimal result of the coarse search; S205: Outputs a fixed time offset Δt_c and an optional time drift parameter a (when clock drift needs to be considered), used to establish the mapping relationship between camera time and IMU time. ; It should be noted that: Δt is used to represent the most basic and simplest time offset.

[0064] When only the fixed delay between the camera and the IMU is considered, the time relationship can be expressed as: .

[0065] Use cases: It does not consider clock drift; it uses a simple time synchronization model; it is mostly used for principle explanation, basic description, and simplified expression of examples.

[0066] It is used to represent the fixed time offset calculated by the time coarse alignment module, and is a specific estimate of Δt. The "c" in the text stands for "coarse".

[0067] Use cases: The fixed offset obtained in the coarse temporal alignment step. This is the time bias estimate directly output after the coarse search or cross-correlation operation. It is subsequently used as the linear time mapping model. The initial value of b is used.

[0068] The specific algorithm for coarse time alignment is as follows: Construction of visual angular velocity sequence : Given camera pose , Optional, a rotation matrix obtained from a target or visual odometry.

[0069] Camera pose between two adjacent frames , The average angular velocity in the camera coordinate system is defined as approximately: For each pair of adjacent poses: (1) Calculate the rotation increment matrix : in, , The camera poses are between two adjacent frames, and T is the transpose of the matrix; ; (2) Put Convert to equivalent rotation vector This can be achieved using the standard algorithm for converting a rotation matrix into a rotation vector, resulting in a 3D vector whose direction is the rotation axis and whose magnitude is the rotation angle.

[0070] (3) Define the visual angular velocity vector: A time-aligned visual angular velocity sequence was obtained. The time can be taken as: ; ; The visual angular velocity sequence is Its time stamp can be: ; The typical values ​​for the time coarse alignment parameter and the engineering setting principles are as follows: (1) Initial search range: Δt∈[−0.1s, +0.1s], that is, −100ms to +100ms; If the hardware trigger has a known approximate delay, the range can be narrowed down to ±50ms.

[0071] (2) Coarse search step size: Recommended value: (0.0005 seconds); If the data is short, 1ms can be used to improve speed.

[0072] (3) Fine search step size: Further search within ±5ms of the optimal point in the coarse search; Recommended precise search step size: (0.0001 seconds).

[0073] (4) Initial and prior values ​​of time drift parameter a: Initial value: =1.0; Prior constraints: It is generally assumed that |a−1.0|<50 ppm, that is, a is between [0.99995, 1.00005]. The square of "a−1.0" can be used as a regularization term in the optimization.

[0074] S3: Write time mapping and re-timestamp stage; S4: Perform the second stage of spatiotemporal joint optimization. Under the constraints of the current time mapping parameters, jointly optimize the spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) parameters. The joint optimization is based on the image reprojection residual, the IMU pre-integration residual, and the time residual to construct the objective function. Estimate the time model parameters to align the visual angular velocity sequence and the IMU angular velocity sequence in time, minimizing their differences.

[0075] Mathematical form: Solve for parameters a and b to minimize the following expression: ； in: Visual angular velocity (calculated from the pose of adjacent images); : Time stamp of visual angular velocity; Angular velocity obtained by interpolation on the IMU time axis.

[0076] Algorithm steps summary: 1) Constructing a visual angular velocity sequence Time tags ; 2) Filter the IMU angular velocity to obtain ; 3) Use normalized cross-correlation coarse search ; 4) with =1, Given the initial value, solve using least squares. The minimum value; 5) Output , As parameters of the time model.

[0077] S5: Determine whether the convergence condition is met. If not, return to step S2 to continue iterating. If yes, go to step S6. The convergence condition includes at least one of the following: The change in time mapping parameters is less than the threshold; The variation range of spatial extrinsic parameters is less than the threshold. Image reprojection error and / or IMU pre-integration residual stabilize within a preset range; The maximum number of iterations has been reached.

[0078] S6: Analyze residuals and parameter changes to conduct a quality assessment; S7: Export final calibration parameters and quality report.

[0079] The method also includes an anomaly data processing step: Detecting anomalous data during temporal coarse alignment and / or spatiotemporal joint optimization; Abnormal data segments are weighted down or segmented for alignment to filter data quality.

[0080] The method also includes a rolling shutter separation modeling step: For rolling shutter cameras, the exposure time for each row of images is modeled, calculated based on the frame reference time and the total row scan time. In the spatiotemporal joint optimization, the rolling shutter residual is included as part of the objective function, calculated based on row-level temporal mapping and image feature point reprojection error.

[0081] The time model in this application embodiment is compatible with constant offset and clock drift, and is distinguished from the rolling shutter line time scale during optimization and evaluation, reducing systematic deviations caused by parameter compensation: ; in, Indicates the time drift parameter. Indicates time offset.

[0082] The time model mainly operates in the following steps: 1. Time coarse alignment stage S2 In the process of solving Δt or (a, b), the time model is directly used as an unknown quantity in the objective function.

[0083] 2. Writing time mapping and re-timestamping stage S3 The obtained time model is used to remap all camera timestamps to obtain a unified timeline.

[0084] 3. Spatiotemporal Joint Optimization Stage S4 In each residual calculation (reprojection, pre-integration), the camera time is first converted to IMU time through the time model, and then the residual is constructed based on the pose and IMU data at the corresponding time.

[0085] If the time mapping parameters are also used as joint optimization variables, then the parameters (a, b) in the time model will be fine-tuned throughout the optimization process.

[0086] In the spatiotemporal joint optimization, the time module incorporates the rolling shutter residual as part of the objective function. The rolling shutter residual is calculated based on row-level time mapping and image feature point reprojection error. Given or currently estimated time models, jointly optimize camera intrinsic parameters, spatial extrinsic parameters, IMU bias, and trajectory state to minimize the following residuals simultaneously.

[0087] In another embodiment, the time model can be extended to a piecewise structure to handle independent errors from different stages or different sensors.

[0088] The set of variables x includes: Camera intrinsic parameters: ; Spatial extrinsic parameters: , ; IMU bias: , ; Keyframe pose: , ; Keyframe speed: ; Time model parameters (optional): , ; Rolling shutter readout time (optional): ; Objective function: in, For robust loss function; , , These are the weighting coefficients; For image reprojection residuals; For the pre-integrated residual of the inertial measurement unit (IMU); For time residuals; This refers to the residual value of the rolling shutter.

[0089] in: 1. Image reprojection residual : For each observation point: Use the corresponding time first Switch to the IMU timeline; From IMU trajectory interpolation, we can obtain , ; ; ; .

[0090] 2. IMU pre-integration residual : .

[0091] 3. Time residual : If the time mapping parameters are not involved in the joint optimization, they can be omitted. If you wish to participate, the following formula can be used: ; or, 4) Rolling shutter residual : For each row Feature observations: Travel time: Mapping to IMU: Interpolate the pose and project it to obtain , Solution method: Use Gauss-Newton or Lewenberg-Marquardt iteration; Each time the Jacobian matrix is ​​constructed, x is updated in small steps until convergence.

[0092] 4. Alternating Iterative Control Phase (S2 ↔ S6 Cycle) The new time model obtained from each round of coarse time alignment will overwrite the previous one; Spatiotemporal joint optimization uses the latest time model; The convergence criterion uses the "change in the time model over two consecutive rounds" as the judgment.

[0093] The time model runs through the three key steps of coarse time alignment, re-timestamping, and spatiotemporal joint optimization, serving as the "timeline bridge" of the entire calibration process.

[0094] Example 1: Implementation of a Spatiotemporal Integrated Calibration System Based on Standalone Offline Processing 1.1 Implementation Environment and Equipment Conditions Hardware equipment: Monocular industrial camera, 1920×1080 resolution, 30 frames per second, Rolling Shutter mode; Inertial Measurement Unit (IMU), sampling frequency 200Hz, outputs triaxial angular velocity and acceleration; Data transmission method: The USB interface outputs image and IMU data respectively, without hardware-triggered synchronization; Computing device required: Industrial PC, i7 CPU, 16GB RAM, operating system can be either Windows or Ubuntu; Software environment: Python or C++ runtime environment, with image processing library (OpenCV) and nonlinear optimization solution library (such as Ceres or G2O) installed; supports ROS (Robot Operating System) data format parsing.

[0095] 1.2 Data Collection and Preparation The data acquisition device carries an AprilTag or a chessboard target indoors and performs six degrees of freedom motion, with a data recording time of about 60 seconds, covering multi-directional rotation and translational motion to ensure sufficient motion excitation; The sensor outputs are recorded into two types of data files: Camera.csv: Image filenames and capture timestamps; Imu.csv: Gyroscope and acceleration data, timestamps; The timestamps are all internal system times of the device, which are subject to deviation, i.e., there are unknown time offsets and possible clock drifts.

[0096] 1.3 Time Coarse Alignment Stage The steps are as follows: Extract the change in motion direction from adjacent image frames and calculate the direction change sequence of the image sequence; Extract the angular velocity magnitude sequence from the IMU and perform low-pass filtering; The correlation coefficient between the two sequences is calculated using a sliding window, and the initial value of the search time offset is determined. Optional: Repeat the search across multiple intervals to estimate whether a time drift trend exists; Output: Initial time offset (in milliseconds) and, if present, time drift rate.

[0097] 1.4 Spatiotemporal Joint Optimization Stage Correct the image time to the "IMU time axis" using an offset value; The input is fed into the calibration and optimization module to construct two types of errors: image spatial reprojection error and inertial pre-integration error; Introducing camera intrinsic parameters, IMU noise model, and extrinsic parameters, and using nonlinear optimization iterative updates; If rolling shutter mode is enabled, an error correction term is included for the readout time parameter per line.

[0098] 1.5 Alternating Iterative Control Such as the following loop: 1.6 Output and Verification The final output includes the following: Extrinsic parameters from IMU to camera (translation vector, rotation matrix); Time offset value (milliseconds) and drift coefficient; In-camera distortion model; Calibration quality report, including residuals, stability, and proportion of outliers; The results file can be used for testing open-source VIO frameworks (such as VINS, OKVIS, R-VIO), and its trajectory jitter is significantly reduced, verifying the reliability of the technology.

[0099] Example 2: This embodiment of the industrial production line batch calibration solution based on server clusters can be extended for use in factory batch production or equipment maintenance.

[0100] 2.1 Hardware Requirements Multiple cameras and IMU modules (more than ten sets) are assembled into robotic arms or AR glasses by an automated assembly line; Sensors are connected to the production line control system in real time; The data is uploaded to the central server.

[0101] 2.2 Software Architecture Terminal APP (Android / Embedded Linux): responsible for triggering data acquisition and detecting frame drops; Central server: performs time alignment, time-space optimization, and quality auditing; Quality Inspection UI: Automatically generates pass / fail reports.

[0102] 2.3 Exception Robustness Strategy Perform data health assessments on each set of equipment data: frame drop detection; static area detection; IMU saturation detection; automatic downweighting or removal of abnormal data segments, thereby ensuring that optimization is not compromised.

[0103] 2.4 Multi-device calibration consistency All devices are mapped to the same master clock; The output forms a traceable quality control record: Equipment number → External parameters → Time delay → Quality score.

[0104] 2.5 Advantages and Effects It can be used to calibrate hundreds of sets of equipment in a single day; Compared with manual one-time calibration, the failure rate is reduced by more than 80%, and the output error is stable and archived.

[0105] Example 3: Rapid On-Site Calibration Solution for UAVs / High-Speed ​​Motion Platforms 3.1 Applicable Scenarios After a drone has its camera, gimbal, or IMU changed quickly in the field, it needs to be recalibrated. There are rolling shutter images and long flight times causing clock drift.

[0106] 3.2 Key Technical Features Motion is stimulated by the trajectory of the flight itself, without the need for a target; Supports highly dynamic scenarios such as sharp turns and high-speed translation; The alternating iteration mechanism of this invention is integrated into the flight control software to achieve on-site calibration.

[0107] 3.3 Execution Process The flight acquired 30 seconds of images and IMU data. Front-end device preprocessing, including frame dropping and timestamp alignment; Coarse temporal alignment: peak alignment and random sampling crossover; Spatial optimization: Reading the flight control attitude as a weak constraint increases speed and accelerates convergence; Output externally involved time synchronization parameters and send them directly to the navigation algorithm.

[0108] 3.4 Expected Results Rolling shutter distortion is significantly reduced; Drift in the flight trajectory decreased, with the trajectory drift rate dropping from 3.5% to 1.2%.

[0109] No manual targeting or human intervention is required, making it suitable for emergency maintenance in the field.

[0110] Example 4: Introducing a continuous-time (spline) model 4.1 Key Differences Use continuous-time curve models (such as B-splines) instead of discrete attitude sequences. B-spline curves represent continuous-time pose trajectories. Time offset is achieved by shifting curve parameters, eliminating the need for discrete resampling.

[0111] 4.2 Implementation Method Fitting a continuous trajectory curve using an IMU; The trajectory is constrained by image feature points and projected onto the curve; Time alignment is transformed into curve slip optimization control; The alternating loop is still used, but the optimization target becomes the curve parameters.

[0112] Comparison of experimental data and results Table 1. Comparison of calibration effects between single camera and single IMU (indoor target and handheld shaking scenarios) - example experimental results As can be seen from Table 1, this invention reduces the reprojection error by about 30% to 40% while slightly increasing the computation time, controls the time offset estimation error within 1ms, significantly improves calibration stability, and achieves 0 failures.

[0113] Table 2: Example experimental results comparing rolling shutter speed and high-speed drone flight scenarios As can be seen from Table 2, in high dynamic scenes, the present invention utilizes rolling shutter separation modeling, which can significantly reduce the error introduced by image deformation, making trajectory estimation more stable and less drifting.

[0114] The technical advantages of this application compared to the prior art are as follows: 1. The alternating iterative mechanism of temporal coarse alignment and spatiotemporal joint optimization brings stable convergence and higher accuracy. In traditional one-time coupled optimization, time error is strongly coupled with spatial extrinsic parameters and inertial bias, which is easily amplified, leading to sensitivity to initial values ​​and local optima. This application alternates between the two stages of time and space in the outer iteration. First, a low-cost step is used to compress the time error to a controllable range, then spatial fine-tuning is performed on a fixed time axis, followed by a small time adjustment. This significantly reduces the dependence on initial values, reduces the probability of divergence, and improves the final accuracy of extrinsic parameters and time parameters. Under the same data conditions, usability and stability are significantly improved, the failure rate is reduced, the number of iterations is reduced, and the reprojection error is smaller.

[0115] 2. The unified time model for constant offset and drift compensation brings improved long-term operational reliability and drift resistance. Existing methods using only fixed time offsets cannot cover dynamic errors caused by clock drift, communication jitter, and operating temperature variations. After prolonged operation, these errors are passively compensated for by external parameters. This application simultaneously represents both the fixed offset and the time-varying drift in the time model, and allows for segmented or channel-specific settings as needed, making time errors an explicitly estimable and correctable variable. During long-term or multi-scenario operation, the time axis maintains consistency, preventing external parameters from being "dragged out"; downstream systems (such as VIO / SLAM) exhibit improved trajectory stability and consistency, extending maintenance cycles.

[0116] 3. Anomaly detection and data quality screening mechanisms improve engineering usability and consistency. Real-world data often contains anomalies such as frame drops, timestamp jumps, overexposure / underexposure, and IMU saturation. Traditional workflows lack systematic handling of these anomalies, leading to residual bursts or optimization oscillations. This application performs quality screening before time estimation and spatial optimization, downweighting or removing abnormal segments, and employing different strategies to distinguish between persistent drift and sudden anomalies. The calibration process is no longer vulnerable to environmental and acquisition conditions, and both batch production and outdoor data can converge stably; the results of multiple repeated acquisitions show better consistency, reducing rework and parameter tuning costs.

[0117] 4. Separate modeling of rolling shutter speed and overall time offset brings advantages in accuracy for high-speed scenes. Existing technologies conflate line scan delay with full-frame time difference, leading to parameter cancellation during high-speed rotation or rapid translation, causing deviations in extrinsic time estimation. This application uses frame-level alignment for cross-modal synchronization, uses line-level readout only for image geometric correction, and processes both types of time factors hierarchically in the residual construction. Under conditions such as high-speed motion, strong vibration, and rapid scanning, image distortion is accurately compensated, and time estimation is not interfered with by rolling shutter speed; extrinsic time parameters remain stable and accurate even in extreme motion.

[0118] 5. A unified time structure for multiple cameras / inertial units (multiple sensors) brings system consistency and scalability. In existing technologies, multi-view vision or multi-IMU systems, if calibrated pair by pair, are prone to accumulated or inconsistent time errors, affecting subsequent fusion and mapping. This application uses the master clock as a reference, sets independent relative offsets for each channel, and supports automatic detection of trigger differences; all channels are jointly evaluated and corrected within the same time frame. When fusing multiple cameras, multiple IMUs, and multiple sensors, time consistency is significantly improved; cross-device and cross-batch deployment is smoother, reducing the cost of rework for single channels and multiple calibrations.

[0119] 6. Automated convergence judgment and quality evaluation enhance process certainty and deliverability. Traditional processes lack clear stop and pass / fail criteria, often relying on engineers' experience, leading to uncontrollable cycles and difficulty in comparing results. This application monitors real-time changes in time parameters, external parameters, residual stability, and parameter uncertainty during iteration, automatically terminating and generating a standardized report upon reaching thresholds. The calibration process has definite termination conditions and verifiable quality evidence, facilitating production acceptance, after-sales comparison, and continuous improvement; results between teams and batches are comparable and traceable.

[0120] 7. The interaction of techniques such as alternating iteration, anomaly robustness, and automatic shutdown reduces computational and operational costs while ensuring accuracy. Existing technologies involve one-time fully coupled optimization, which requires repeated attempts when initial values ​​are poor. Abnormal data slows down iteration and triggers manual intervention, while unclear shutdown criteria lead to unnecessary computation. This application achieves rapid convergence to the solvable domain through a phased strategy prioritizing time and then space optimization; anomaly screening reduces invalid iterations; and automatic shutdown avoids over-solving. Overall computational load is reduced, and tuning and manual intervention are minimized. Under the same hardware resources and data scale, the time and manpower costs required to complete a reliable calibration are significantly reduced.

[0121] 8. This application's coarse time alignment, unified time model, anomaly robustness, and downtime assessment all adopt a modular approach to interface with mainstream calibration and VIO tools; inputs and outputs follow common data formats and reporting standards. Calibration capabilities can be upgraded without changing hardware or the main workflow, lowering the implementation threshold and shortening the path from R&D to mass production.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A method for joint spatiotemporal calibration of block coordinate descent, characterized in that, include: Acquire camera image sequences and inertial measurement unit (IMU) data, wherein the camera image sequence includes image frames and their timestamps, and the inertial measurement unit (IMU) data includes angular velocity and angular acceleration; The first stage of coarse temporal alignment is performed, and the temporal mapping parameters between the camera and the inertial measurement unit (IMU) are estimated based on the camera image sequence and the IMU data. The temporal mapping parameters include time offset and time drift parameters. The second stage of spatiotemporal joint optimization is performed. Under the constraints of the time mapping parameters, the spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) data are jointly optimized. The joint optimization is based on the image reprojection residual, the IMU pre-integration residual, and the temporal residual to construct the objective function. Alternately iterate through the first stage of temporal coarse alignment and the second stage of spatiotemporal joint optimization until the preset convergence condition is met; The calibration results are output, including spatial extrinsic parameters, time mapping parameters, noise model, and calibration quality report.

2. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The execution of the first phase of coarse time alignment includes: Visual motion features are extracted from the camera image sequence to construct a visual angular velocity sequence; Angular velocity features are extracted from the inertial measurement unit (IMU) data, and the IMU angular velocity sequence is filtered. By using a normalized cross-correlation coarse search, the correlation between the visual angular velocity sequence and the filtered angular velocity sequence of the inertial measurement unit (IMU) is calculated within a preset time offset search range to estimate the initial time offset.

3. The block coordinate descent joint spatiotemporal calibration method according to claim 2, characterized in that, The time drift parameter is estimated using a linear time model; The time model is: in, Indicates the time drift parameter, Indicates time offset.

4. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The objective function for the spatiotemporal joint optimization is: in, For robust loss function; , , These are the weighting coefficients; For image reprojection residuals; For the pre-integrated residual of the inertial measurement unit (IMU); For time residuals; This refers to the residual value of the rolling shutter.

5. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The method also includes an anomaly data processing step: Detecting anomalous data during temporal coarse alignment and / or spatiotemporal joint optimization; Abnormal data segments are weighted down or segmented for alignment to filter data quality.

6. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The method also includes a rolling shutter separation modeling step: For rolling shutter cameras, the exposure time of each line of images is modeled, and the exposure time is calculated based on the frame reference time and the total line scan time; In the spatiotemporal joint optimization, the rolling shutter residual is used as part of the objective function, wherein the rolling shutter residual is calculated based on row-level time mapping and image feature point reprojection error.

7. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The convergence condition includes at least one of the following: The change in time mapping parameters is less than the threshold; The variation range of spatial extrinsic parameters is less than the threshold. Image reprojection error and / or IMU pre-integration residual stabilize within a preset range; The maximum number of iterations has been reached.

8. The block coordinate descent joint spatiotemporal calibration method according to claim 1, characterized in that, The method also supports multi-sensor calibration, including: Set a unified master timeline and set independent time mapping parameters for each camera or inertial measurement unit (IMU); In the spatiotemporal joint optimization, consistent time mapping and joint evaluation are performed on multi-sensor data.

9. A block coordinate descent joint spatiotemporal calibration system, executing the block coordinate descent joint spatiotemporal calibration method as described in any one of claims 1 to 8, characterized in that, include: The data access module is used to acquire camera image sequences and inertial measurement unit (IMU) data; The coarse time alignment module is used to estimate the time mapping parameters between the camera and the inertial measurement unit (IMU). The spatiotemporal joint optimization module is used to optimize spatial extrinsic parameters, camera intrinsic parameters, and inertial measurement unit (IMU) data under time mapping parameter constraints. The alternating iteration control module is used to control the alternating iteration of temporal coarse alignment and spatiotemporal joint optimization, and to determine convergence. The results output module is used to evaluate the calibration quality and output the calibration results, which include spatial extrinsic parameters, time mapping parameters, noise model, and calibration quality report.

10. The block coordinate descent joint spatiotemporal calibration system according to claim 9, characterized in that, The system also includes a rolling shutter modeling module, which is used to correct the exposure time of each line based on the image line number and readout time.

Images