A binocular camera-imu system joint online calibration method based on extended kalman filter
The online calibration method for binocular camera-IMU systems using extended Kalman filtering solves the problems of installation difficulties and insufficient offline calibration accuracy of traditional contact monitoring methods, and realizes high-precision, real-time three-dimensional dynamic monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-12
AI Technical Summary
Traditional contact-based 3D dynamic monitoring methods are inconvenient to install and have high maintenance costs in complex scenarios. The offline calibration methods of existing binocular camera-IMU systems cannot adapt to real-time dynamic changes in parameters, resulting in a decrease in monitoring accuracy.
A joint online calibration method for a binocular camera-IMU system based on extended Kalman filtering is adopted. Initial parameters are obtained through offline calibration, and the state prediction, update and accuracy evaluation modules of the extended Kalman filtering algorithm are combined to correct sensor parameter drift in real time, so as to achieve high-precision and real-time calibration parameter adjustment.
It improves the ease of installation, continuity of monitoring, and calibration accuracy of three-dimensional dynamic monitoring, adapts to complex scenarios, reduces equipment costs, and ensures real-time and accurate monitoring.
Smart Images
Figure CN122192291A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multi-sensor fusion technology, and in particular relates to a joint online calibration method for a camera-inertial measurement unit (IMU) system based on extended Kalman filtering. Background Technology
[0002] The online calibration method for a binocular camera-inertial measurement unit system based on extended Kalman filtering is a key approach to supporting non-contact 3D dynamic monitoring technology. 3D dynamic monitoring technology can monitor and analyze the dynamic behaviors of various targets in real time, including displacement, deformation, and vibration, under the influence of external environment and during their own operation. It possesses strong environmental adaptability and is suitable for various complex scenarios such as structural health monitoring, equipment operation monitoring, and seismic response monitoring. Non-contact 3D dynamic monitoring, due to its advantages of high monitoring accuracy, no contact interference, simple equipment, and wide applicability, has become an important means of assessing the health status of target structures and ensuring their safety and reliability, with broad application prospects in civil engineering, mechanical engineering, disaster prevention and mitigation, and other fields.
[0003] Traditional three-dimensional dynamic monitoring methods are primarily based on contact monitoring, which involves directly mounting sensors. The core idea is to directly install sensors such as accelerometers, GPS devices, and linear variable differential transformers on the target. These sensors directly collect dynamic monitoring data of the target and then analyze its three-dimensional dynamic behavior. In this process, the installation method, measurement accuracy, and number of sensors have a decisive impact on the monitoring results. Traditional methods typically require deploying a large number of different types of sensors on the target, increasing equipment costs and installation difficulty. Furthermore, the direct contact between the sensors and the target makes installation difficult in high-altitude, complex structures, or precision equipment locations. In addition, contact sensors have high maintenance costs and are prone to failure with long-term use, affecting the continuity of monitoring.
[0004] To overcome the limitations of traditional contact-based monitoring methods, non-contact 3D dynamic monitoring systems based on vision sensors have been proposed and rapidly developed. With advancements in hardware and image processing technology, vision sensors can measure target structure displacement in a low-cost, high-precision manner. The core idea is to estimate the target's horizontal or vertical displacement by matching feature points between real-time and initial images. When multiple feature points are used for matching, homography transformation between images can be calculated, yielding more accurate target translation and rotation parameters. However, existing vision sensor-based monitoring methods still have several limitations: monocular cameras are limited to horizontal and vertical displacement monitoring, and cannot directly obtain 3D displacement parameters, requiring additional methods to measure the distance between the camera sensor and the target, increasing the complexity of the monitoring process; secondly, the camera sensor itself may experience slight displacement during monitoring, affecting the accuracy of feature point matching between preceding and following images, thus introducing additional measurement errors and reducing the reliability of 3D dynamic monitoring results.
[0005] To address the limitations of the aforementioned visual monitoring methods, a dual-sensor system consisting of a binocular camera and an IMU has been introduced into the field of 3D dynamic monitoring. The binocular camera acquires depth information through extrinsic parameter calibration, solving the problem of obtaining 3D displacement parameters; the IMU measures system acceleration and angular acceleration to determine its own displacement, correcting measurement errors caused by minute camera displacements. The complementary data from both systems provides more accurate and robust pose estimation, improving the precision of 3D dynamic monitoring. However, the prerequisite for efficient collaboration in a dual-sensor system is accurate joint calibration. Due to differences in sampling frequencies between the IMU and camera, and the need for translational and rotational transformations of the coordinate systems, joint calibration is necessary to determine the spatial pose transformation relationship and temporal synchronization deviation between the two, ensuring data consistency and measurement accuracy. Currently, existing systems primarily employ offline calibration methods. While existing offline calibration methods are simple to operate, they have significant limitations in actual 3D dynamic monitoring scenarios: due to factors such as differences in sensor sampling frequency, environmental noise, and equipment vibration, the time offset between the camera and the IMU changes in real time. Offline calibration treats relevant parameters as fixed values, which cannot correct dynamic deviations in real time, easily causing calibration errors that affect monitoring accuracy. It also cannot correct parameter drift during system operation, and long-term use will lead to a decrease in system detection accuracy, making it difficult to meet the long-term real-time monitoring requirements.
[0006] In summary, traditional contact-based 3D dynamic monitoring methods rely on the direct installation of various sensors, which is difficult to implement conveniently in complex scenarios and suffers from high maintenance costs and poor monitoring continuity. While binocular camera-IMU dual-sensor systems can overcome the shortcomings of contact-based 3D dynamic monitoring methods, existing offline calibration methods cannot adapt to real-time dynamic parameter changes. This invention aims to propose a joint online calibration method for binocular camera-IMU systems based on extended Kalman filtering. This method can achieve high-precision, real-time joint calibration adapted to 3D dynamic monitoring scenarios, correct errors caused by dynamic parameter changes, provide reliable parameter support for non-contact 3D dynamic monitoring, and fill the gaps in existing technologies. Summary of the Invention
[0007] The purpose of this invention is to provide a joint online calibration method for a binocular camera-IMU system based on extended Kalman filtering, so as to overcome the problems of inconvenient equipment installation, high maintenance costs, and inability to dynamically correct changes in calibration parameters in traditional technologies.
[0008] This invention is achieved through the following technical solution:
[0009] A joint online calibration method for a stereo camera-IMU system based on extended Kalman filtering includes the following steps:
[0010] Step 1: For the three-dimensional dynamic monitoring scenario, build a dual-sensor monitoring system consisting of a binocular camera and an inertial measurement unit, and obtain the initial calibration parameters of the system as the initial benchmark for joint online calibration.
[0011] The initial calibration parameters were obtained through an offline calibration method, including the internal parameters of the binocular camera, the relative pose parameters between the binocular cameras, the random error parameters of the inertial measurement unit, and the initial relative pose parameters and initial time synchronization deviation between the binocular camera and the inertial measurement unit.
[0012] The offline calibration methods are specifically divided into three categories: offline calibration of the binocular camera alone, pre-calibration of the inertial measurement unit alone, and joint offline calibration of the two sensors. The three types of offline calibration methods work together to provide a high-precision initial benchmark for online calibration, effectively improving the convergence speed of online calibration and reducing the initial error.
[0013] The internal parameters of the binocular camera include focal length, optical center coordinates, and distortion coefficients. The relative pose parameters between the binocular cameras include rotation matrix and translation vector. The random error parameters of the inertial measurement unit include Gaussian white noise and random walk coefficients. The initial relative pose parameters between the binocular camera and the inertial measurement unit include rotation quaternions and translation vectors for coordinate system transformation. The above initial calibration parameters need to be validated and abnormal parameters are removed before being used as the initial input for online calibration to ensure the reliability of the initial reference.
[0014] Step 2: Perform the necessary initialization settings and data preprocessing for the dual-sensor system. The purpose is to eliminate the impact of invalid data and interference factors on the subsequent calibration accuracy, and to provide support for the stable operation of the extended Kalman filter algorithm.
[0015] Initialization settings mainly involve determining the core parameters of the extended Kalman filter algorithm, covering the state vector, observations, initial covariance matrix, process noise covariance matrix, and observation noise covariance matrix. Each covariance matrix parameter needs to be initially set based on the noise characteristics of the 3D dynamic monitoring scene. Specifically, the state vector is...
[0016] (1)
[0017] (2)
[0018] in These are the motion state parameters of the inertial measurement unit at time t. This is the quaternion used for the transformation from the world coordinate system to the inertial measurement unit coordinate system. and These are the position and velocity vectors of the inertial measurement unit in the world coordinate system, respectively. and These are the angular velocity zero bias and acceleration zero bias of the inertial measurement unit, respectively. , These are the transformation quaternions and translation vectors from the inertial measurement unit to the binocular camera coordinate system, respectively. This refers to the time synchronization deviation between the binocular camera and the inertial measurement unit.
[0019] The data preprocessing process involves simultaneously acquiring binocular camera image data and IMU inertial measurement data of a 3D dynamic scene. The acquired image data is then subjected to distortion correction and stereo correction to correct camera optical distortion and binocular camera image misalignment, ensuring the accuracy of subsequent feature extraction and matching. Outlier removal and noise filtering are performed on the system inertial measurement data. A sliding window filtering method is used to suppress Gaussian white noise and random interference in the IMU data, ultimately obtaining effective data that meets the requirements of joint online calibration.
[0020] Step 3: Design a joint online calibration model based on extended Kalman filtering. Its structure mainly consists of three core modules: state prediction, state update, and accuracy evaluation. These three modules work together to achieve real-time estimation and optimization of calibration parameters. The specific design is as follows:
[0021] State prediction module: Based on preprocessed inertial measurement data and combined with the system dynamics model, the state variables of the filtering algorithm are predicted and updated to obtain the predicted state values and the predicted covariance matrix. The state variables include the motion state parameters of the inertial measurement unit, the relative position parameters between the stereo camera and the inertial measurement unit, and their time synchronization deviation. The dynamic equations and noise descriptions applied to the system are as follows:
[0022]
[0023]
[0024]
[0025] in This is an estimate of the angular velocity. This is an estimate of the acceleration. Let be the rotation matrix from the world coordinate system to the inertial coordinate system. This is a zero-bias estimate of the angular velocity, used to correct for fixed biases present in IMU angular velocity measurements and reduce measurement errors. This is a zero-bias estimate of acceleration, used to correct for fixed biases present in IMU acceleration measurements and reduce acceleration measurement errors. and n a These are Gaussian white noise present during angular velocity and acceleration measurements, respectively. and These are the random walk noise measurements for zero angular velocity bias and zero acceleration bias, respectively.
[0026] State update module: Based on preprocessed image data, it acquires observation data through feature extraction, matching, tracking, and 3D reconstruction methods. Combined with the camera imaging model, it corrects and updates the state prediction values, optimizes the state estimates and covariance matrix, and focuses on correcting the relative pose parameters and time synchronization deviation between the stereo camera and the inertial measurement unit. Feature extraction uses the ORB algorithm to achieve fast extraction and robust description of image feature points. Feature matching uses the K-nearest neighbor matching algorithm combined with the RANSAC algorithm to eliminate incorrect matching pairs. 3D reconstruction obtains the 3D coordinates of the target point through triangulation, which are used as the input of the observation data.
[0027] Accuracy Evaluation Module: The module uses preset error evaluation indicators to determine the accuracy of the state estimate, providing a basis for calibration convergence judgment. The error evaluation indicators include reprojection error, which reflects the goodness of fit of the calibration model, and variance analysis indicators, which analyze the noise characteristics of the inertial measurement unit.
[0028] The reprojection error is used to measure the goodness of fit of the calibration model to the sensor geometry. The smaller the error, the more accurate the calibration parameters are. The variance analysis index uses Allen variance to distinguish the different components of the inertial measurement unit noise (Gaussian white noise and random walk coefficients), clarify the noise source, and provide direction for subsequent parameter adjustments.
[0029] Step 4: Run the joint online calibration model. The specific method is as follows:
[0030] First, the preprocessed valid data is input into the calibration model. It then sequentially passes through the state prediction, state update, and accuracy evaluation modules to complete one round of state prediction, correction, and accuracy determination, obtaining estimated values of the joint calibration parameters of the binocular camera and inertial measurement unit. Next, using an error calculation method, the deviation between the estimated calibration parameter values and the true values is compared. This error calculation method employs the average endpoint error, combined with error indicators (reprojection error, Allen variance) output by the accuracy evaluation module, to comprehensively determine whether the accuracy of the current calibration result meets the requirements of 3D dynamic monitoring. Finally, based on the error calculation results, the core parameters of the filtering algorithm and related model parameters are adjusted to reduce calibration errors.
[0031] Step 5: Repeat the iterative optimization process of Step 4 until the error value of the calibration model converges below the preset threshold, or the number of iterations reaches the preset upper limit, at which point the iteration stops. The preset threshold is set according to the accuracy requirements of 3D dynamic monitoring, and the upper limit of the number of iterations is reasonably configured according to the real-time requirements of the monitoring scenario to avoid excessive iteration affecting the real-time performance of online calibration. After iteration stops, the optimized model parameters are stored as the benchmark parameters for subsequent system operations to ensure the consistency and accuracy of subsequent real-time calibrations.
[0032] Step Six: During the actual 3D dynamic monitoring process, load the model parameters and final calibration parameters saved in Step Five, collect new 3D dynamic scene image data and inertial measurement data in real time, perform the same preprocessing as in Step Two on the collected new data, input it into the joint online calibration model, and output real-time calibration parameters to provide accuracy assurance for displacement and deformation measurement in 3D dynamic monitoring.
[0033] Beneficial effects
[0034] The present invention has the following advantages: it adopts a binocular camera-IMU system joint online calibration method based on extended Kalman filtering, which has significant advantages in terms of installation convenience, monitoring continuity and cost control compared with the traditional contact-type three-dimensional dynamic monitoring calibration method. Compared with existing offline calibration methods, single vision monitoring calibration methods and conventional online calibration methods, it achieves a comprehensive improvement in calibration accuracy, real-time performance and scene adaptability. Attached Figure Description
[0035] Figure 1 This is an overall flowchart of the online calibration method for a binocular camera-IMU system based on extended Kalman filtering according to the present invention. The core processes of steps one to six are marked in the figure.
[0036] Figure 2 This invention provides a detailed flowchart of the state update and prediction process based on Extended Kalman Filter (EKF). It clearly presents the complete workflow of the state prediction module and the state update module in the EKF joint online calibration model, and clarifies the core operation logic and data input-output relationship of the two modules. Detailed Implementation
[0037] To more clearly illustrate the online joint calibration method for a binocular camera-IMU system based on extended Kalman filtering of the present invention, the specific implementation of the present invention will be described in detail below with reference to embodiments. The method of the present invention is mainly applied to three-dimensional dynamic monitoring scenarios of buildings, realizing high-precision online joint calibration of a binocular camera and an inertial measurement unit system, providing parameter support for monitoring building displacement and deformation.
[0038] This embodiment employs a joint online calibration model based on extended Kalman filtering. Using the ROS system, the Python open-source framework, and the OpenCV library, the model is deployed to a host computer. The calibration process is completed using a stereo camera and an IMU sensor, ensuring both calibration accuracy and real-time performance.
[0039] Step 1: Build a binocular camera-IMU dual-sensor monitoring system, collect offline data and obtain initial calibration parameters.
[0040] An industrial binocular camera with a resolution of no less than 1920×1080 and a frame rate of more than 30fps is selected, and a six-axis IMU sensor with a sampling frequency of no less than 100Hz is used to combine them. The sensor system is fixed in front of the building monitoring point at a location where the building image can be completely captured. The IMU and the binocular camera are rigidly connected to avoid relative displacement. The two sensors are connected to the host computer through a data transmission line to ensure that the sampling frequency is matched and the data is transmitted synchronously.
[0041] Three offline calibration methods were used to collaboratively acquire initial parameters: the binocular camera was calibrated using the Zhang Zhengyou calibration method, taking at least 12 chessboard images from different angles, and the camera was calibrated offline to acquire internal parameters and relative pose between the two eyes; the IMU was calibrated statically using Allen's analysis of variance to acquire random error parameters; and the dual sensors collected synchronous rotation data to acquire the initial relative pose and time synchronization deviation, which were saved as the initial reference after validity verification.
[0042] Step 2: Initialize the dual-sensor system and preprocess the collected data.
[0043] Initialization settings: An extended Kalman filter algorithm runtime environment is set up on the host computer. Core parameters are set, and the observation values are selected as the coordinates of feature points from binocular 3D reconstruction. The initial covariance matrix, process noise covariance matrix, and observation noise covariance matrix are all set as diagonal matrices. The state vector adopts... Set the initial state vector.
[0044] Data preprocessing: Dual sensors were activated to acquire building scene data. Image data was distorted using OpenCV's undistort method and stereorectified using the stereoRectify method. IMU data was filtered using the 3σ criterion to remove outliers and a sliding window filter to suppress noise. The preprocessed data was aligned by timestamps and saved as a txt file as model input.
[0045] Step 3: Deploy the joint online calibration model based on extended Kalman filtering on the host computer to complete the construction of the three core modules.
[0046] The model comprises three modules: state prediction, state update, and accuracy evaluation. The state prediction module calculates the predicted state value and covariance matrix based on IMU data and dynamic equations. The state update module uses the ORB algorithm to extract image features, triangulation to acquire observation data, and combines the camera imaging model to correct the state value. The accuracy evaluation module uses reprojection error and Allen variance to determine the calibration accuracy.
[0047] Step 4: Input the preprocessed valid data, start the calibration model, and perform iterative optimization operations.
[0048] The model sequentially calls the three main modules to complete one iteration, obtains the estimated values of the calibration parameters, and calculates the pose of the stereo camera in the world coordinate system. The deviation between the estimated value and the true value is calculated using the average endpoint error, and the result is judged in combination with the accuracy index. If the preset accuracy is not achieved, the backpropagation optimization algorithm is used to adjust the parameters of the extended Kalman filter covariance matrix and minimize the calibration error.
[0049] Step 5: Set the iterative convergence condition, repeat Step 4, and complete the model parameter optimization and saving. When the convergence condition is met or the iteration limit is reached, stop the iteration, save the optimized model parameters and the final calibration parameters as the benchmark for subsequent real-time calibration.
[0050] Step 6: Load the reference parameters and carry out real-time calibration applications to support three-dimensional dynamic monitoring of buildings.
[0051] New scene data is collected and preprocessed, then input into the calibration model to obtain real-time calibration parameters, continuously correcting parameter drift. The real-time parameters are input into the monitoring module, and combined with binocular reconstruction and IMU measurement technology, real-time high-precision monitoring of building displacement and deformation is achieved.
[0052] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A joint online calibration method for a binocular camera-IMU system based on extended Kalman filtering, characterized in that, Includes the following steps: Step 1: For the 3D dynamic scene, build a dual-sensor monitoring system consisting of a binocular camera and an inertial measurement unit, and obtain the initial calibration parameters of the system as the initial reference for joint online calibration; The initial calibration parameters were obtained through an offline calibration method, including the internal parameters of the binocular camera, the relative pose parameters between the binocular cameras, the random error parameters of the inertial measurement unit, and the initial relative pose parameters and initial time synchronization deviation between the binocular camera and the inertial measurement unit. Step 2: Perform the necessary initialization settings and data preprocessing for the dual-sensor system; Initialization settings include determining the core parameters of the filtering algorithm, covering the state vector, observations, initial covariance matrix, process noise covariance matrix, and observation noise covariance matrix; Data preprocessing includes synchronizing the image data of the three-dimensional dynamic scene with the system inertial measurement data. The image data is subjected to distortion correction and stereo correction, and the system inertial measurement data is subjected to outlier removal and noise filtering to obtain valid data that meets the calibration requirements. Step 3: Design a joint online calibration model based on extended Kalman filtering, whose structure mainly consists of three modules: State prediction module: Based on the preprocessed inertial measurement data and combined with the system dynamics model, the state variables of the filtering algorithm are predicted and updated to obtain the state prediction value and the prediction covariance matrix. The state variables include the motion state parameters of the inertial measurement unit, the relative position parameters between the binocular camera and the inertial measurement unit, and the time synchronization deviation between the two. State update module: Based on preprocessed image data, it obtains observation data through feature extraction, matching, tracking and 3D reconstruction methods, and combines the camera imaging model to correct and update the state prediction value, optimize the state estimate value and covariance matrix, and focus on correcting the relative pose parameters and time synchronization deviation between the binocular camera and the inertial measurement unit. Accuracy Evaluation Module: The module uses preset error evaluation indicators to determine the accuracy of the state estimate, providing a basis for calibration convergence judgment. The error evaluation indicators include reprojection error, which reflects the goodness of fit of the calibration model, and variance analysis indicators, which analyze the noise characteristics of the inertial measurement unit. Step 4: Run the joint online calibration model. The specific method is as follows: First, the preprocessed valid data is input into the calibration model, and then passed through the state prediction, state update and accuracy evaluation modules in sequence to obtain the estimated values of the joint calibration parameters of the binocular camera and the inertial measurement unit. Then, by using error calculation methods, the deviation between the estimated values and the true values of the calibration parameters is compared, and the accuracy of the calibration results is determined by combining the accuracy evaluation indicators. Then, based on the error calculation results, the core parameters of the filtering algorithm and related parameters of the model are adjusted to reduce the calibration error; Step 5: Repeat step 4 until the error value of the calibration model converges to below the preset threshold, or the number of iterations reaches the preset upper limit; Finally, the optimized model parameters and the final calibration parameters are saved as the reference parameters for the system in the future. Step Six: During the actual 3D dynamic monitoring process, load the model parameters and final calibration parameters saved in Step Five, collect 3D dynamic scene image data and inertial measurement data in real time, perform the same preprocessing as in Step Two, input the data into the joint online calibration model, and output real-time calibration parameters to provide accuracy assurance for displacement and deformation measurement in 3D dynamic monitoring.
2. The method for joint online calibration of a binocular camera-IMU system based on extended Kalman filtering as described in claim 1, characterized in that, In step one, the offline calibration methods are specifically divided into three categories: offline calibration of the binocular camera alone, pre-calibration of the inertial measurement unit alone, and joint offline calibration of the two sensors. The three types of offline calibration methods work together to provide an initial benchmark for online calibration, improve the convergence speed of online calibration, and reduce the initial error.
3. The method for joint online calibration of a binocular camera-IMU system based on extended Kalman filtering as described in claim 1, characterized in that, In step one, the internal parameters of the binocular camera include focal length, optical center coordinates, and distortion coefficients; the relative pose parameters between the binocular cameras include rotation matrix and translation vector; the random error parameters of the inertial measurement unit include Gaussian white noise and random walk coefficients; and the initial relative pose parameters between the binocular camera and the inertial measurement unit include rotation quaternions and translation vectors resulting from the coordinate system transformation between the two.
4. The method for joint online calibration of a binocular camera-IMU system based on extended Kalman filtering as described in claim 1, characterized in that, In step two, the state vector is specifically... in These are the motion state parameters of the inertial measurement unit at time t. This is the quaternion used for the transformation from the world coordinate system to the inertial measurement unit coordinate system. and These are the position and velocity vectors of the inertial measurement unit in the world coordinate system, respectively. and These are the angular velocity zero bias and acceleration zero bias of the inertial measurement unit, respectively. , These are the transformation quaternions and translation vectors from the inertial measurement unit to the binocular camera coordinate system, respectively. This refers to the time synchronization deviation between the binocular camera and the inertial measurement unit.
5. The method for joint online calibration of a binocular camera-IMU system based on extended Kalman filtering as described in claim 1, characterized in that, In step three, the reprojection error is used to measure the goodness of fit of the calibration model to the sensor geometry. The smaller the error, the more accurate the calibration parameters are. The variance analysis index uses Allen's variance to distinguish the different components of the inertial measurement unit noise.
6. The method for joint online calibration of a binocular camera-IMU system based on extended Kalman filtering as described in claim 1, characterized in that, In step four, the error calculation method uses the average endpoint error to accurately calculate the deviation between the estimated and true values of the calibration parameters. The parameter adjustment uses a backpropagation-type optimization algorithm to dynamically adjust the covariance matrix parameters of the filtering algorithm to achieve adaptive minimization of the calibration error.