Collaborative correction method and system for visual-inertial odometry for rolling shutter cameras

Abstract

This invention relates to the field of visual navigation technology, and discloses a collaborative correction method and system for visual inertial odometry (IMU) for rolling shutter cameras, to solve the technical problem of poor positioning accuracy. The method includes: acquiring image data and IMU measurement data; calculating rotation compensation relationships based on the IMU measurement data, and performing rotation compensation on the image data of the current frame to obtain actual observed data of feature points; performing pre-integration on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result and construct an IMU pre-integration error term; acquiring the true camera pose corresponding to each feature point in the image data of the current frame, calculating the equivalent camera pose based on the rotation compensation relationship and acquiring theoretical observed data of feature points, constructing a visual reprojection error based on the actual and theoretical observed data of feature points; and performing joint optimization based on the IMU pre-integration error and the visual reprojection error to obtain the camera pose correction result.

Description

Technical Field

[0001] This invention relates to the field of visual navigation technology, and more particularly to a method, system, electronic device, computer storage medium, and computer program product for co-calibrating a visual inertial odometry for a rolling shutter camera. Background Technology

[0002] Visual-Inertial Odometry (VIO) is one of the core technologies for positioning and navigation on mobile platforms. By fusing camera image information with IMU inertial data, it achieves high-frequency and stable motion state estimation, enabling devices to output their pose information in real time even without external positioning. It is currently widely used in intelligent devices such as drones, AR / VR glasses, and robots.

[0003] In practical applications, due to cost and hardware limitations, most devices requiring visual navigation employ rolling shutter cameras. Unlike global shutter cameras, which expose the entire frame at a single moment, these cameras use line-by-line exposure, with different rows of pixels corresponding to different exposure times. When the device moves, due to the line-by-line exposure characteristic, the observations of each pixel in the image do not originate from the same camera pose, but rather correspond to different states at different moments during the camera's continuous movement. If the generated image pose is considered as the camera pose at a single moment, and visual-inertial odometry (VIO) is used for positioning and navigation, a large systematic error will be introduced, severely affecting the accuracy of positioning and navigation.

[0004] To address the aforementioned technical issues, existing technologies typically employ compensation and optimization at both the visual front-end and back-end models. However, current optimization methods struggle to fundamentally resolve the systematic errors caused by rolling shutter cameras and may also lead to model fragmentation and insufficient error coupling, failing to meet the requirements for positioning accuracy and robustness in fast-moving scenarios. Therefore, a method for finely calibrating the visual inertial odometry (VIO) of rolling shutter cameras is urgently needed to improve the accuracy and robustness of the VIOS system in high-speed motion scenarios, thereby enhancing positioning precision. Summary of the Invention

[0005] The main objective of this invention is to solve the technical problem in the prior art where positioning based on visual images acquired by a rolling shutter camera using a visual inertial odometer results in poor positioning accuracy because the poses of different pixels in the same image are not completely identical, and the visual inertial odometer also has systematic errors.

[0006] The first aspect of this invention provides a method for co-calibrating a visual inertial odometry for a rolling shutter camera, comprising: Acquire the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when the image data is acquired; The rotation compensation relationship is calculated based on the IMU measurement data, and the rotation compensation is performed on the image data of the current frame to obtain the actual observed data of the feature points of the image data. Pre-integration is performed on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and an IMU pre-integration error term is constructed based on the pre-integration result. Obtain the real camera pose corresponding to each feature point in the image data of the current frame, and calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship; The theoretical observation data of feature points are calculated based on the equivalent camera pose, and the visual reprojection error is constructed based on the actual observation data of the feature points and the theoretical observation data of the feature points. Based on the IMU pre-integration error and the visual reprojection error, joint optimization is performed. When the joint optimization objective is achieved, the camera pose correction result corresponding to the current frame reference time is obtained.

[0007] Optionally, in a first implementation of the first aspect of the present invention, before calculating the rotation compensation relationship based on the IMU measurement data, the method further includes: The image data is subjected to distortion correction and normalized planar projection based on the camera intrinsic parameters and distortion coefficients of the rolling shutter camera. After obtaining the actual observed data of feature points in the image data, the method further includes: Feature matching and tracking are performed on the actual observation data of feature points in the image data of different frames, and after removing outliers, the optimized actual observation information of feature points in the current frame is obtained.

[0008] Optionally, in a second implementation of the first aspect of the present invention, the IMU measurement data includes at least angular velocity data; The step of calculating the rotation compensation relationship based on the IMU measurement data and performing rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data includes: By selecting a reference time, the angular velocity data of each pixel row of the image data between the actual exposure time and the reference time are integrated to obtain a rotation compensation matrix that describes the rotation compensation relationship. Based on the rotation compensation matrix, rotation compensation is performed on each pixel row of the image data to obtain the actual observed data of feature points unified to the reference time.

[0009] Optionally, in a third implementation of the first aspect of the present invention, obtaining the true camera pose corresponding to each feature point in the image data of the current frame, and calculating the equivalent camera pose corresponding to the actual observed data of the feature points based on the rotation compensation relationship includes: Based on the IMU measurement data and the line exposure time of the rolling shutter camera, the true camera pose corresponding to each feature point in the image data is calculated; Based on the actual camera pose and the rotation compensation relationship at the reference time, the equivalent camera pose corresponding to each feature point in the image data is calculated; wherein the equivalent camera pose and the actual observation data of the feature points are aligned in time.

[0010] Optionally, in a fourth implementation of the first aspect of the present invention, the step of calculating theoretical observation data of feature points based on the equivalent camera pose, and constructing visual reprojection error based on the actual observation data of the feature points and the theoretical observation data of the feature points, includes: For the original spatial position data of three-dimensional feature points in space, the original spatial position data is transformed into the camera coordinate system by the equivalent camera pose to obtain the camera coordinate position data. The camera projection function is called to project the camera coordinate position data onto the normalized plane to obtain the theoretical observation data of the feature points; The visual reprojection error is obtained by subtracting the theoretical observation data of the feature points from the actual observation data of the feature points.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, the joint optimization based on the IMU pre-integration error and the visual reprojection error, and obtaining the camera pose correction result corresponding to the current frame reference time when the joint optimization objective is achieved, includes: Based on the IMU pre-integration error and the visual reprojection error, an overall objective function is constructed and jointly solved to optimize the image state data. The image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. When the overall objective function value is minimized, the joint optimization objective is achieved, and the camera pose corresponding to the current frame reference time is output to obtain the camera pose correction result.

[0012] Optionally, in a sixth implementation of the first aspect of the present invention, after acquiring the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when acquiring the image data, the method further includes calibrating the time system of the camera and the visual inertial odometry by hardware triggering or software synchronization, so that the image data and the IMU measurement data are clock synchronized and have synchronized timestamps.

[0013] A second aspect of the present invention provides a co-calibration system for a visual inertial odometry for a rolling shutter camera, comprising: The data acquisition module is used to acquire the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when acquiring the image data; The front-end compensation module is used to calculate the rotation compensation relationship based on the IMU measurement data and perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. The IMU compensation module is used to perform pre-integration on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and to construct the IMU pre-integration error term based on the pre-integration result. The backend compensation module is used to obtain the real camera pose corresponding to each feature point in the image data of the current frame, calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship, calculate the theoretical observation data of the feature points based on the equivalent camera pose, and construct the visual reprojection error based on the actual observation data of the feature points and the theoretical observation data of the feature points. The joint optimization module is used to perform joint optimization based on the IMU pre-integration error and the visual reprojection error. When the joint optimization target is achieved, the camera pose correction result corresponding to the current frame reference time is obtained.

[0014] Optionally, in a first implementation of the second aspect of the present invention, the co-calibration system for the visual inertial odometry of the rolling shutter camera further includes a rolling shutter camera module and an inertial measurement module. The rolling shutter camera module is used to acquire multi-frame image data, and the inertial measurement module is used to acquire IMU measurement data; The data acquisition module is used to acquire data from the rolling shutter camera module and the inertial measurement module.

[0015] Optionally, in a first implementation of the second aspect of the present invention, the cooperative calibration system for the visual inertial odometry of the rolling shutter camera further includes a clock synchronization module. The clock synchronization module is used to calibrate the time system of the camera and the visual inertial odometry through hardware triggering or software synchronization, so that the image data and the IMU measurement data are clock synchronized and have synchronized timestamps.

[0016] Optionally, in a second implementation of the second aspect of the present invention, the front-end compensation module includes a distortion correction unit, a feature detection unit, and a feature compensation unit; The feature compensation unit is used to calculate the rotation compensation relationship based on the IMU measurement data, and to perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. The distortion correction unit is used to perform distortion correction processing and normalized planar projection on the image data based on the camera intrinsic parameters and distortion coefficients of the rolling shutter camera. The matching optimization unit is used to perform feature matching and tracking on the actual observation data of feature points in the image data of different frames, and obtain the optimized actual observation information of feature points in the current frame after removing outliers.

[0017] Optionally, in a third implementation of the second aspect of the present invention, the IMU measurement data includes at least angular velocity data; The feature compensation unit of the front-end compensation module is specifically used for: By selecting a reference time, the angular velocity data of each pixel row of the image data between the actual exposure time and the reference time are integrated to obtain a rotation compensation matrix that describes the rotation compensation relationship. Based on the rotation compensation matrix, rotation compensation is performed on each pixel row of the image data to obtain the actual observed data of feature points unified to the reference time.

[0018] Optionally, in a fourth implementation of the second aspect of the present invention, the backend compensation module is specifically used for: Based on the IMU measurement data and the line exposure time of the rolling shutter camera, the true camera pose corresponding to each feature point in the image data is calculated; Based on the actual camera pose and the rotation compensation relationship at the reference time, the equivalent camera pose corresponding to each feature point in the image data is calculated; wherein the equivalent camera pose and the actual observation data of the feature points are aligned in time.

[0019] Optionally, in a fifth implementation of the second aspect of the present invention, the backend compensation module is further configured to: For the original spatial position data of three-dimensional feature points in space, the original spatial position data is transformed into the camera coordinate system by the equivalent camera pose to obtain the camera coordinate position data. The camera projection function is called to project the camera coordinate position data onto the normalized plane to obtain the theoretical observation data of the feature points; The visual reprojection error is obtained by subtracting the theoretical observation data of the feature points from the actual observation data of the feature points.

[0020] Optionally, in a sixth implementation of the second aspect of the present invention, the joint optimization module is specifically used for: Based on the IMU pre-integration error and the visual reprojection error, an overall objective function is constructed and jointly solved to optimize the image state data. The image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. When the overall objective function value is minimized, the joint optimization objective is achieved, and the camera pose corresponding to the current frame reference time is output to obtain the camera pose correction result.

[0021] A third aspect of the present invention provides a co-calibration device for a visual inertial odometry (VIO) of a rolling shutter camera, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the co-calibration device for the VIO of a rolling shutter camera to perform the steps of the aforementioned co-calibration method for the VIO of a rolling shutter camera.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described co-calibration method for a visual inertial odometry for a rolling shutter camera.

[0023] A fifth aspect of the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the co-calibration method for a visual inertial odometry for a rolling shutter camera as described above.

[0024] The technical solution provided by this invention involves acquiring image data of the current frame captured by a rolling shutter camera and IMU measurement data output by a visual inertial odometry system during image data acquisition; calculating rotation compensation relationships based on the IMU measurement data and performing rotation compensation on the image data of the current frame to obtain actual observed data of feature points in the image data; performing pre-integration on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and constructing an IMU pre-integration error term based on the pre-integration result; acquiring the true camera pose corresponding to each feature point in the image data of the current frame, and calculating the equivalent camera pose corresponding to the actual observed data of the feature points based on the rotation compensation relationship; calculating theoretical observed data of the feature points based on the equivalent camera pose, and constructing a visual reprojection error based on the actual observed data and the theoretical observed data of the feature points; and performing joint optimization based on the IMU pre-integration error and the visual reprojection error. When the joint optimization objective is achieved, the camera pose correction result corresponding to the reference time of the current frame is obtained. This method can perform collaborative compensation optimization based on the visual front-end and back-end model calculation stages. Through joint optimization, it can finely correct systematic errors that are prone to occur in multiple stages, thereby improving the positioning accuracy. The system, electronic device, computer-readable storage medium, and computer program product provided by this invention also solve the corresponding technical problems. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the first embodiment of the collaborative calibration method for a visual inertial odometer used in a rolling shutter camera according to the present invention. Figure 2 This is a flowchart illustrating a second embodiment of the collaborative calibration method for a visual inertial odometer used in a rolling shutter camera according to an embodiment of the present invention. Figure 3 This is a schematic diagram of an embodiment of a co-calibration system for a visual inertial odometer used in a rolling shutter camera according to the present invention. Figure 4 This is a schematic diagram of another embodiment of the co-calibration system for a visual inertial odometer used in a rolling shutter camera according to the present invention; Figure 5 This is a schematic diagram of an embodiment of a co-calibration device for a visual inertial odometer used in a rolling shutter camera according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the principle of a computer-readable medium according to an embodiment of the present invention. Detailed Implementation

[0026] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of them will be omitted.

[0027] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0028] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0029] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0032] See Figure 1 The first embodiment of the cooperative calibration method for the visual inertial odometry of a rolling shutter camera in this invention is as follows: It is understood that the executing entity of this invention can be a collaborative calibration system for a visual inertial odometry system used in a rolling shutter camera, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example of the executing entity.

[0033] This embodiment mainly involves two stages of error correction: a front-end feature point rotation correction stage and a back-end continuous time fine correction stage. The positioning accuracy is improved through the coordinated correction of the two stages.

[0034] However, it's important to note that the solution in this embodiment is not a simple superposition of the two correction stages. This is because a simple superposition of the front-end correction and back-end modeling schemes would result in a lack of unified temporal and geometric semantic constraints between the front-end corrected feature point observations and the back-end pose model. This could lead to implicit inconsistencies between observations and state variables, making it difficult for the reprojection residuals to accurately reflect imaging errors, thus affecting optimization convergence and estimation accuracy. This embodiment, by uniformly modeling the front-end feature correction relationship and the back-end equivalent exposure pose within a continuous time frame, ensures that the equivalent observation position of the feature points is strictly consistent with their corresponding equivalent camera pose in a physical sense. This avoids the model fragmentation problem caused by simple module-level combinations, significantly improving the overall optimization stability and effectiveness. This effect cannot be achieved by simply combining front-end correction and back-end modeling schemes individually or in a simple way. The following is a detailed explanation step by step: S101. Acquire the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when acquiring the image data; First, in this embodiment, the server acquires image data of the current frame from the rolling shutter camera, and simultaneously obtains IMU (Inertial Measurement Unit) measurement data output by the Visual-Inertial Odometry (VIO) while acquiring the image data. The IMU measurement data includes at least angular velocity data and acceleration data.

[0035] Furthermore, after acquiring the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when acquiring the image data, the system also includes calibrating the time system of the camera and the visual inertial odometry through hardware triggering or software synchronization, so that the image data and the IMU measurement data are clock synchronized and have synchronized timestamps.

[0036] After acquiring the image data, the process also includes distortion correction and normalized planar projection of the image data based on the camera intrinsic parameters and distortion coefficients of the rolling shutter camera.

[0037] S102. Calculate the rotation compensation relationship based on the IMU measurement data, and perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. Since the rolling shutter camera described in this embodiment uses line-by-line exposure, if the camera rotates during the acquisition of a complete image, the image will be distorted to some extent. Therefore, the image data cannot be simply described by the camera pose at a certain point in time. Based on this, this embodiment calculates the rotation compensation relationship based on IMU measurement data and performs rotation compensation on the image data of the current frame to obtain image data that can be considered as being in the same pose at the same moment. In this embodiment, rotation compensation is performed using feature points contained in the image to obtain the actual observed data of the feature points.

[0038] In one specific implementation, a reference time is first selected, and the angular velocity data of each pixel row of the image data between the actual exposure time and the reference time are integrated to obtain a rotation compensation matrix that describes the rotation compensation relationship; based on the rotation compensation matrix, rotation compensation is performed on each pixel row of the image data to obtain actual observation data of feature points unified to the reference time.

[0039] In a preferred embodiment, the method further includes feature matching and tracking of the actual observation data of feature points in the image data of different frames, and obtaining optimized actual observation information of feature points in the current frame after removing outliers.

[0040] S103. Perform pre-integration on the image state data jointly optimized by IMU measurement data and previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and construct the IMU pre-integration error term based on the pre-integration result. Pre-integration is performed based on the IMU measurement data and the jointly optimized image state data of the previous frame to obtain the pre-integration result between the current frame and the previous frame. This includes the jointly optimized camera corrected pose, velocity, gyroscope bias, and accelerometer bias information. If the current image data is the first frame, the image state data of the previous frame after joint optimization takes the preset default value. The pre-integration result is used to represent the relative motion constraint between the two frames.

[0041] Based on the pre-integration results, an IMU pre-integration error term is constructed. Specifically, the IMU pre-integration error term can be constructed in the form of an IMU residual function, which includes rotation error, velocity error, and position error, and is an IMU constraint term obtained by weighting the covariance.

[0042] S104. Obtain the real camera pose corresponding to each feature point in the image data of the current frame, and calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship. Based on the IMU measurement data and the line exposure time of the rolling shutter camera, the true camera pose corresponding to each feature point in the image data is calculated; based on the true camera pose and combined with the rotation compensation relationship at the reference time, the equivalent camera pose corresponding to each feature point in the image data is calculated; wherein, the equivalent camera pose and the actual observation data of the feature points are aligned in time.

[0043] S105. Calculate the theoretical observation data of feature points based on the equivalent camera pose, and construct the visual reprojection error based on the actual observation data and theoretical observation data of feature points. For the original spatial position data of three-dimensional feature points in space, the original spatial position data is transformed into the camera coordinate system by using the equivalent camera pose to obtain the camera coordinate position data; the camera projection function is called to project the camera coordinate position data onto the normalized plane to obtain the theoretical observation data of the feature points; the difference between the theoretical observation data of the feature points and the actual observation data of the feature points is calculated to obtain the visual reprojection error.

[0044] S106. Perform joint optimization based on IMU pre-integration error and visual reprojection error. When the joint optimization objective is achieved, obtain the camera pose correction result corresponding to the current frame reference time.

[0045] An overall objective function is constructed based on the IMU pre-integration error and the visual reprojection error, and then jointly solved to optimize the image state data. The image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. When the overall objective function value is minimized, the joint optimization objective is achieved, and the camera pose corresponding to the current frame reference time is output to obtain the camera pose correction result.

[0046] The method provided in this embodiment of the invention can finely compensate and correct system errors that are prone to occur in various stages when positioning based on visual images acquired by a rolling shutter camera using a visual inertial odometer, thereby improving the accuracy and robustness of the visual inertial odometer in high-speed motion scenarios and enhancing positioning accuracy.

[0047] See Figure 2 The second embodiment of the cooperative calibration method for visual inertial odometry in a rolling shutter camera according to the present invention includes the following: It is understood that the executing entity of this invention can be a collaborative calibration system for a visual inertial odometry system used in a rolling shutter camera, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example of the executing entity.

[0048] First, in this embodiment, the server acquires image data of the current frame from the rolling shutter camera, and simultaneously obtains IMU (Inertial Measurement Unit) measurement data output by the Visual-Inertial Odometry (VIO) while the image data is being captured. The IMU measurement data includes at least angular velocity data and acceleration data.

[0049] S201: Time synchronization and line exposure time calculation; The rolling shutter camera obtains image data line by line through exposure. Based on the specific parameters of the camera, the exposure time of the r-th row of pixels in the image data is: ; in, 'r' represents the first row exposure time, and 'r' represents the pixel row number. The exposure time interval between adjacent scan lines is given by camera parameters.

[0050] Meanwhile, clock synchronization is achieved by aligning the timestamp of the IMU (Inertial Measurement Unit) with the camera's time system through hardware triggering or software interpolation synchronization.

[0051] S202: IMU rotation calculation; Obtain a pre-built IMU coordinate system, and in the IMU coordinate system, calculate the angular velocity values ​​given by the IMU. During the period Integrating within the range, we obtain the rotation matrix of the IMU between the first row exposure and the r-th row exposure: ; in, Indicates from Time's up The rotation matrix at time step, the function This represents an exponential map.

[0052] Subsequently, in order to unify the feature points to the reference time... ,beg The inverse matrix is ​​used to obtain the line exposure time. to reference time Rotation matrix: ; in, It can be used for time alignment of subsequent feature points.

[0053] S203: Rotational transformation between IMU and camera coordinate systems; Since the coordinate system of multiple feature points in the image data directly acquired in this embodiment is the camera coordinate system, it is necessary to unify the coordinate system of the rotation matrix acquired based on the IMU and the coordinate system of each feature point in the image data. Specifically, this can be achieved through an extrinsic parameter calibration matrix. The rotation matrix of the IMU is transformed to the camera coordinate system using the following formula: ; in, This represents the rotation matrix from the IMU coordinate system (I) to the camera coordinate system (C). A fixed parameter can be obtained through pre-calibration. This represents the rotation from the camera to the IMU. This mapping transformation ensures that compensation is performed in the correct camera coordinate system, guaranteeing consistency between the compensation direction and the IMU measurement data.

[0054] In one specific implementation, the It can be obtained based on external parameter calibration methods, such as using tools like Kalibr or Basalt calibration, and calibrated using a calibration plate with a known geometry.

[0055] S204: Image distortion correction processing; Since the image data acquired by the camera usually has radial and tangential distortion, in order to ensure geometric accuracy, this embodiment will also use a radial-tangential distortion model or a fisheye distortion model to perform distortion correction operation on each frame of image data.

[0056] Given the camera intrinsic parameters K and distortion coefficients D, the original pixel coordinates Mapped to distortion-free coordinates This eliminates the impact of lens optical distortion on geometric consistency. ; Among them, the Let (u, v) represent the distortion correction function, (u', v') represent the original distorted pixel coordinates, (u', v') represent the distorted pixel coordinates, K represent the pre-calibrated camera intrinsic parameters, and D represent the pre-calibrated camera distortion parameters.

[0057] During distortion correction, based on the camera intrinsic parameter K and the distortion parameter D, the original pixel coordinates are mapped to distortion-free coordinates that conform to the ideal pinhole imaging model, thereby eliminating the influence of lens optical distortion on the image geometry and ensuring the accuracy of subsequent feature processing and geometric constraints.

[0058] S205: Normalized plane projection; The feature points of the image after removing rotation and distortion are represented as homogeneous coordinates; ; in, This represents the pixel coordinates of the feature points after removing rotation and distortion.

[0059] Subsequently, using the camera's intrinsic parameter inverse matrix, the distortion-corrected feature points are mapped onto the camera's normalized imaging plane, resulting in normalized plane coordinates. for: ; The normalized plane corresponds to the direction vector in the camera coordinate system, providing a unified mathematical space for compensation.

[0060] S206: Row-level feature compensation based on IMU rotation; For a feature point in pixel row r, its original normalized planar coordinates are: The original normalized coordinates represent the camera's position in time. The observation is then performed through a rotation matrix. Perform point compensation to obtain the observation direction corresponding to the feature, which is then compensated back to the reference time. Result: ; at this time This involves unifying the features to the direction vector of the reference exposure time, eliminating rotational distortion caused by the difference in exposure time between rows. After this step, all feature points in the entire image correspond to the same pose, obtaining the actual observation data of the feature points, thus forming a "pseudo-global shutter view".

[0061] The compensated normalized coordinates Reprojected to pixel coordinates using camera intrinsic parameters: ; in, These coordinates serve as the final feature point coordinates for front-end feature matching and tracking.

[0062] S207: Matching and tracking based on correction features; Next, optical flow or descriptor matching points are used to correct the feature points of the current frame. The feature points are tracked with those of the previous frame to obtain initial feature point matching pairs.

[0063] Based on the initial feature matching point pair, the fundamental matrix F between the two frames of images is estimated, and the epipolar geometric constraints defined by the fundamental matrix are used to remove matching points that do not meet the epipolar constraints, thereby eliminating erroneous matches and obtaining highly reliable feature matching point pairs.

[0064] Generally, the solution of the fundamental matrix F is easily affected by shutter distortion. In this embodiment, the aforementioned steps have eliminated the main shutter distortion through row-level rotation correction, ensuring the accuracy of the solution of the fundamental matrix F and thus improving the reliability of outlier removal. Finally, stable feature point matching pairs are obtained, and the actual observation data of the matched feature points are output to provide high-quality initial input for backend optimization.

[0065] S208: IMU pre-integration calculation; In this embodiment, the visual front-end correction stage described in steps S201-S207 only performs rotation correction and does not rely on feature point depth information, thus avoiding the introduction of unstable translation compensation. Visual front-end correction can significantly reduce feature drift caused by rapid rotation, improve feature tracking and matching stability, and provide high-quality initial observations for back-end optimization. In this embodiment, steps S208-S214 constitute the back-end optimization part.

[0066] In this step, the angular velocity and acceleration data of the IMU are read, and IMU pre-integration is performed based on the optimized state of the previous frame to obtain the pre-integration result between the current frame and the previous frame. At the same time, the pre-integration covariance matrix is ​​calculated for the construction of the IMU residual in the back-end optimization.

[0067] Specifically, when performing IMU pre-integration, the actual measurement results of the IMU are first obtained, and between two frames of images, three core quantities for pre-integration are calculated respectively, including the rotation increment representing the relative rotation between the two frames, the velocity increment representing the velocity change between the two frames, and the position increment representing the displacement change between the two frames.

[0068] S209: Calculation of true exposure pose of feature points; In the case of a rolling shutter, different feature points in the same image depend on the camera pose at different times. If a camera pose is introduced for optimization for each feature, it would result in the need to optimize hundreds of camera poses, leading to an extremely large computational load.

[0069] Therefore, to optimize computational efficiency, the rotation and velocity of the IMU during image acquisition are used to calculate the true camera exposure pose: ; ; ; in, This represents the actual exposure pose corresponding to the exposure time in row r. This represents the camera rotation attitude at the reference time obtained during constraint optimization in IMU pre-integration. This indicates the camera's pose at the current exposure moment. This represents the camera position at the reference time obtained during IMU pre-integration and constraint optimization. It has already been calculated during front-end compensation. The velocity at the reference time obtained by pre-integration of the IMU.

[0070] S210: Calculation of equivalent exposure pose for feature points; Since the front end has unified the actual observation data of feature points to the reference pose, if the back end still directly uses the camera pose at the actual exposure time, it will lead to inconsistency between the pose and the observation.

[0071] Therefore, for each visual feature point, based on the camera pose corresponding to its true exposure time and combined with the rotation compensation relationship performed in the front-end stage, the equivalent camera pose corresponding to the feature point observation after front-end correction is calculated.

[0072] Assuming the true exposure time of the feature point The corresponding camera pose is ( Based on the front-end rotation correction process, determine the rotation compensation matrix corresponding to the feature points from the actual exposure time to the unified reference time: ; Applying the rotation compensation relationship to the actual exposure pose, an equivalent camera pose is constructed: ; in, This represents the equivalent camera pose under the same pose semantics as the feature point observations after front-end rotation correction.

[0073] S211: Fine-tuning of observations based on continuous-time pose; Assuming the spatial position of the 3D feature point is P (in the world coordinate system), the equivalent exposure camera pose is used. Transform P to the camera coordinate system: ; in, This represents the 3D feature points transformed to the camera coordinate system.

[0074] Calling the camera projection function Π(·) will Projecting onto the normalized plane yields the theoretical normalized coordinates Π( This enables precise correction of feature point observations, compensating for translational motion and feature point depth effects not considered at the front end.

[0075] S212: Constructing a reprojection error model for rolling shutter; Based on the equivalent projection position in step S211 and the front-end correction observation in step S206 Construct the visual reprojection error term: ; Where K is the camera intrinsic parameter matrix. The reprojection errors of all feature points are combined to form the visual residual vector.

[0076] S213: Visual-IMU joint optimization; The camera pose, velocity, gyroscope bias, accelerometer bias, and spatial position of 3D feature points within the sliding window are used as optimization variables to construct the overall objective function. : ; ; ; in, To account for the reprojection error of the rolling shutter after line-by-line exposure and equivalent pose, This represents the sum of the reprojection errors of the rolling shutter. For IMU pre-integration error, This represents the sum of the IMU pre-integration errors.

[0077] S214: Output the optimized pose of the backend; During the specific joint optimization, an overall objective function is constructed based on the IMU pre-integration error and the visual reprojection error, and then jointly solved to optimize the image state data. This image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. The joint optimization objective is achieved when the overall objective function value is minimized. The image state data is then saved and can be used as the image state information during the pre-integration of the next frame image during the repeated compensation and correction step S208 when processing the next frame.

[0078] After optimization convergence, the optimal parameter estimates of the model are obtained by solving the system state variables, and the optimal camera pose output corresponding to the current frame is extracted to complete the visual inertial odometry estimation of a single frame and obtain the optimal estimated pose of the current frame.

[0079] Furthermore, in actual use, steps S201-S214 are executed cyclically to achieve high-precision pose output for continuous frames.

[0080] The specific solutions in this embodiment can be referred to in conjunction with the solutions in the first embodiment described above. For example, S202-S207 correspond to the content of S101, S208 corresponds to the content of S103, S209-S211 correspond to the content of S104-S105, and S212-S214 correspond to the content of S106.

[0081] The method provided in this embodiment of the invention enables fine-grained collaborative correction of visual inertial odometry (VIO) systems based on rolling shutter cameras, thereby improving the accuracy and robustness of the VIOS system in high-speed motion scenarios. Specifically, this method effectively reduces rolling shutter distortion, improves feature point stability, and increases feature matching success rate through front-end IMU-based row-level rotation compensation; it adopts a layered strategy of "lightweight front-end correction + fine-grained back-end modeling" to achieve a balance between accuracy and real-time performance while ensuring computational efficiency; it maintains stable tracking and positioning capabilities even under turbulent motion conditions, enhancing robustness in high-speed motion scenarios; furthermore, it uses equivalent exposure pose modeling to ensure that front-end observation and back-end optimization remain consistent in both time and geometry, improving optimization convergence. Furthermore, based on the collaborative correction method provided in this embodiment, the accuracy and robustness of VIOS systems based on rolling shutter cameras in high-speed motion scenarios can also be improved, achieving high-precision and stable navigation and positioning.

[0082] The above describes the cooperative calibration method for the visual inertial odometry of a rolling shutter camera according to embodiments of the present invention. The following describes the cooperative calibration system for the visual inertial odometry of a rolling shutter camera according to embodiments of the present invention. (See reference...) Figure 3 One embodiment of the co-calibration system for a visual inertial odometry of a rolling shutter camera in this invention includes: Data acquisition module 301 is used to acquire image data of the current frame captured by the rolling shutter camera and IMU measurement data output by the visual inertial odometry when acquiring the image data; The front-end compensation module 302 is used to calculate the rotation compensation relationship based on the IMU measurement data and perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. IMU compensation module 303 is used to perform pre-integration on image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and to construct an IMU pre-integration error term based on the pre-integration result. The backend compensation module 304 is used to obtain the real camera pose corresponding to each feature point in the image data of the current frame, calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship, calculate the theoretical observation data of the feature points based on the equivalent camera pose, and construct the visual reprojection error based on the actual observation data of the feature points and the theoretical observation data of the feature points. The joint optimization module 305 is used to perform joint optimization based on the IMU pre-integration error and the visual reprojection error. When the joint optimization target is achieved, the camera pose correction result corresponding to the current frame reference time is obtained.

[0083] The system provided in this embodiment of the invention can perform fine compensation and correction of system errors that are prone to occur in various stages when positioning based on visual images acquired by a rolling shutter camera using a visual inertial odometer, thereby improving the accuracy and robustness of the visual inertial odometer in high-speed motion scenarios and enhancing positioning accuracy.

[0084] In another embodiment of this application, the co-calibration system for the visual inertial odometry of the rolling shutter camera further includes a rolling shutter camera module 306 and an inertial measurement module 307. The rolling shutter camera module 306 is used to acquire multi-frame image data, and the inertial measurement module 307 is used to acquire IMU measurement data; The data acquisition module 301 is used to acquire data from the rolling shutter camera module and the inertial measurement module.

[0085] In another embodiment of this application, the co-calibration system for the visual inertial odometry of the rolling shutter camera further includes a clock synchronization module 308; The clock synchronization module 308 is used to synchronize and calibrate the time system of the camera and the visual inertial odometry through hardware triggering or software synchronization, so that the image data and the IMU measurement data are clock synchronized and have synchronized timestamps.

[0086] In another embodiment of this application, the front-end compensation module 302 includes a distortion correction unit, a feature detection unit, and a feature compensation unit; The feature compensation unit is used to calculate the rotation compensation relationship based on the IMU measurement data, and to perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. The distortion correction unit is used to perform distortion correction processing and normalized planar projection on the image data based on the camera intrinsic parameters and distortion coefficients of the rolling shutter camera. The matching optimization unit is used to perform feature matching and tracking on the actual observation data of feature points in the image data of different frames, and obtain the optimized actual observation information of feature points in the current frame after removing outliers.

[0087] In another embodiment of this application, the IMU measurement data includes at least angular velocity data; The feature compensation unit of the front-end compensation module 302 is specifically used for: By selecting a reference time, the angular velocity data of each pixel row of the image data between the actual exposure time and the reference time are integrated to obtain a rotation compensation matrix that describes the rotation compensation relationship. Based on the rotation compensation matrix, rotation compensation is performed on each pixel row of the image data to obtain the actual observed data of feature points unified to the reference time.

[0088] In another embodiment of this application, the backend compensation module 304 is specifically used for: Based on the IMU measurement data and the line exposure time of the rolling shutter camera, the true camera pose corresponding to each feature point in the image data is calculated; Based on the actual camera pose and the rotation compensation relationship at the reference time, the equivalent camera pose corresponding to each feature point in the image data is calculated; wherein the equivalent camera pose and the actual observation data of the feature points are aligned in time.

[0089] In another embodiment of this application, the backend compensation module 304 is further configured to: For the original spatial position data of three-dimensional feature points in space, the original spatial position data is transformed into the camera coordinate system by the equivalent camera pose to obtain the camera coordinate position data. The camera projection function is called to project the camera coordinate position data onto the normalized plane to obtain the theoretical observation data of the feature points; The visual reprojection error is obtained by subtracting the theoretical observation data of the feature points from the actual observation data of the feature points.

[0090] In another embodiment of this application, the joint optimization module 305 is specifically used for: Based on the IMU pre-integration error and the visual reprojection error, an overall objective function is constructed and jointly solved to optimize the image state data. The image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. When the overall objective function value is minimized, the joint optimization objective is achieved, and the camera pose corresponding to the current frame reference time is output to obtain the camera pose correction result.

[0091] The system provided in this embodiment of the invention enables a method for fine-grained collaborative calibration of the visual inertial odometry (VIO) of a rolling shutter camera, thereby improving the accuracy and robustness of the VIOS system in high-speed motion scenarios. Furthermore, based on the collaborative calibration method provided in this embodiment, the accuracy and robustness of the VIOS system based on the rolling shutter camera can also be improved in high-speed motion scenarios, achieving high-precision and stable navigation and positioning.

[0092] Based on the same inventive concept, this specification also provides an electronic device for co-calibrating the visual inertial odometry of a rolling shutter camera. The electronic device for co-calibrating the visual inertial odometry of a rolling shutter camera is described in detail below from the perspective of hardware processing.

[0093] Figure 5This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 5 To describe the electronic device 500 according to this embodiment of the invention. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0094] like Figure 5 As shown, the electronic device 500 is presented in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), a display unit 540, etc.

[0095] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform, for example... Figure 1 or Figure 2 The steps of the method shown.

[0096] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 5201 and / or a cache storage unit 5202, and may further include a read-only memory unit (ROM) 5203.

[0097] The storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0098] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0099] Electronic device 500 can also communicate with one or more external devices 100 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. Network adapter 560 can communicate with other modules of electronic device 500 via bus 530. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0100] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 or Figure 2 The method shown.

[0101] Figure 6 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.

[0102] accomplish Figure 1 or Figure 2The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0103] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Furthermore, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the co-calibration method for a visual inertial odometry for a rolling shutter camera as described in any of the above embodiments.

[0105] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0107] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0108] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0109] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for co-calibrating a visual inertial odometry for a rolling shutter camera, characterized in that, include: Acquire the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when the image data is acquired; The rotation compensation relationship is calculated based on the IMU measurement data, and the rotation compensation is performed on the image data of the current frame to obtain the actual observed data of the feature points of the image data. Pre-integration is performed on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and an IMU pre-integration error term is constructed based on the pre-integration result. Obtain the real camera pose corresponding to each feature point in the image data of the current frame, and calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship; The theoretical observation data of feature points are calculated based on the equivalent camera pose, and the visual reprojection error is constructed based on the actual observation data of the feature points and the theoretical observation data of the feature points. Based on the IMU pre-integration error and the visual reprojection error, joint optimization is performed. When the joint optimization objective is achieved, the camera pose correction result corresponding to the current frame reference time is obtained.

2. The co-calibration method for a visual inertial odometry (VIO) for a rolling shutter camera according to claim 1, further comprising, before calculating the rotation compensation relationship based on the IMU measurement data: The image data is subjected to distortion correction and normalized planar projection based on the camera intrinsic parameters and distortion coefficients of the rolling shutter camera. After obtaining the actual observed data of feature points in the image data, the method further includes: Feature matching and tracking are performed on the actual observation data of feature points in the image data of different frames, and after removing outliers, the optimized actual observation information of feature points in the current frame is obtained.

3. The method for co-calibrating a visual inertial odometry for a rolling shutter camera according to claim 2, characterized in that, The IMU measurement data includes at least angular velocity data; The step of calculating the rotation compensation relationship based on the IMU measurement data and performing rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data includes: By selecting a reference time, the angular velocity data of each pixel row of the image data between the actual exposure time and the reference time are integrated to obtain a rotation compensation matrix that describes the rotation compensation relationship. Based on the rotation compensation matrix, rotation compensation is performed on each pixel row of the image data to obtain the actual observed data of feature points unified to the reference time.

4. The method for co-calibrating a visual inertial odometry for a rolling shutter camera according to claim 3, characterized in that, The step of obtaining the true camera pose corresponding to each feature point in the image data of the current frame, and calculating the equivalent camera pose corresponding to the actual observed data of the feature points based on the rotation compensation relationship, includes: Based on the IMU measurement data and the line exposure time of the rolling shutter camera, the true camera pose corresponding to each feature point in the image data is calculated; Based on the actual camera pose and the rotation compensation relationship at the reference time, the equivalent camera pose corresponding to each feature point in the image data is calculated; wherein the equivalent camera pose and the actual observation data of the feature points are aligned in time.

5. The method for co-calibrating a visual inertial odometry for a rolling shutter camera according to claim 1, characterized in that, The calculation of theoretical observation data of feature points based on the equivalent camera pose, and the construction of visual reprojection error based on the actual observation data of feature points and the theoretical observation data of feature points, include: For the original spatial position data of three-dimensional feature points in space, the original spatial position data is transformed into the camera coordinate system by the equivalent camera pose to obtain the camera coordinate position data. The camera projection function is called to project the camera coordinate position data onto the normalized plane to obtain the theoretical observation data of the feature points; The visual reprojection error is obtained by subtracting the theoretical observation data of the feature points from the actual observation data of the feature points.

6. The method for co-calibrating a visual inertial odometry for a rolling shutter camera according to any one of claims 1-5, characterized in that, The joint optimization based on the IMU pre-integration error and the visual reprojection error, when the joint optimization objective is achieved, yields the camera pose correction result corresponding to the current frame reference time, including: Based on the IMU pre-integration error and the visual reprojection error, an overall objective function is constructed and jointly solved to optimize the image state data. The image state data includes camera pose, velocity, gyroscope bias, and accelerometer bias. When the overall objective function value is minimized, the joint optimization objective is achieved, and the camera pose corresponding to the current frame reference time is output to obtain the camera pose correction result.

7. A cooperative calibration system for a visual inertial odometry system for a rolling shutter camera, characterized in that, The co-calibration system for the visual inertial odometry used in rolling shutter cameras includes: The data acquisition module is used to acquire the image data of the current frame captured by the rolling shutter camera and the IMU measurement data output by the visual inertial odometry when acquiring the image data; The front-end compensation module is used to calculate the rotation compensation relationship based on the IMU measurement data and perform rotation compensation on the image data of the current frame to obtain the actual observed data of the feature points of the image data. The IMU compensation module is used to perform pre-integration on the image state data jointly optimized by the IMU measurement data and the previous frame image data to obtain the pre-integration result between the current frame and the previous frame, and to construct the IMU pre-integration error term based on the pre-integration result. The backend compensation module is used to obtain the real camera pose corresponding to each feature point in the image data of the current frame, calculate the equivalent camera pose corresponding to the actual observation data of the feature points based on the rotation compensation relationship, calculate the theoretical observation data of the feature points based on the equivalent camera pose, and construct the visual reprojection error based on the actual observation data of the feature points and the theoretical observation data of the feature points. The joint optimization module is used to perform joint optimization based on the IMU pre-integration error and the visual reprojection error. When the joint optimization target is achieved, the camera pose correction result corresponding to the current frame reference time is obtained.

8. A co-calibration device for a visual inertial odometry system used in a rolling shutter camera, characterized in that, The co-calibration device for the visual inertial odometry of the rolling shutter camera includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the co-calibration device for the visual inertial odometry of a rolling shutter camera to perform the steps of the co-calibration method for the visual inertial odometry of a rolling shutter camera as claimed in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program / instructions thereon, characterized in that, When the program / instructions are executed by the processor, they implement the steps of the cooperative calibration method for a visual inertial odometry for a rolling shutter camera as described in any one of claims 1-6.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the co-calibration method for a visual inertial odometer for a rolling shutter camera as described in any one of claims 1-6 are implemented.

Images