A camera and imu joint calibration method

Abstract

The application provides a camera and IMU joint calibration method, which comprises the following steps: step 1, initializing a target camera and a target IMU to obtain initialization parameters; step 2, constructing a first error L1 and performing one-time iterative optimization by using an LM algorithm; step 3, updating the initialization parameters and the bias of the target IMU acceleration, and determining whether the total error determined by the first error is less than a set threshold value; if yes, entering step 7, otherwise, entering the next step; step 4, obtaining gyro noise and acceleration noise according to the bias of the gyroscope and the acceleration of the target IMU at each time after updating; step 5, constructing a second error according to the acceleration noise and the first error L1; step 6, performing iterative optimization on the second total error L2 by using the LM algorithm, and determining whether the error is less than a preset threshold value; if yes, entering the next step, otherwise, entering step 3; and step 7, obtaining a calibration quantity. The application realizes joint calibration of the camera and the IMU, and has high efficiency and high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of SLAM technology, and particularly relates to a camera and IMU joint calibration method. BACKGROUND

[0002] In SLAM (Simultaneous Localization and Mapping), if only a camera or an IMU is used for mapping or positioning, the accuracy is not high and the robustness is not good. If only a camera is used, mapping cannot continue or the mapping error is large in areas with less texture or high motion speed (resulting in image blur); if only an IMU is used, the IMU is easily disturbed by noise, and there will be a large error after a long time. Therefore, a scheme of using a camera and an IMU together is proposed, the camera is used to calibrate the IMU in a place with good texture, and the IMU is used to provide mapping and positioning information when appearing in a weak or no texture or image blur area, so as to improve the accuracy and robustness of mapping and positioning.

[0003] When fusing the data of the camera and the IMU, the camera or the IMU sensor needs to be described in the same coordinate system. The purpose of calibration is to convert the motion information between the two sensors to the same coordinate system and minimize the error.

[0004] At present, there is no camera and IMU joint calibration method to calibrate the transformation matrix between the IMU and the camera and the timestamp registration between the two sensors, so as to facilitate the description in the same reference coordinate system during fusion. SUMMARY

[0005] The present application provides a camera and IMU joint calibration method, which aims to solve the defects in the prior art, realizes joint calibration of the camera and the IMU, and has high efficiency and high accuracy.

[0006] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0007] The present application provides a camera and IMU joint calibration method, which includes:

[0008] Step 1, initializing a target camera and a target IMU to obtain initialization parameters, the initialization parameters including a time difference of the target camera and the target IMU time, a transformation matrix of the coordinate system between the target camera and the target IMU, a bias of a gyroscope of the target IMU, and a value of gravitational acceleration in a world coordinate system;

[0009] Step 2, constructing a first error , and using an LM algorithm to perform iterative optimization once, the first error = wherein, represents the re-projection error, represents the acceleration error, represents the angular velocity error, represents the error caused by the angular velocity random walk;

[0010] Step 3, updating the initialization parameters and the bias of the target IMU acceleration, determining whether the total error determined by the first error is less than a set threshold, yes to step 7, otherwise to the next step;

[0011] Step 4, obtaining the gyroscope noise and the acceleration noise respectively according to the updated bias of the gyroscope and the acceleration of the target IMU at each time;

[0012] Step 5, obtaining the second error according to the first error and the acceleration noise , the second error wherein, represents the error caused by the acceleration random walk;

[0013] Step 6, using the LM algorithm to iteratively optimize the second total error L2, determining whether the error is less than a preset threshold, yes to the next step, otherwise to step 3;

[0014] Step 7, obtaining the calibration quantity.

[0015] Specifically, the step 1 comprises:

[0016] Step 101, mounting the target camera and the target IMU on a fixing device, and collecting pictures taken by the target camera and accelerations and angular velocities output by the target IMU;

[0017] Step 102, using feature matching to solve discrete poses of the target camera according to the pictures taken by the target camera;

[0018] Step 103, using pre-integration to calculate poses at each collection time for the target IMU;

[0019] Step 104, obtaining a rotation vector curve and a translation curve of the target camera according to the discrete poses of the target camera;

[0020] Step 105, obtaining an acceleration curve and an angular velocity curve of the IMU respectively according to the accelerations and the angular velocities of the target IMU;

[0021] ​Step 106, initializing the target camera and the target IMU according to the rotation vector curve, the translation curve, the acceleration curve and the angular velocity curve, obtaining initialization parameters, the initialization including: initializing a time difference of the target camera and the target IMU time, initializing the transformation matrix T, initializing a bias of the gyroscope of the target IMU at each time, and initializing a value of the gravity acceleration in the world coordinate system.

[0022] Specifically, the step 102 includes:

[0023] Step 1021, performing feature extraction and matching on the pictures taken by the target camera.

[0024] Step 1022, calculating a transformation matrix corresponding to a fundamental matrix or a homography matrix according to the matching result.

[0025] Step 1023, calculating discrete poses of the target camera according to the transformation matrix, the discrete poses including rotation and displacement.

[0026] Specifically, initializing the time difference of the target camera and the target IMU time includes:

[0027] Step A1, obtaining a rotation vector of a time point of the target camera corresponding to a collection time of the target IMU according to the rotation vector curve and a first preset relationship.

[0028] Step A2, obtaining an angular velocity of the target camera at an arbitrary collection time in the camera coordinate system according to the rotation vector.

[0029] Step A3, converting the angular velocity of the target IMU to the same coordinate system, and minimizing an angular velocity error sum of the two.

[0030] Step A4, calculating an initial value of the time difference according to a second preset relationship.

[0031] Specifically, the first preset relationship is:

[0032]

[0033] wherein, is the time difference, t cam represents a camera time, t imu represents an IMU collection time.

[0034] The second preset relationship is:

[0035]

[0036] wherein, represents an angular velocity of the IMU at the time. angular velocity of the camera at time.

[0037] Specifically, the transformation matrix T of the coordinate system between the target camera and the target IMU and the bias of the gyroscope of the target IMU at each time point include:

[0038] Step B1, obtaining the angular velocity of the target camera at any acquisition time in the camera coordinate system according to the rotation vector curve;

[0039] Step B2, converting the angular velocity of the target IMU at the same acquisition time to the same coordinate system;

[0040] Step B3, obtaining the rotation between the target camera and the target IMU, and determining the bias of the gyroscope at each time point according to a third preset formula;

[0041] Step B4, determining the translation of the target camera at any time according to the translation curve;

[0042] Step B5, converting the translation of the target camera to the target IMU coordinate system to obtain the representation of the displacement of the target camera in the target IMU coordinate system;

[0043] Step B6, the target IMU obtains the displacement by pre-integration;

[0044] Step B7, determining the initial value of the displacement by a fourth preset relationship.

[0045] Specifically, the third preset relationship is:

[0046]

[0047] wherein, represents the rotation between the target camera and the target IMU, represents the angular velocity of the IMU at time, represents the angular velocity of the camera at time, represents the bias of the gyroscope of the IMU at the time, and the superscript indicates that the parameter is to be optimized.

[0048] Specifically, the fourth preset relationship is:

[0049]

[0050] wherein, represents the translation of the origin of the world coordinate system at time to the target IMU coordinate system; represents Translation from the world coordinate system to the camera coordinate system at any moment. This indicates the rotation between the target camera and the target IMU.

[0051] Specifically, the initial value of gravitational acceleration in the world coordinate system includes: determining the initial value of gravitational acceleration in the world coordinate system according to a fifth preset relation, wherein the fifth preset relation is:

[0052]

[0053] in, This indicates the rotation between the target camera and the target IMU. Indicates the first The target IMU acceleration obtained at each IMU acquisition time; Indicates the first The rotation of the target camera in the world coordinate system at each IMU acquisition time; This represents the value of gravitational acceleration in the world coordinate system. The superscript * indicates that this parameter needs to be optimized.

[0054] Specifically, the reprojection error is determined by a sixth preset relation, which is:

[0055]

[0056] in, For the first The first position under the first pose One real projection point, Indicates the first The first camera pose The predicted value for each projection point, where angle brackets in the parameters indicate that the parameter contains a quantity to be optimized. Represents the information matrix.

[0057] Specifically, the acceleration error is determined by a seventh preset relation, which is:

[0058]

[0059] in, Indicates IMU number The rotation from the world coordinate system to the camera coordinate system at each moment; Indicates IMU number At any given moment, the camera's acceleration in the world coordinate system; Indicates the first in the IMU coordinate system Acceleration at any moment; Bias of the jth IMU acquisition time acceleration, all initial 0; The angle brackets in the parameter indicate that the parameter contains the quantity to be optimized.

[0060] Specifically, the angular velocity error is determined by an eighth preset relationship, and the eighth preset relationship is:

[0061]

[0062] Wherein, The angular velocity of the camera in the world coordinate system at the jth IMU acquisition time is represented by The bias of the gyroscope at the jth IMU acquisition time is represented by The bias of the gyroscope at the jth IMU acquisition time is represented by

[0063] The beneficial effects of the present application are that the present application initializes the target camera and the target IMU, obtains the initialization parameters, and performs the first iteration on the initialization parameters through the first error, and then optimizes the iteration on all observation parameters according to the second error, so as to obtain the calibration quantity, realizes the joint calibration of the camera and the IMU, and has high efficiency and high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The flowchart of the camera and IMU joint calibration method of the present application. DETAILED DESCRIPTION

[0065] The embodiments of the present application will be specifically described below in combination with the drawings, which are only used for reference and illustration, and do not constitute a limitation on the scope of patent protection of the present application.

[0066] In the flow described in the specification, claims or drawings of the present application, the serial numbers of each step (such as steps 10, 20, etc.) are only used to distinguish each step, and the serial numbers themselves do not represent any execution order. It should be noted that the "first", "second" and the like in this paper are only used to distinguish the description objects and the like, and do not represent the order of precedence.

[0067] As shown in Figure 1 The present embodiment provides a camera and IMU joint calibration method, which comprises:

[0068] Step 1, initializing the target camera and the target IMU to obtain initialization parameters, the initialization parameters including the time difference of the target camera and the target IMU time, the transformation matrix of the coordinate system between the target camera and the target IMU, the bias of the target IMU gyroscope, and the value of the gravity acceleration in the world coordinate system.

[0069] In the embodiment, the step 1 comprises:

[0070] Step 101, mounting a target camera and a target IMU on a fixing device, and collecting pictures taken by the target camera and acceleration and angular velocity output by the target IMU.

[0071] Step 102, calculating discrete poses of the target camera according to the pictures taken by the target camera by using feature matching.

[0072] In the embodiment, the step 102 comprises:

[0073] Step 1021, performing feature extraction and matching on the pictures taken by the target camera.

[0074] Methods of feature extraction include SIFT, SURF, ORB and deep learning.

[0075] In the calibration process, real-time performance is not considered, and therefore, in order to improve the accuracy of the calibration parameters, the SIFT method with high accuracy can be used to extract features; the deep learning method has better robustness in extracting feature points, but needs to rely on training data.

[0076] Methods of feature matching include descriptor matching and KNN matching.

[0077] Step 1022, calculating a transformation matrix T corresponding to a fundamental matrix or a homography matrix according to the matching result.

[0078] In the embodiment, the transformation matrix T is represented as follows:

[0079]

[0080] wherein R represents rotation, t represents translation, 0 T represents a transposed matrix.

[0081] In the embodiment, the RANSAC method is used to solve the transformation matrix T corresponding to the fundamental matrix or the homography matrix, and the one with higher accuracy is selected as the final transformation matrix T.

[0082] Step 1023, calculating discrete poses of the target camera according to the transformation matrix T, wherein the discrete poses comprise rotation R and displacement t.

[0083] In specific implementation, the solution of the discrete poses is divided into two cases:

[0084] ① If the matching points are not all on a plane, the discrete poses of the camera are solved by the following formula:

[0085]

[0086] If all the matched points are in one plane, the discrete pose of the camera is solved by the following formula:

[0087]

[0088] wherein, is the point of a certain point in the world coordinate system in the normalized plane under the camera coordinate system of camera 1 and camera 2 respectively; is the pixel point of the point of a certain point in the world coordinate system in the normalized plane under the camera coordinate system of camera 1 and camera 2 respectively after being projected to the pixel plane by the camera projection model; denotes the essential matrix (Essential Matrix), denotes the fundamental matrix (Fundamental Matrix), denotes the homograph matrix (Homograph Matrix).

[0089] The matched points of a plurality of points at two different positions (the points of a world point on the normalized plane under the camera coordinate system of different cameras or the points on the pixel plane) are used to form a matrix expression, and the unique rotation R and displacement t can be obtained by SVD decomposition and the fact that there is a positive depth at the positions of the two cameras.

[0090] Step 103, using pre-integration calculation to calculate the pose of each collection time for the target IMU.

[0091] In this step, the IMU pre-integration can be expressed as:

[0092]

[0093] wherein, denotes the process of integration, denotes the rotation obtained by integration at the last time (the IMU collection time of the first denotes the displacement obtained by integration at the last time, denotes the acceleration measured at the current time (the IMU collection time of the first denotes the angular velocity measured at the current time, denotes the noise of the acceleration, denotes the noise of the angular velocity, denotes the bias of the acceleration at the current time, denotes the bias of the gyroscope at the current time.

[0094] Step 104, obtaining the rotation vector curve and the translation curve of the target camera according to the discrete pose of the target camera.

[0095] ​​In implementation, the rotation vector curve of the target camera can be obtained by B-Splines according to the rotation in the discrete poses of the target camera, and the translation curve of the target camera can be obtained by B-Splines according to the translation in the discrete poses of the target camera.

[0096] The rotation vector curve and the translation curve obtained by B-Splines are prior art and will not be described in detail.

[0097] Step 105, obtaining the acceleration curve and the angular velocity curve of the IMU according to the acceleration and the angular velocity of the target IMU, respectively.

[0098] In implementation, the acceleration curve and the angular velocity curve of the target IMU can be obtained by B-Splines.

[0099] Step 106, initializing the target camera and the target IMU according to the rotation vector curve, the translation curve, the acceleration curve and the angular velocity curve to obtain initialization parameters, the initialization including: initializing the time difference of the target camera and the target IMU, initializing the transformation matrix T, initializing the bias of the gyroscope of the target IMU at each time, and initializing the value of the gravitational acceleration in the world coordinate system.

[0100] In the embodiment, initializing the time difference of the target camera and the target IMU includes:

[0101] Step A1, obtaining the rotation vector of the target camera at the time point corresponding to the target IMU acquisition time according to the rotation vector curve and a first preset relationship. .

[0102] In the embodiment, the first preset relationship is:

[0103]

[0104] wherein, is the time difference, t cam represents the camera time, t imu represents the IMU acquisition time.

[0105] It can be seen that the rotation vector is a function of time.

[0106] Step A2, obtaining the angular velocity ω of the target camera at any acquisition time in the camera coordinate system according to the rotation vector c .

[0107] In implementation, the angular velocity ω c is determined according to the following formula:

[0108]

[0109] wherein, is an increase of time at the collection moment .

[0110] Step A3, the angular velocity of the target IMU is converted to the same coordinate system, and the total angular velocity error of the two is minimized.

[0111] Step A4, the initial value of the time difference is calculated according to a second preset relationship.

[0112] In the embodiment, the second preset relationship is:

[0113]

[0114] wherein, denotes the angular velocity of the IMU at the moment, denotes the angular velocity of the camera at the moment (physically at the same moment as the IMU).

[0115] In the embodiment, the transformation matrix T between the coordinate systems of the target camera and the target IMU, and the bias of the gyroscope of the target IMU at each moment are initialized, which includes:

[0116] Step B1, the angular velocity ω c of the target camera at any collection moment in the camera coordinate system is obtained according to the rotation vector curve.

[0117] Step B2, the angular velocity of the target IMU at the same collection moment is converted to the same coordinate system.

[0118] Step B3, the rotation between the target camera and the target IMU is obtained, and the bias of the gyroscope at each moment is determined according to a third preset formula.

[0119] In the embodiment, the third preset relationship is:

[0120]

[0121] wherein, denotes the rotation between the target camera and the target IMU, denotes the angular velocity of the IMU at the moment, denotes the angular velocity of the camera at the moment, denotes the bias of the gyroscope of the IMU at the first The gyroscope bias at each moment; * indicates that this parameter needs to be optimized.

[0122] Step B4: Determine the translation of the target camera at any given time based on the translation curve.

[0123] Step B5: Transform the translation of the target camera into the target IMU coordinate system to obtain the representation of the target camera displacement in the target IMU coordinate system.

[0124] Step B6: The target IMU obtains the displacement using pre-integration.

[0125] Step B7: Determine the initial value of the displacement using the fourth preset relation.

[0126] In this embodiment, the fourth preset relation is:

[0127]

[0128] in, express Translation from the origin of the world coordinate system to the target IMU coordinate system at any given time; express Translation from the world coordinate system to the camera coordinate system at any moment. This indicates the rotation between the target camera and the target IMU.

[0129] In this embodiment, the initialization of the gravitational acceleration in the world coordinate system includes: determining the initial value of the gravitational acceleration in the world coordinate system according to a fifth preset relation, wherein the fifth preset relation is:

[0130]

[0131] in, This indicates the rotation between the target camera and the target IMU. Indicates the first The target IMU acceleration (in the IMU coordinate system) obtained at each IMU acquisition time; Indicates the first The rotation of the target camera in the world coordinate system at each IMU acquisition moment can be obtained through B-Splines curves. This represents the value of gravitational acceleration in the world coordinate system. The superscript * indicates that this parameter needs to be optimized.

[0132] When using the fifth preset formula for calculation, the acceleration bias and noise should be ignored, and the acceleration at calibration should be assumed to be 0.

[0133] Step 2: Construct the first error and use the LM algorithm to optimize once.

[0134] In this embodiment, the first error where, represents the re-projection error, represents the acceleration error, represents the angular velocity error, represents the error caused by the angular velocity random walk.

[0135] The B-Splines curve is obtained by using the bias of the gyroscope of the IMU at each time in step 1 through the B-Splines method, and then the first derivative with respect to time is obtained to obtain the noise of the gyroscope.

[0136] 1. Reprojection error

[0137] The camera projection model can be simply written as:

[0138]

[0139] The meaning represented by the above formula is: the world coordinate point is converted to the camera coordinate system through the pose transformation matrix , and then projected to the image plane through (including normalization and the specific model and distortion of the camera), and the image plane is converted to the pixel coordinate system through the camera intrinsic parameter .

[0140] Because the rotation and translation between the IMU and the camera need to be calibrated, the pose of the IMU in the world coordinate system is projected after being converted from the IMU to the camera, and the homogeneous coordinates are used, and are expressed as follows:

[0141]

[0142] where, represents the pose transformation from the IMU to the camera, represents the pose of the IMU in the world coordinate system, represents the point in the world coordinate system. Wherein:

[0143]

[0144] After dehomogenization, it can be expressed as:

[0145]

[0146] Suppose , in the above formula , the following represents taking dimensional values.

[0147] ​Therefore, in this embodiment, the reprojection error is determined by a sixth preset relation, which is:

[0148]

[0149] in, For the first The first position under the first pose One real projection point, Indicates the first The first camera pose The predicted value for each projection point, where angle brackets in the parameters indicate that the parameter contains a quantity to be optimized. Represents the information matrix.

[0150] The purpose is to normalize the error data in each dimension, which can increase the solution accuracy and speed up the iteration convergence. It can be determined based on the feature layer or based on the absolute deviation of the termination.

[0151] For example, based on the layer where the feature is located, assuming the current feature point is in... If the layer is:

[0152]

[0153] 2. Acceleration error

[0154] The acceleration bias is affected by noise and varies at each moment. Assuming the acceleration noise... ,but:

[0155] In this embodiment, the acceleration error is determined by a seventh preset relation, which is:

[0156]

[0157] in, Indicates IMU number The rotation from the world coordinate system to the camera coordinate system at any given moment (the rotation at any moment can be obtained using the B-splines method); Indicates IMU number At any given moment, the camera's acceleration in the world coordinate system; Indicates the first in the IMU coordinate system Acceleration at any moment; This represents the bias of the acceleration at the j-th acquisition time of the IMU, which is initially 0; the angle brackets in the parameter indicate that the parameter contains a quantity to be optimized.

[0158] In this embodiment, The second derivative can be obtained by using the translation curve of the target camera described in step 104.

[0159] 3. Angular velocity error

[0160] Angular velocity bias is affected by noise and varies at every moment, assuming acceleration noise. ,but:

[0161] In this embodiment, the angular velocity error is determined by the eighth preset relation, and the seventh and eighth preset relations are:

[0162]

[0163] in, Indicates that IMU is in The angular velocity of the camera in the world coordinate system at each acquisition moment; This represents the gyroscope bias at the j-th IMU acquisition time, and its initial value can be obtained from the initialization process. The variables enclosed in angle brackets above represent variables to be optimized.

[0164] In this embodiment, The first derivative can be obtained by taking the Lie algebra corresponding to the rotation vector curve of the target camera.

[0165] 4. Errors caused by random walk of angular velocity

[0166] After power is applied, the bias of the IMU changes at every moment. This is the zero bias instability of the IMU. This situation can be regarded as the bias of the IMU being disturbed by noise. After a period of time, it becomes another value. This process is the bias on the basis of the previous value plus the integral of the noise during this period.

[0167] Assuming the noise effect is a Gaussian process, then:

[0168]

[0169]

[0170] in, Random noise representing acceleration, This represents the random noise of the gyroscope. Indicates acceleration at Time offset, Indicates angular velocity at The time offset.

[0171] Assuming the processes are statistically independent, then:

[0172]

[0173] The error caused by the random walk of angular velocity can be expressed as:

[0174]

[0175] The random walk error is updated after each iteration of updating the bias.

[0176] Step 3, update the initialization parameters and the bias of the target IMU acceleration, and determine whether the total error determined by the first error is less than a set threshold value. If yes, go to step 7, otherwise go to the next step.

[0177] Step 4, according to the updated bias of the target IMU gyroscope and acceleration at each time, respectively obtain the gyroscope noise and acceleration noise.

[0178] In this step, the B-Splines method is used to obtain the bias curve, and then the first derivative of time is obtained, respectively, to obtain the gyroscope noise and the acceleration noise.

[0179] Step 5, according to the acceleration noise and the first error Construct the second error .

[0180] In this embodiment, the second error wherein, represents the error caused by the random walk of acceleration.

[0181] The angular velocity bias at each IMU collection time can be roughly determined by the third preset relationship, and the B-Splines curve can be obtained by the bias at each time, and then the derivative of t is obtained to obtain the random walk of acceleration. The bias of acceleration is not obtained in the above initialization, so the initial value is zero when the error is calculated for the first time, and after the first update, the B-Splines method is used to obtain the curve, and then the derivative is obtained to obtain the random walk of acceleration.

[0182] Assuming that the process is statistically independent of each other, then:

[0183]

[0184] The error caused by the random walk of acceleration can be expressed as:

[0185]

[0186] The random walk error is updated after each iteration of updating the bias.

[0187] Step 6, using the LM algorithm to iteratively optimize the second total error L2, and determining whether the error is less than a preset threshold, if yes, entering the next step, otherwise, entering step 3.

[0188] Step 7, obtaining a calibration quantity.

[0189] In this step, the calibration quantity is a rotation, a translation, and a time difference between the target IMU and the target camera.

[0190] The above only discloses the preferred embodiments of the present application, and cannot limit the protection scope of the present application, so equivalent changes made in the patent application range of the present application still belong to the scope covered by the present application.

Claims

1. A method for camera and IMU joint calibration, characterized in that, The method comprises the following steps: Step 1, initializing the target camera and the target IMU to obtain initialization parameters, wherein the initialization parameters comprise a time difference between the target camera and the target IMU, a transformation matrix between the coordinate systems of the target camera and the target IMU, a bias of the gyroscope of the target IMU, and a value of the gravity acceleration in the world coordinate system; Step 2, construct the first error and use the LM algorithm to perform one iteration of optimization, the first error where, represents the re-projection error, represents the acceleration error, represents the angular velocity error, represents the error due to angular velocity random walk;​ Step 3, updating the initialization parameters and the bias of the acceleration of the target IMU, and determining whether the total error determined by the first error is less than a set threshold value; if yes, proceeding to step 7; otherwise, proceeding to the next step; Step 4, obtaining the gyroscope noise and the acceleration noise respectively according to the updated bias of the gyroscope and the acceleration of the target IMU at each time point; Step 5, determining a second error based on the acceleration noise and the first error constructing a second error , the second error wherein, denotes the error due to the acceleration random walk Step 6, using the LM algorithm to iteratively optimize the second error L2, and determining whether the error is less than a preset threshold value; if yes, proceeding to the next step; otherwise, proceeding to step 3; Step 7, obtaining a calibration quantity, wherein the calibration quantity is a rotation, a translation and a time difference between the target IMU and the target camera.

2. The camera and IMU joint calibration method according to claim 1, characterized in that, The step 1 comprises the following steps: Step 101, mounting the target camera and the target IMU on a fixing device, and collecting pictures taken by the target camera and acceleration and angular velocity output by the target IMU; Step 102, using feature matching to solve discrete poses of the target camera according to the pictures taken by the target camera; Step 103, calculating poses at each collection time point of the target IMU using pre-integration; Step 104, obtaining a rotation vector curve and a translation curve of the target camera according to the discrete poses of the target camera; Step 105, obtaining an acceleration curve and an angular velocity curve of the target IMU according to the acceleration and the angular velocity of the target IMU respectively; Step 106, initializing the target camera and the target IMU to obtain the initialization parameters according to the rotation vector curve, the translation curve, the acceleration curve and the angular velocity curve, wherein the initialization comprises initializing the time difference between the target camera and the target IMU, initializing the transformation matrix, initializing the bias of the gyroscope of the target IMU at each time point, and initializing the value of the gravity acceleration in the world coordinate system.

3. The camera and IMU joint calibration method according to claim 2, characterized in that, The step 102 comprises the following steps: Step 1021, performing feature extraction and matching on the pictures taken by the target camera; Step 1022, calculating a transformation matrix corresponding to a fundamental matrix or a homography matrix according to a matching result; Step 1023, calculating discrete poses of the target camera according to the transformation matrix, wherein the discrete poses comprise rotation and displacement.

4. The camera and IMU joint calibration method of claim 2, wherein, Initializing the time difference between the target camera and the target IMU comprises the following steps: Step A1, obtaining a rotation vector of a time point of the target camera corresponding to a collection time point of the target IMU according to the rotation vector curve and a first preset relationship; Step A2, obtaining an angular velocity of the target camera at an arbitrary collection time point in a camera coordinate system according to the rotation vector; Step A3, converting the angular velocity of the target IMU to the same coordinate system, and minimizing the total angular velocity error; Step A4, calculating an initial value of the time difference according to a second preset relationship.

5. The camera and IMU joint calibration method according to claim 4, characterized in that, The first preset relationship is: wherein, is the time difference, t cam denotes the camera time instant, t imu denotes the IMU acquisition time instant; The second preset relationship is: in, Indicates that IMU is in angular velocity at time t, Indicates the camera is in Angular velocity at time t.

6. The camera and IMU joint calibration method according to claim 2, characterized in that, The initialization of the transformation matrix of the coordinate system between the target camera and the target IMU, and the bias of the gyroscope of the target IMU at each time includes: Step B1, obtaining the angular velocity of the target camera at any acquisition time in the camera coordinate system according to the rotation vector curve; Step B2, converting the angular velocity of the target IMU at the same acquisition time to the same coordinate system; Step B3, obtaining the rotation between the target camera and the target IMU, and determining the bias of the gyroscope at each time according to a third preset relationship; Step B4, determining the translation of the target camera at any time according to the translation curve; Step B5, converting the translation of the target camera to the target IMU coordinate system to obtain the representation of the displacement of the target camera in the target IMU coordinate system; Step B6, the target IMU obtains the displacement by pre-integration; Step B7, determining the initial value of the displacement by a fourth preset relationship.

7. The camera and IMU joint calibration method according to claim 6, characterized in that, The third preset relationship is: wherein, Rtarget represents the rotation between the target camera and the target IMU, ωtarget represents the angular velocity of the IMU at ttarget, ωcam represents the angular velocity of the camera at tcam, bgy represents the bias of the gyroscope at the th time instant of the IMU, the superscript * indicates that this parameter is to be optimized. Specifically, the fourth preset relationship is: wherein, denotes the translation of the world coordinate system origin to the target IMU coordinate system at time t; denotes the translation of the world coordinate system to the camera coordinate system at time t, denotes the rotation between the target camera and the target IMU.

8. The camera and IMU joint calibration method according to claim 2, characterized in that, The value of the initial gravity acceleration in the world coordinate system includes: determining the initial value of the gravity acceleration in the world coordinate system according to a fifth preset relationship, and the fifth preset relationship is: wherein, represents a rotation between the target camera and the target IMU, represents a target IMU acceleration at the time of the first IMU acquisition; represents a rotation of the target camera in the world coordinate system corresponding to the time of the first IMU acquisition; represents the value of the gravitational acceleration in the world coordinate system, the superscript indicates that this parameter is to be optimized.

9. The camera and IMU joint calibration method according to claim 1, characterized in that, The re-projection error is determined by a sixth preset relationship, and the sixth preset relationship is: in, For the first The first position under the first pose One real projection point, Indicates the first The first camera pose The predicted value for each projection point, where angle brackets in the parameters indicate that the parameter contains a quantity to be optimized. Represents the information matrix; The acceleration error is determined by a seventh preset relationship, and the seventh preset relationship is: wherein, represents the rotation of the IMU at the jth time instant from the world coordinate to the camera coordinate system; represents the acceleration of the camera in the world coordinate system at the jth time instant; represents the acceleration in the IMU coordinate system at the jth time instant; represents the bias of the acceleration at the jth time instant of the IMU, which is initially 0; the angle brackets in the parameters indicate that the parameters contain quantities to be optimized;​​​ The angular velocity error is determined by an eighth preset relationship, and the eighth preset relationship is: wherein, represents the angular velocity of the camera in the world coordinate system at the jth acquisition instant of the IMU; represents the bias of the gyroscope at the jth acquisition instant of the IMU, the initial value is obtained from the initialization process, and the quantity with angle brackets above represents a variable to be optimized.​

Images