Multi-camera and IMU (Inertial Measurement Unit) collaborative calibration method and device

Abstract

The invention provides a multi-camera and IMU collaborative calibration method and device. The method comprises the steps that auxiliary camera arrays are dispersedly arranged around a main camera and an IMU; using the auxiliary camera array to obtain a horizontal focal length set, a vertical focal length set, a relative attitude matrix set and a relative translation matrix set; acquiring an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence and an IMU acquisition time sequence by using an IMU and an auxiliary camera array; acquiring world coordinates of the main camera; and obtaining an attitude calibration matrix and a translation calibration matrix. Therefore, the method provided by the invention can improve the robustness, accuracy and stability of the calibration result and the real-time performance of the calibration process.

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor calibration, and specifically relates to a multi-camera and IMU collaborative calibration method and device. Background Art

[0002] Camera and IMU (Inertial Measurement Unit) co-calibration is a technique for accurately estimating external parameters between the camera and IMU. Camera and IMU are two different types of sensors, the camera is used to obtain image data, while the IMU measures the acceleration and angular velocity of the object. The purpose of camera and IMU co-calibration is to determine the accurate conversion relationship between them so that accurate attitude and motion information can be obtained when fusing visual and inertial data.

[0003] Camera and IMU co-calibration has a wide range of applications in many fields. These include robot navigation, augmented reality (AR), virtual reality (VR), and autonomous vehicles. In robot navigation, camera and IMU co-calibration can be used to provide accurate position and posture information to help robots locate and avoid obstacles in complex environments. In augmented reality and virtual reality, camera and IMU co-calibration provides the user's head posture and motion information for real-time rendering of virtual scenes. For autonomous vehicles, camera and IMU co-calibration can provide the vehicle's precise position and posture data to assist navigation and path planning.

[0004] The importance of camera and IMU collaborative calibration is mainly reflected in the following aspects. First, cameras and IMUs are two types of sensors with different measurement principles and coordinate systems, so accurate calibration is required to effectively fuse their data. Relying solely on camera data may not provide accurate posture and motion information, while the joint use of cameras and IMUs makes up for each other's shortcomings and improves the performance and accuracy of the entire system. Secondly, camera and IMU collaborative calibration provides more accurate 3D reconstruction and posture estimation, which has important practical value in many application fields. Finally, camera and IMU collaborative calibration also helps solve the problem of time synchronization between sensors to ensure data consistency and accuracy.

[0005] From a technical perspective, the collaborative calibration of cameras and IMUs involves issues such as external parameter calculation, distortion correction, coordinate system conversion, and error compensation between the camera and IMU. During the calibration process, known reference objects or features are usually required for matching and calculation. The accuracy of the calibration results directly affects the effect of subsequent data fusion and application, so the stability and reliability of the calibration process are very important.

[0006] Although camera and IMU collaborative calibration technology has made great progress in practice, there are still some problems and challenges:

[0007] The first is the problem related to feature points: a) Feature extraction and matching: The robustness and accuracy of feature extraction is still a challenge, especially in complex dynamic scenes. There are problems such as mismatching and occlusion during feature matching, which may lead to inaccurate calibration results. b) Feature point selection: The selection of feature points is crucial to the accuracy and stability of calibration. However, how to select appropriate feature points to obtain high-quality calibration results is still an open problem.

[0008] The second is motion-related issues: a) Accuracy of motion model: The accuracy of calibration is limited by the motion model of the camera and IMU. If the motion model assumptions are inaccurate or the nonlinear factors of motion are not considered, it may lead to errors in the calibration results. b) Challenges in dynamic scenes: In dynamic scenes, the posture and motion of the camera and IMU will change rapidly, making it difficult to accurately obtain the correspondence between them.

[0009] The third is optimization-related issues: a) The impact of initial values ​​on results: The optimization method needs to provide appropriate initial parameter values, otherwise it may fall into a local optimal solution. Determining the exact initial value is still a complex problem. b) Computational complexity: Some optimization methods have high computational complexity when processing large-scale data, which affects the real-time and practicality of calibration.

[0010] Fourth, data acquisition and synchronization issues: The time synchronization error between the camera and IMU will affect the calibration results. Accurate time synchronization is very important for obtaining accurate calibration results.

[0011] These problems limit the application of camera and IMU collaborative calibration technology. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a multi-camera and IMU collaborative calibration method and device, which is conducive to improving the robustness, accuracy and stability of the calibration results, as well as the real-time performance of the calibration process.

[0013] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a multi-camera and IMU collaborative calibration method, the method comprising:

[0014] S1, a first auxiliary camera, a second auxiliary camera, a third auxiliary camera and a fourth auxiliary camera are dispersedly arranged around a preset main camera and an IMU; the main camera is fixedly connected to the IMU; the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera together constitute an auxiliary camera array;

[0015] S2, using the auxiliary camera array, obtaining a horizontal focal length set, a vertical focal length set, a relative posture matrix set, and a relative translation matrix set;

[0016] The horizontal focal length set includes a first horizontal focal length, a second horizontal focal length, a third horizontal focal length and a fourth horizontal focal length; the vertical focal length set includes a first vertical focal length, a second vertical focal length, a third vertical focal length and a fourth vertical focal length; the relative posture matrix set includes a first relative posture matrix, a second relative posture matrix and a third relative posture matrix; the relative translation matrix set includes a first relative translation matrix, a second relative translation matrix and a third relative translation matrix;

[0017] S3, using the IMU and the auxiliary camera array, acquiring an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence, and an IMU acquisition time sequence;

[0018] The auxiliary camera image sequence set includes a first auxiliary camera image sequence, a second auxiliary camera image sequence, a third auxiliary camera image sequence and a fourth auxiliary camera image sequence; the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence each include N auxiliary camera images; N is an integer greater than 1;

[0019] S4, processing the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates;

[0020] S5, acquiring a posture calibration matrix and a translation calibration matrix based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates.

[0021] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of the IMU and the auxiliary camera array to obtain an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence, and an IMU acquisition time sequence includes:

[0022] S31, connecting the IMU and the auxiliary camera array to a preset clock synchronization unit; the clock synchronization unit is used to generate a time synchronization signal;

[0023] S32, moving the main camera and the IMU within the field of view of the auxiliary camera; the field of view of the auxiliary camera is the intersection of the fields of view of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera;

[0024] S33, using the IMU to perform acquisition based on the time synchronization signal to obtain the acceleration acquisition sequence, the angular velocity acquisition sequence and the IMU acquisition time series;

[0025] S34, using the auxiliary camera array to perform synchronous shooting based on the time synchronization signal to obtain the auxiliary camera image sequence set and the camera acquisition time sequence.

[0026] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set, and the relative translation matrix set to obtain the main camera world coordinates includes:

[0027] S41, using the auxiliary camera image sequence set to obtain a main camera feature point descriptor;

[0028] S42, using a feature point extraction model, respectively processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence, and the fourth auxiliary camera image sequence to obtain extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera;

[0029] The extracted feature point descriptor sequence includes more than one extracted feature point descriptor;

[0030] S43, respectively combining the extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera with the feature point descriptor of the main camera to obtain four combined feature point descriptor sequences;

[0031] S44, using a feature point matching model, processing each of the combined feature point description subsequences respectively to obtain a corresponding similarity sequence; the similarity sequence includes more than one similarity value, and the length is the same as the extracted feature point description subsequence of the corresponding feature point description group;

[0032] S45, using each of the similarity sequences respectively, screening the extracted feature point descriptor sequence of the corresponding feature point description group to obtain four matching feature point descriptors;

[0033] S46, respectively obtaining pixel coordinates of the four matching feature point descriptors to obtain four matching pixel coordinates;

[0034] S47, using a normalized pixel coordinate calculation model, processing the matching pixel coordinates, the horizontal focal length set, and the vertical focal length set to obtain a normalized pixel coordinate set; the normalized pixel coordinate set includes 4 normalized pixel coordinates;

[0035] The normalized pixel coordinate calculation model is:

[0036]

[0037] Wherein, x1, x2, x3 and x4 are the horizontal components of the coordinates of the four matching pixels; y1, y2, y3 and y4 are the vertical components of the coordinates of the four matching pixels; x n,1 、x n,2 、x n,3 、x n,4 are the horizontal components of the four normalized pixel coordinates respectively; n,1 ,y n,2 ,y n,3 and n,4 are the vertical components of the four normalized pixel coordinates respectively; x,1 、f x,2 、f x,3 and f x,4 are the first horizontal focal length, the second horizontal focal length, the third horizontal focal length and the fourth horizontal focal length respectively; y,1 、f y,2 、f y,3 and f y,4 are respectively the first vertical focal length, the second vertical focal length, the third vertical focal length and the fourth vertical focal length;

[0038] S48, processing the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set to obtain the main camera world coordinates.

[0039] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the use of the feature point extraction model to process the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence respectively to obtain the extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera includes:

[0040] S421, using the first convolution layer of the feature point extraction model, processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence, or the fourth auxiliary camera image sequence to obtain first convolution information;

[0041] S422, using a first normalization layer of the feature point extraction model, performing a batch normalization operation on the first convolution information to obtain first normalized information;

[0042] S423, using the first pooling layer of the feature point extraction model, performing a pooling operation on the first normalized information to obtain first pooled information;

[0043] S424, using the second convolution layer of the feature point extraction model to perform convolution processing on the first pooling information to obtain second convolution information;

[0044] S425, using the second normalization layer of the feature point extraction model, performing a batch normalization operation on the second convolution information to obtain second normalized information;

[0045] S426, using the second pooling layer of the feature point extraction model, performing a pooling operation on the second normalized information to obtain second pooled information;

[0046] S427, using the first fully connected layer of the feature point extraction model, performing fully connected processing on the second pooling information to obtain first fully connected information;

[0047] S428: Perform full-connection processing on the first fully-connected information by using the second fully-connected layer of the feature point extraction model to obtain the extracted feature point descriptor sequence of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, or the fourth auxiliary camera.

[0048] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of the feature point matching model to process each of the combined feature point description subsequences respectively to obtain a corresponding similarity sequence includes:

[0049] S441, using the third convolution layer of the feature point matching model, performing convolution processing on the combined feature point descriptor sequence to obtain third convolution information;

[0050] S442, using the third normalization layer of the feature point matching model, performing a batch normalization operation on the third convolution information to obtain third normalized information;

[0051] S443, using the third pooling layer of the feature point matching model, performing a pooling operation on the third normalized information to obtain third pooled information;

[0052] S444, using the fourth convolution layer of the feature point matching model to perform convolution processing on the third pooling information to obtain fourth convolution information;

[0053] S445, using the fourth normalization layer of the feature point matching model, performing a batch normalization operation on the fourth convolution information to obtain fourth normalized information;

[0054] S446, using the fourth pooling layer of the feature point matching model, performing a pooling operation on the fourth normalized information to obtain fourth pooled information;

[0055] S447, using the third fully connected layer of the feature point matching model, performing fully connected processing on the fourth pooling information to obtain third fully connected information;

[0056] S448: Use the fourth fully connected layer of the feature point matching model to perform fully connected processing on the third fully connected information to obtain the corresponding similarity sequence.

[0057] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set to obtain the main camera world coordinates includes:

[0058] S481, establishing a depth coordinate calculation model using the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set;

[0059] The depth coordinate calculation model is:

[0060]

[0061] In the formula, x n,1 、x n,2 、x n,3 、x n,4 are the horizontal components of the four normalized pixel coordinates respectively; n,1 ,y n,2 ,y n,3 and n,4 are the vertical components of the four normalized pixel coordinates respectively; n,1 、z n,1 、z n,1 and z n,4 are respectively the first depth coordinate, the second depth coordinate, the third depth coordinate and the fourth depth coordinate; R1, R2 and R3 are respectively the first relative posture matrix, the second relative posture matrix and the third relative posture matrix; T1, T2 and T3 are respectively the first relative translation matrix, the second relative translation matrix and the third relative translation matrix;

[0062] S482, solving a depth coordinate calculation model to obtain the first depth coordinate, the second depth coordinate, the third depth coordinate, and the fourth depth coordinate;

[0063] S483, using a world coordinate calculation model, processing the relative posture matrix set, the relative translation matrix set, the first depth coordinate, the second depth coordinate, the third depth coordinate, and the fourth depth coordinate to obtain three world coordinates;

[0064] The world coordinate calculation model is:

[0065]

[0066] In the formula, (x w,1 ,y w,1 ,z w,1 )、(x w,2 ,y w,2 ,z w,2 ) and (x w,3 ,y w,3 ,z w,3 ) are the three world coordinates respectively;

[0067] S484, using a main camera world coordinate calculation model, processing the three world coordinates to obtain the main camera world coordinates;

[0068] The main camera world coordinate calculation model is:

[0069]

[0070] In the formula, x w ,y w and z w are respectively the horizontal component, vertical component and depth component of the main camera world coordinates.

[0071] As an optional implementation manner, in the first aspect of the embodiment of the present invention, acquiring the attitude calibration matrix and the translation calibration matrix based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates includes:

[0072] S51, determining that the current number of cycles is 1;

[0073] S52, processing the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence, and the main camera world coordinates to obtain an IMU position sequence, a main camera position sequence, and an IMU attitude sequence, and initialize the attitude calibration matrix and the translation calibration matrix;

[0074] S53, using the trajectory error calculation model, the IMU position sequence P imu , the attitude calibration matrix Rcam_imu , the main camera position sequence P cam , the translation calibration matrix T cam_imu and the IMU attitude sequence R imu Processing is performed to obtain a trajectory error F, a first error matrix G, and a second error matrix H;

[0075] The trajectory error calculation model is:

[0076]

[0077] In the formula, α is the preset weight coefficient, and i is the element number;

[0078] S54, using the calibration matrix to update the model, processing the attitude calibration matrix and the translation calibration matrix to obtain updated attitude calibration matrix and translation calibration matrix;

[0079] The calibration matrix update model is:

[0080]

[0081] In the formula, and R cam_imu are the values ​​of the attitude calibration matrix before and after updating, and T cam_imu are respectively the values ​​of the translation calibration matrix before and after updating;

[0082] S55, adding 1 to the current cycle number;

[0083] S56, repeat S53 to S55 until the trajectory error value is less than a preset target threshold, or the current number of cycles reaches a preset maximum number of cycles.

[0084] As an optional implementation, in the first aspect of the embodiment of the present invention, the processing based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates to obtain the attitude calibration matrix and the translation calibration matrix includes:

[0085] S521, interpolating the acceleration acquisition sequence and the angular velocity acquisition sequence to obtain an interpolation time series, an acceleration interpolation sequence, and an angular velocity interpolation sequence;

[0086] S522, establishing a calibration equation using the camera acquisition time series and the IMU acquisition time series;

[0087] The calibration equation is:

[0088] T1=T2+δt

[0089] Wherein, T1, T2 and δ are column vectors composed of the camera acquisition time series, the IMU acquisition time series and the initial time offset sequence respectively;

[0090] S523, solving the calibration equation to obtain the initial time offset sequence;

[0091] S524, adding the acceleration acquisition sequence to the elements in the initial time offset sequence correspondingly to obtain the acceleration calibration sequence;

[0092] S525, adding the angular velocity acquisition sequence and the elements of the initial time offset sequence correspondingly to obtain the angular velocity calibration sequence;

[0093] S526, performing fusion processing on the angular velocity calibration sequence to obtain an initial IMU attitude matrix;

[0094] S527, using the acceleration calibration sequence, the angular velocity calibration sequence and the initial IMU attitude matrix, obtain an IMU position sequence, a main camera position sequence and an IMU attitude sequence, as well as initial values ​​of the attitude calibration matrix and the translation calibration matrix.

[0095] A second aspect of an embodiment of the present invention discloses a multi-camera and IMU collaborative calibration device, the device comprising:

[0096] The auxiliary camera calibration module 201 is used to obtain a horizontal focal length set, a vertical focal length set, a relative posture matrix set and a relative translation matrix set using an auxiliary camera array.

[0097] The data acquisition module 202 is used to obtain the acceleration acquisition sequence, angular velocity acquisition sequence, auxiliary camera image sequence set, camera acquisition time sequence and IMU acquisition time sequence acquired by the IMU and the auxiliary camera array;

[0098] A world coordinate calculation module 203 is used to process the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates;

[0099] The calibration calculation module 204 is used to process based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates to obtain the attitude calibration matrix and the translation calibration matrix.

[0100] A third aspect of an embodiment of the present invention discloses another multi-camera and IMU collaborative calibration device, the device comprising:

[0101] A memory storing executable program code;

[0102] a processor coupled to the memory;

[0103] The processor calls the executable program code stored in the memory to execute some or all of the steps in a multi-camera and IMU collaborative calibration method disclosed in the first aspect of an embodiment of the present invention.

[0104] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in a multi-camera and IMU collaborative calibration method disclosed in the first aspect of an embodiment of the present invention.

[0105] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0106] (1) The present invention uses four auxiliary cameras to observe the movement of the main camera from different perspectives to obtain four auxiliary camera image sequences. After being processed by a feature point extraction model and a feature point matching model, a matching feature point descriptor is obtained. The matching feature point descriptor is used to sequentially obtain matching pixel coordinates, normalized pixel coordinates, depth coordinates, world coordinates and main camera world coordinates, which can improve the robustness, accuracy and stability of the calibration results. The algorithm is simple and has high real-time performance.

[0107] (2) The IMU and the auxiliary camera collect data based on the time synchronization signal to obtain the acceleration collection sequence, angular velocity collection sequence, IMU collection time series and camera collection time series; the calibration equation is established and solved using the IMU collection time series and the camera collection time series, and the collected data is time-calibrated after obtaining the initial time offset sequence. The acceleration calibration sequence and angular velocity calibration sequence obtained after calibration are used to initialize the attitude calibration matrix and the translation calibration matrix, thereby eliminating the error caused by the time asynchrony between the camera and the IMU, and improving the accuracy of the initial estimate of the calibration result;

[0108] (3) By cyclically calculating the trajectory error, the first error matrix and the second error matrix, and updating the attitude calibration matrix and the translation calibration matrix, the calibration result is optimized with low algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0110] Figure 1A schematic diagram of a multi-camera and IMU collaborative calibration method proposed in the present invention;

[0111] Figure 2 A schematic diagram of the movement trajectory of a multi-camera and IMU collaborative calibration method proposed in the present invention;

[0112] Figure 3 It is a structural schematic diagram of a multi-camera and IMU collaborative calibration device disclosed in an embodiment of the present invention;

[0113] Figure 4 It is a structural schematic diagram of another multi-camera and IMU collaborative calibration device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0115] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0116] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein are combined with other embodiments.

[0117] Embodiment 1

[0118] See also Figure 1 , Figure 1 is a flow chart of a multi-camera and IMU collaborative calibration method disclosed in an embodiment of the present invention. Figure 1The multi-camera and IMU collaborative calibration method described is applicable to the field of camera and IMU collaborative calibration, such as robot navigation, augmented reality and virtual reality, etc., and is not limited in the embodiments of the present invention. Figure 1 As shown, the method includes:

[0119] S1, a first auxiliary camera, a second auxiliary camera, a third auxiliary camera and a fourth auxiliary camera are dispersedly arranged around a preset main camera and an IMU; the main camera is fixedly connected to the IMU; the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera together constitute an auxiliary camera array;

[0120] It should be noted that the positions of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera are independent of each other, and the lenses are all facing the main camera;

[0121] S2, using the auxiliary camera array, obtaining a horizontal focal length set, a vertical focal length set, a relative posture matrix set, and a relative translation matrix set;

[0122] The above-mentioned horizontal focal length set includes a first horizontal focal length, a second horizontal focal length, a third horizontal focal length and a fourth horizontal focal length; the above-mentioned vertical focal length set includes a first vertical focal length, a second vertical focal length, a third vertical focal length and a fourth vertical focal length; the above-mentioned relative posture matrix set includes a first relative posture matrix, a second relative posture matrix and a third relative posture matrix; the above-mentioned relative translation matrix set includes a first relative translation matrix, a second relative translation matrix and a third relative translation matrix;

[0123] It should be noted that the first horizontal focal length, the second horizontal focal length, the third horizontal focal length and the fourth horizontal focal length are respectively the horizontal focal lengths of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera; the first vertical focal length, the second vertical focal length, the third vertical focal length and the fourth vertical focal length are respectively the vertical focal lengths of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera; the first relative posture matrix, the second relative posture matrix and the third relative posture matrix are respectively used to represent the rotation degree of the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera relative to the first auxiliary camera; the first relative translation matrix, the second relative translation matrix and the third relative translation matrix are respectively used to represent the offset degree of the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera relative to the first auxiliary camera;

[0124] S3, using the IMU and the auxiliary camera array, to obtain an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence, and an IMU acquisition time sequence;

[0125] The auxiliary camera image sequence set includes a first auxiliary camera image sequence, a second auxiliary camera image sequence, a third auxiliary camera image sequence and a fourth auxiliary camera image sequence; the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence each include N auxiliary camera images; N is an integer greater than 1;

[0126] It should be noted that the auxiliary camera images in the above-mentioned first auxiliary camera image sequence, second auxiliary camera image sequence, third auxiliary camera image sequence and fourth auxiliary camera image sequence are captured by the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera respectively;

[0127] S4, processing the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates;

[0128] S5, based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates, obtain the attitude calibration matrix and the translation calibration matrix.

[0129] In an optional embodiment, the above-mentioned use of the auxiliary camera array to obtain a horizontal focal length set, a vertical focal length set, a relative posture matrix set and a relative translation matrix set includes:

[0130] S21, using a first auxiliary camera, a second auxiliary camera, a third auxiliary camera and a fourth auxiliary camera to photograph a preset calibration plate to obtain a calibration image set; the calibration image set includes more than one calibration image;

[0131] S22, preprocessing all calibration images in the calibration image set to obtain a preprocessed image set;

[0132] Optionally, the above preprocessing includes distortion removal and noise reduction, which is implemented by calling MATLAB's Camera Calibrator;

[0133] S23, performing corner point detection on the preprocessed image set to obtain a corner point coordinate set of the calibration plate;

[0134] Optionally, the above corner point detection is implemented by calling the detectHarrisFeatures function in Computer Vision Toolbox of MATLAB;

[0135] S24, performing a calibration operation using the corner point coordinate set and the length and width values ​​of the calibration plate to obtain a horizontal focal length set and a vertical focal length set;

[0136] Optionally, the above calibration operation is implemented by calling MATLAB's Camera Calibrator;

[0137] S25, performing posture estimation processing on the corner point coordinate set to obtain a relative posture matrix set and a relative translation matrix set.

[0138] Optionally, the above-mentioned attitude estimation process is implemented by calling the camera attitude estimation toolbox of MATLAB.

[0139] In another optional embodiment, the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera are used to photograph a preset calibration plate to obtain a calibration image set, including:

[0140] S211, using a first auxiliary camera, a second auxiliary camera, a third auxiliary camera, and a fourth auxiliary camera to respectively photograph the calibration plate to obtain four calibration images;

[0141] Preferably, the length and width of the calibration plate are 11 mm and 8.60 mm respectively;

[0142] S212, changing the position of the calibration plate;

[0143] S213, repeating S211 to S212 for N0 times to obtain a calibration image set consisting of 4×N0 calibration images;

[0144] Preferably, the above N0 is 30.

[0145] It should be noted that the complete calibration plate is visible in all calibration images.

[0146] In another optional embodiment, the above-mentioned use of the IMU and the auxiliary camera array to obtain the acceleration acquisition sequence, the angular velocity acquisition sequence, the auxiliary camera image sequence set, the camera acquisition time sequence and the IMU acquisition time sequence includes:

[0147] S31, connecting the IMU and the auxiliary camera array to a preset clock synchronization unit; the clock synchronization unit is used to generate a time synchronization signal;

[0148] S32, moving the main camera and the IMU within the field of view of the auxiliary camera; the field of view of the auxiliary camera is the intersection of the fields of view of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera;

[0149] It should be noted that if Figure 2 As shown in the figure, although the main camera and IMU are moved according to the preset trajectory within the field of view of the auxiliary camera, the actual trajectories of the two have a small deviation from the preset trajectory;

[0150] S33, using the IMU to collect data based on the time synchronization signal to obtain an acceleration collection sequence, an angular velocity collection sequence, and an IMU collection time series;

[0151] It should be noted that, during the acquisition process, the above-mentioned IMU uses the time synchronization signal to obtain the timestamp of each acquisition moment to obtain the IMU acquisition time series;

[0152] S34, using the auxiliary camera array to perform synchronous shooting based on the time synchronization signal to obtain an auxiliary camera image sequence set and a camera acquisition time sequence.

[0153] Preferably, the frame rates of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera are the same and are all lower than the sampling frequency of the IMU.

[0154] It should be noted that the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera all shoot at the same time, and during the shooting process, the time stamp of each shooting moment is obtained by using the time synchronization signal to obtain the camera acquisition time sequence. In addition, in the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence, the auxiliary camera images with the same number are shot at the same time.

[0155] In yet another optional embodiment, the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set, and the relative translation matrix set are processed to obtain the main camera world coordinates, including:

[0156] S41, using the auxiliary camera image sequence set to obtain the main camera feature point descriptor;

[0157] It should be noted that the above main camera feature point descriptor is used to describe the position of the main camera in the auxiliary camera image;

[0158] S42, using a feature point extraction model, respectively processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence to obtain extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera;

[0159] The above-mentioned extracted feature point descriptor sequence includes more than one extracted feature point descriptor;

[0160] It should be noted that the above-mentioned extracted feature point descriptor is used to describe the local image area around the key point extracted from the auxiliary camera image;

[0161] S43, respectively combining the extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera with the feature point descriptor of the main camera to obtain four combined feature point descriptor sequences;

[0162] S44, using the feature point matching model, processing each combined feature point description subsequence respectively to obtain a corresponding similarity sequence; the similarity sequence includes more than one similarity value, and the length is the same as the extracted feature point description subsequence of the corresponding feature point description group;

[0163] It should be noted that the above similarity value is used to describe the similarity between the main camera feature point descriptor and the extracted feature point descriptor with the same sequence number as the similarity value in the feature point descriptor sequence of the corresponding feature point description group;

[0164] S45, using each similarity sequence respectively, screening the extracted feature point descriptor sequence of the corresponding feature point description group to obtain 4 matching feature point descriptors;

[0165] Optionally, in the above screening process, firstly, the highest similarity value in the similarity sequence is obtained, and then the extracted feature point descriptor corresponding to the highest similarity value in the extracted feature point descriptor sequence corresponding to the similarity sequence is obtained to obtain the matching feature point descriptor;

[0166] S46, respectively obtaining pixel coordinates of four matching feature point descriptors to obtain four matching pixel coordinates;

[0167] Optionally, the above matching pixel coordinates are obtained by calling the OpenCV function library;

[0168] S47, using a normalized pixel coordinate calculation model, processing the matching pixel coordinates, the horizontal focal length set, and the vertical focal length set to obtain a normalized pixel coordinate set; the normalized pixel coordinate set includes 4 normalized pixel coordinates;

[0169] The above normalized pixel coordinate calculation model is:

[0170]

[0171] Where x1, x2, x3 and x4 are the horizontal components of the four matching pixel coordinates; y1, y2, y3 and y4 are the vertical components of the four matching pixel coordinates; x n,1 、x n,2 、x n,3 、x n,4 They are the horizontal components of the 4 normalized pixel coordinates; y n,1 ,y n,2 ,y n,3 andn,4 are the vertical components of the four normalized pixel coordinates; f x,1 、f x,2 、f x,3 and f x,4 are the first horizontal focal length, the second horizontal focal length, the third horizontal focal length and the fourth horizontal focal length respectively; f y,1 、f y,2 、f y,3 and f y,4 are respectively a first vertical focal length, a second vertical focal length, a third vertical focal length and a fourth vertical focal length;

[0172] S48, processing the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set to obtain the main camera world coordinates.

[0173] In yet another optional embodiment, the above-mentioned use of the auxiliary camera image sequence set to obtain the main camera feature point descriptor includes:

[0174] S411, randomly selecting an auxiliary camera image from the auxiliary camera image sequence set to obtain an image to be judged;

[0175] S412, obtaining the center coordinates of the camera area of ​​the main camera on the image to be determined, and obtaining the main camera mark coordinates;

[0176] S413: Obtain a main camera feature point descriptor based on the image to be determined and the main camera marker coordinates.

[0177] Optionally, the main camera feature point descriptor is calculated by calling the OpenCV function library.

[0178] It should be noted that the main camera feature point descriptor is used to describe the local image area around the main camera marker coordinates. First, a 16*16 neighborhood is selected with the main camera marker coordinates as the center, and then the neighborhood is divided into 4*4 sub-regions. Then the gradient direction is divided into 8 intervals. In this way, a 4*4*8=128-dimensional feature vector will be obtained in each sub-region, and the vector element size is the weight of each gradient direction interval.

[0179] In another optional embodiment, the feature point extraction model is used to process the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence respectively to obtain the extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera, including:

[0180] S421, using the first convolution layer of the feature point extraction model, processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence or the fourth auxiliary camera image sequence to obtain first convolution information;

[0181] S422, using a first normalization layer of the feature point extraction model, performing a batch normalization operation on the first convolution information to obtain first normalized information;

[0182] S423, using the first pooling layer of the feature point extraction model, performing a pooling operation on the first normalized information to obtain first pooled information;

[0183] S424, using the second convolution layer of the feature point extraction model to perform convolution processing on the first pooling information to obtain second convolution information;

[0184] S425, using the second normalization layer of the feature point extraction model, performing a batch normalization operation on the second convolution information to obtain second normalized information;

[0185] S426, using the second pooling layer of the feature point extraction model, performing a pooling operation on the second normalized information to obtain second pooled information;

[0186] S427, using the first fully connected layer of the feature point extraction model, performing fully connected processing on the second pooling information to obtain first fully connected information;

[0187] S428, using the second fully connected layer of the feature point extraction model, performing fully connected processing on the first fully connected information to obtain a sequence of extracted feature point descriptors of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, or the fourth auxiliary camera.

[0188] In another optional embodiment, the feature point matching model is used to process each combined feature point description subsequence separately to obtain a corresponding similarity sequence, including:

[0189] S441, using the third convolution layer of the feature point matching model, performing convolution processing on the combined feature point descriptor sequence to obtain third convolution information;

[0190] S442, using the third normalization layer of the feature point matching model, performing a batch normalization operation on the third convolution information to obtain third normalized information;

[0191] S443, using the third pooling layer of the feature point matching model, performing a pooling operation on the third normalized information to obtain third pooled information;

[0192] S444, using the fourth convolution layer of the feature point matching model, performing convolution processing on the third pooling information to obtain fourth convolution information;

[0193] S445, using a fourth normalization layer of the feature point matching model, performing a batch normalization operation on the fourth convolution information to obtain fourth normalized information;

[0194] S446, using a fourth pooling layer of the feature point matching model, performing a pooling operation on the fourth normalized information to obtain fourth pooled information;

[0195] S447, using the third fully connected layer of the feature point matching model, performing fully connected processing on the fourth pooled information to obtain third fully connected information;

[0196] S448, using the fourth fully connected layer of the feature point matching model, perform fully connected processing on the third fully connected information to obtain a corresponding similarity sequence.

[0197] In yet another optional embodiment, the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set are processed to obtain the main camera world coordinates, including:

[0198] S481, establishing a depth coordinate calculation model using the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set;

[0199] The above depth coordinate calculation model is:

[0200]

[0201] In the formula, x n,1 、x n,2 、x n,3 、x n,4 They are the horizontal components of the 4 normalized pixel coordinates; y n,1 ,y n,2 ,y n,3 and n,4 They are the vertical components of the 4 normalized pixel coordinates; z n,1 、z n,1 、z n,1 and z n,4 are respectively the first depth coordinate, the second depth coordinate, the third depth coordinate and the fourth depth coordinate; R1, R2 and R3 are respectively the first relative posture matrix, the second relative posture matrix and the third relative posture matrix; T1, T2 and T3 are respectively the first relative translation matrix, the second relative translation matrix and the third relative translation matrix;

[0202] S482, solving a depth coordinate calculation model to obtain a first depth coordinate, a second depth coordinate, a third depth coordinate, and a fourth depth coordinate;

[0203] S483, using a world coordinate calculation model, processing the relative posture matrix set, the relative translation matrix set, the first depth coordinate, the second depth coordinate, the third depth coordinate, and the fourth depth coordinate to obtain three world coordinates;

[0204] The above world coordinate calculation model is:

[0205]

[0206] In the formula, (x w,1 ,y w,1 ,z w,1 )、(x w,2 ,y w,2 ,z w,2 ) and (x w,3 ,y w,3 ,z w,3 ) are three world coordinates respectively;

[0207] S484, using the main camera world coordinate calculation model, processing the three world coordinates to obtain the main camera world coordinates;

[0208] The above main camera world coordinate calculation model is:

[0209]

[0210] In the formula, x w ,y w and z w They are the horizontal component, vertical component and depth component of the main camera's world coordinates respectively.

[0211] In yet another optional embodiment, the above-mentioned acquisition of the attitude calibration matrix and the translation calibration matrix based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates includes:

[0212] S51, determining that the current number of cycles is 1;

[0213] S52, processing the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence, and the main camera world coordinates to obtain the IMU position sequence, the main camera position sequence, and the IMU attitude sequence, and initialize the attitude calibration matrix and the translation calibration matrix;

[0214] S53, using the trajectory error calculation model, the IMU position sequence P imu , attitude calibration matrix R cam_imu , main camera position sequence P cam , translation calibration matrix T cam_imu and IMU attitude sequence R imuProcessing is performed to obtain a trajectory error F, a first error matrix G, and a second error matrix H;

[0215] The above trajectory error calculation model is:

[0216]

[0217] In the formula, α is the preset weight coefficient, P imu (i) P cam (i) and R imu (i) are the corresponding sequences P imu , P cam and R imu The i-th element in ;

[0218] It should be noted that and The target error F versus R cam_imu (i) is a sequence of first-order partial derivatives and second-order partial derivatives, and F vs. T cam_imu The sequence of first-order partial derivatives and second-order partial derivatives of ;

[0219] S54, using the calibration matrix to update the model, processing the attitude calibration matrix and the translation calibration matrix to obtain updated attitude calibration matrix and translation calibration matrix;

[0220] The above calibration matrix update model is:

[0221]

[0222] In the formula, and R cam_imu are the values ​​of the attitude calibration matrix before and after updating, and T cam_imu They are the values ​​of the translation calibration matrix before and after updating respectively;

[0223] S55, adding 1 to the current cycle number;

[0224] S56, repeat S53 to S55 until the trajectory error value is less than a preset target threshold, or the current number of cycles reaches a preset maximum number of cycles.

[0225] In another optional embodiment, the acceleration acquisition sequence, angular velocity acquisition sequence, camera acquisition time sequence, IMU acquisition time sequence, and main camera world coordinates are processed to obtain an IMU position sequence, a main camera position sequence, and an IMU attitude sequence, and initialize an attitude calibration matrix and a translation calibration matrix, including:

[0226] S521, interpolating the acceleration acquisition sequence and the angular velocity acquisition sequence to obtain an interpolation time series, an acceleration interpolation sequence, and an angular velocity interpolation sequence;

[0227] Optionally, the above interpolation is linear interpolation or polynomial interpolation;

[0228] It should be noted that the lengths of the above interpolation time series, acceleration interpolation sequence and angular velocity interpolation sequence are the same, and the differences between adjacent elements in the interpolation time series and the camera acquisition time series are the same; the above interpolation time series is used to describe the acquisition time of each element in the acceleration interpolation sequence and the angular velocity interpolation sequence;

[0229] S522, establishing a calibration equation using the camera acquisition time series and the IMU acquisition time series;

[0230] The calibration equation above is:

[0231] T1=T2+δt

[0232] Where T1, T2 and δ are column vectors composed of camera acquisition time series, IMU acquisition time series and initial time offset series respectively;

[0233] S523, solving the calibration equation to obtain an initial time offset sequence;

[0234] Optionally, the least square method is used to solve the calibration equation;

[0235] S524, adding the acceleration acquisition sequence to the elements in the initial time offset sequence to obtain an acceleration calibration sequence;

[0236] S525, adding the angular velocity acquisition sequence and the elements of the initial time offset sequence correspondingly to obtain an angular velocity calibration sequence;

[0237] S526, performing fusion processing on the angular velocity calibration sequence to obtain an initial IMU attitude matrix;

[0238] Optionally, the fusion processing is Kalman filtering;

[0239] S527, using the acceleration calibration sequence, the angular velocity calibration sequence and the initial IMU attitude matrix, obtain the IMU position sequence, the main camera position sequence and the IMU attitude sequence, as well as the initial values ​​of the attitude calibration matrix and the translation calibration matrix.

[0240] In yet another optional embodiment, the acceleration calibration sequence, the angular velocity calibration sequence and the initial IMU attitude matrix are used to obtain the IMU position sequence, the main camera position sequence and the IMU attitude sequence, as well as the initial values ​​of the attitude calibration matrix and the translation calibration matrix, including:

[0241] S5271, integrating the acceleration calibration sequence and the angular velocity calibration sequence twice respectively to obtain an IMU position increment sequence and an IMU attitude increment sequence;

[0242] S5272, based on the IMU position increment sequence and the main camera world coordinates, obtain the IMU position sequence;

[0243] S5273, acquiring an IMU attitude sequence based on the IMU attitude increment sequence and the initial IMU attitude matrix;

[0244] S5274, using the IMU position sequence and the main camera world coordinates, initialize the attitude calibration matrix and the translation calibration matrix;

[0245] S5275, using the main camera position calculation model, the attitude calibration matrix R cam_imu , translation calibration matrix T cam_imu Process the IMU displacement increment sequence to obtain the main camera position sequence;

[0246] The above main camera position calculation model is:

[0247] P cam (i) = T cam_imu R cam_imu dP imu (i)

[0248] Where P cam (i) and dP imu (i) are the i-th element in the main camera position sequence and the IMU displacement increment sequence, respectively.

[0249] In yet another optional embodiment, the obtaining of the IMU position sequence based on the IMU position increment sequence and the main camera world coordinates includes:

[0250] S52721, initialize the number of iterations t1 to 1, and initialize the IMU position sequence to include only the IMU initial position;

[0251] S52722, obtain the t1th element of the IMU position increment sequence to obtain the current position increment;

[0252] S52723, adding the current position increment to the last element of the IMU position sequence, and inserting it to the end of the IMU position sequence;

[0253] S52724, repeat S52722 to S52723 for N1 times to obtain the final IMU position sequence; the above N1 is the length of the IMU position increment sequence.

[0254] In yet another optional embodiment, the above-mentioned obtaining the IMU attitude sequence based on the IMU attitude increment sequence and the initial IMU attitude matrix includes:

[0255] S52731, initialize the number of iterations t2 to 1, and initialize the IMU attitude sequence to include only the initial IMU attitude matrix;

[0256] S52732, obtain the t2th element of the IMU attitude increment sequence to obtain the current attitude increment;

[0257] S52733, adding the current attitude increment to the last element of the IMU attitude sequence, and inserting it to the end of the IMU attitude sequence;

[0258] S52734, repeat S52732 to S52733 N2 times to obtain the final IMU attitude sequence; the above N2 is the length of the IMU attitude increment sequence.

[0259] In yet another optional embodiment, the above-mentioned use of the IMU position sequence and the main camera world coordinates to initialize the attitude calibration matrix and the translation calibration matrix includes:

[0260] S52741, using IMU position sequence P imu and the main camera world coordinate P w , establish the initial estimation equation;

[0261] The above initial estimation equation is:

[0262] P w =R0P imu +T0

[0263] In the formula, R0 and T0 are the initial attitude matrix and initial translation matrix respectively;

[0264] S52742, solve the initial estimation equation to obtain the initial attitude matrix and the initial translation matrix;

[0265] S52743, determine the attitude calibration matrix and the translation calibration matrix, which are the initial attitude matrix and the initial translation matrix respectively.

[0266] It can be seen that the multi-camera and IMU collaborative calibration method described in the embodiment of the present invention is implemented, and four auxiliary cameras are used to observe the movement of the main camera from different perspectives to obtain four auxiliary camera image sequences. After being processed by the feature point extraction model and the feature point matching model, the matching feature point descriptor is obtained, and the matching feature point descriptor is used to sequentially obtain the matching pixel coordinates, normalized pixel coordinates, depth coordinates, world coordinates and main camera world coordinates, which can improve the robustness, accuracy and stability of the calibration result, and the algorithm is simple and has high real-time performance. The IMU and the auxiliary cameras are used to collect based on the time synchronization signal to obtain the acceleration collection sequence and the angular velocity collection sequence. , IMU acquisition time series and camera acquisition time series, and use the IMU acquisition time series and camera acquisition time series to establish and solve the calibration equation, and then perform time calibration on the acquired data after obtaining the initial time offset sequence, and use the acceleration calibration sequence and angular velocity calibration sequence obtained after calibration to initialize the attitude calibration matrix and the translation calibration matrix, thereby eliminating the error caused by the time asynchrony between the camera and IMU, and improving the accuracy of the initial estimate of the calibration result; by cyclically calculating the trajectory error, the first error matrix and the second error matrix, and updating the attitude calibration matrix and the translation calibration matrix, the calibration result is optimized, and the algorithm complexity is low.

[0267] Embodiment 2

[0268] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of a multi-camera and IMU collaborative calibration device disclosed in an embodiment of the present invention. Figure 3 The multi-camera and IMU collaborative calibration device described can be applied to the field of camera and IMU collaborative calibration, such as robot navigation, augmented reality and virtual reality, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include:

[0269] The auxiliary camera calibration module 201 is used to obtain a horizontal focal length set, a vertical focal length set, a relative posture matrix set and a relative translation matrix set using an auxiliary camera array.

[0270] The data acquisition module 202 is used to obtain the acceleration acquisition sequence, angular velocity acquisition sequence, auxiliary camera image sequence set, camera acquisition time sequence and IMU acquisition time sequence acquired by the IMU and the auxiliary camera array;

[0271] A world coordinate calculation module 203 is used to process the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates;

[0272] The calibration calculation module 204 is used to process based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates to obtain the attitude calibration matrix and the translation calibration matrix.

[0273] In an optional embodiment, the auxiliary camera calibration module 201 uses the auxiliary camera array to obtain a horizontal focal length set, a vertical focal length set, a relative posture matrix set, and a relative translation matrix set, including:

[0274] Preprocessing all calibration images in the calibration image set to obtain a preprocessed image set;

[0275] Perform corner point detection on the preprocessed image set to obtain a set of corner point coordinates of the calibration plate;

[0276] Calibrate using the corner point coordinate set and the length and width values ​​of the calibration plate to obtain a horizontal focal length set and a vertical focal length set;

[0277] The corner point coordinate set is subjected to attitude estimation processing to obtain a relative attitude matrix set and a relative translation matrix set.

[0278] In another optional embodiment, the world coordinate calculation module 203 processes the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates, including:

[0279] Using the auxiliary camera image sequence set, obtain the main camera feature point descriptor;

[0280] Using the feature point extraction model, the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence are processed respectively to obtain extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera;

[0281] The extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera are respectively combined with the feature point descriptor of the main camera to obtain four combined feature point descriptor sequences;

[0282] Using the feature point matching model, each combined feature point description subsequence is processed separately to obtain the corresponding similarity sequence;

[0283] Using each similarity sequence, the extracted feature point descriptor sequence of the corresponding feature point description group is screened to obtain 4 matching feature point descriptors;

[0284] Obtain the pixel coordinates of the four matching feature point descriptors respectively to obtain four matching pixel coordinates;

[0285] Using a normalized pixel coordinate calculation model, the matching pixel coordinates, the horizontal focal length set, and the vertical focal length set are processed to obtain a normalized pixel coordinate set;

[0286] The relative attitude matrix set, the relative translation matrix set and the normalized pixel coordinate set are processed to obtain the main camera world coordinates.

[0287] In another optional embodiment, the calibration calculation module 204 performs processing based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates to obtain the attitude calibration matrix and the translation calibration matrix, including:

[0288] Step 1, determine that the current number of cycles is 1;

[0289] Step 2: Process the acceleration acquisition sequence, angular velocity acquisition sequence, camera acquisition time sequence, IMU acquisition time sequence, and main camera world coordinates to obtain the IMU position sequence, main camera position sequence, and IMU attitude sequence, and initialize the attitude calibration matrix and translation calibration matrix;

[0290] Step 3: Use the trajectory error calculation model to calculate the IMU position sequence P imu , attitude calibration matrix R cam_imu , main camera position sequence P cam , translation calibration matrix T cam_imu and IMU attitude sequence R imu Processing is performed to obtain a trajectory error F, a first error matrix G, and a second error matrix H;

[0291] Step 4: Use the calibration matrix to update the model, process the attitude calibration matrix and the translation calibration matrix, and obtain updated attitude calibration matrix and translation calibration matrix;

[0292] Step 5, add 1 to the current number of cycles;

[0293] Step 6: Repeat steps 3 to 5 until the trajectory error value is less than a preset target threshold, or the current number of cycles reaches a preset maximum number of cycles.

[0294] It can be seen that the implementation Figure 3 The described multi-camera and IMU collaborative calibration device can improve the robustness, accuracy and stability of the calibration results, as well as the real-time performance of the calibration process.

[0295] Embodiment 3

[0296] See also Figure 4 , Figure 4 is a schematic diagram of the structure of another multi-camera and IMU collaborative calibration device disclosed in an embodiment of the present invention. Figure 4 The multi-camera and IMU collaborative calibration device described can be applied to the field of camera and IMU collaborative calibration, such as robot navigation, augmented reality and virtual reality, etc., and the embodiments of the present invention are not limited thereto. Figure 4 As shown, the multi-camera and IMU collaborative calibration device may include the following parts:

[0297] A memory 301 storing executable program codes;

[0298] a processor 302 coupled to the memory 301;

[0299] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the multi-camera and IMU collaborative calibration method described in the first embodiment.

[0300] Embodiment 4

[0301] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the multi-camera and IMU collaborative calibration method described in the first embodiment.

[0302] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0303] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0304] Finally, it should be noted that the multi-camera and IMU collaborative calibration method and device disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera and IMU collaborative calibration method, characterized in that: include: S1, a first auxiliary camera, a second auxiliary camera, a third auxiliary camera and a fourth auxiliary camera are dispersedly arranged around a preset main camera and an IMU; the main camera is fixedly connected to the IMU; the first auxiliary camera, the second auxiliary camera, the third auxiliary camera and the fourth auxiliary camera together constitute an auxiliary camera array; S2, using the auxiliary camera array, obtaining a horizontal focal length set, a vertical focal length set, a relative posture matrix set, and a relative translation matrix set; The horizontal focal length set includes a first horizontal focal length, a second horizontal focal length, a third horizontal focal length and a fourth horizontal focal length; the vertical focal length set includes a first vertical focal length, a second vertical focal length, a third vertical focal length and a fourth vertical focal length; the relative posture matrix set includes a first relative posture matrix, a second relative posture matrix and a third relative posture matrix; the relative translation matrix set includes a first relative translation matrix, a second relative translation matrix and a third relative translation matrix; S3, using the IMU and the auxiliary camera array, acquiring an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence, and an IMU acquisition time sequence; The auxiliary camera image sequence set includes a first auxiliary camera image sequence, a second auxiliary camera image sequence, a third auxiliary camera image sequence and a fourth auxiliary camera image sequence; the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence and the fourth auxiliary camera image sequence each include N auxiliary camera images; N is an integer greater than 1; S4, processing the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates; S5, acquiring a posture calibration matrix and a translation calibration matrix based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates.

2. The multi-camera and IMU collaborative calibration method according to claim 1, characterized in that: The method of using the IMU and the auxiliary camera array to obtain an acceleration acquisition sequence, an angular velocity acquisition sequence, an auxiliary camera image sequence set, a camera acquisition time sequence, and an IMU acquisition time sequence includes: S31, connecting the IMU and the auxiliary camera array to a preset clock synchronization unit; the clock synchronization unit is used to generate a time synchronization signal; S32, moving the main camera and the IMU within the field of view of the auxiliary camera; the field of view of the auxiliary camera is the intersection of the fields of view of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera; S33, using the IMU to perform acquisition based on the time synchronization signal to obtain the acceleration acquisition sequence, the angular velocity acquisition sequence and the IMU acquisition time series; S34, using the auxiliary camera array to perform synchronous shooting based on the time synchronization signal to obtain the auxiliary camera image sequence set and the camera acquisition time sequence.

3. The multi-camera and IMU collaborative calibration method according to claim 1, characterized in that: The processing of the auxiliary camera image sequence set, the horizontal focal length set, the vertical focal length set, the relative posture matrix set and the relative translation matrix set to obtain the main camera world coordinates includes: S41, using the auxiliary camera image sequence set to obtain a main camera feature point descriptor; S42, using a feature point extraction model, respectively processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence, and the fourth auxiliary camera image sequence to obtain extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera; The extracted feature point descriptor sequence includes more than one extracted feature point descriptor; S43, respectively combining the extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera with the feature point descriptor of the main camera to obtain four combined feature point descriptor sequences; S44, using a feature point matching model, processing each of the combined feature point description subsequences respectively to obtain a corresponding similarity sequence; the similarity sequence includes more than one similarity value, and the length is the same as the extracted feature point description subsequence of the corresponding feature point description group; S45, using each of the similarity sequences respectively, screening the extracted feature point descriptor sequence of the corresponding feature point description group to obtain four matching feature point descriptors; S46, respectively obtaining pixel coordinates of the four matching feature point descriptors to obtain four matching pixel coordinates; S47, using a normalized pixel coordinate calculation model, processing the matching pixel coordinates, the horizontal focal length set, and the vertical focal length set to obtain a normalized pixel coordinate set; the normalized pixel coordinate set includes 4 normalized pixel coordinates; The normalized pixel coordinate calculation model is: Wherein, x1, x2, x3 and x4 are the horizontal components of the four matching pixel coordinates; y1, y2, y3 and y4 are the vertical components of the four matching pixel coordinates; x n,1 、x n,2 、x n,3 、x n,4 are the horizontal components of the four normalized pixel coordinates respectively; n,1 ,y n,2 ,y n,3 and n,4 are the vertical components of the four normalized pixel coordinates respectively; x,1 、f x,2 、f x,3 and f x,4 are the first horizontal focal length, the second horizontal focal length, the third horizontal focal length and the fourth horizontal focal length respectively; y,1 、f y,2 、f y,3 and f y,4 are respectively the first vertical focal length, the second vertical focal length, the third vertical focal length and the fourth vertical focal length; S48, processing the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set to obtain the main camera world coordinates.

4. The multi-camera and IMU collaborative calibration method according to claim 3, characterized in that: The method of using the feature point extraction model to process the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence, and the fourth auxiliary camera image sequence respectively to obtain extracted feature point descriptor sequences of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, and the fourth auxiliary camera includes: S421, using the first convolution layer of the feature point extraction model, processing the first auxiliary camera image sequence, the second auxiliary camera image sequence, the third auxiliary camera image sequence, or the fourth auxiliary camera image sequence to obtain first convolution information; S422, using a first normalization layer of the feature point extraction model, performing a batch normalization operation on the first convolution information to obtain first normalized information; S423, using the first pooling layer of the feature point extraction model, performing a pooling operation on the first normalized information to obtain first pooled information; S424, using the second convolution layer of the feature point extraction model to perform convolution processing on the first pooling information to obtain second convolution information; S425, using the second normalization layer of the feature point extraction model, performing a batch normalization operation on the second convolution information to obtain second normalized information; S426, using the second pooling layer of the feature point extraction model, performing a pooling operation on the second normalized information to obtain second pooled information; S427, using the first fully connected layer of the feature point extraction model, performing fully connected processing on the second pooling information to obtain first fully connected information; S428: Perform full-connection processing on the first fully-connected information by using the second fully-connected layer of the feature point extraction model to obtain the extracted feature point descriptor sequence of the first auxiliary camera, the second auxiliary camera, the third auxiliary camera, or the fourth auxiliary camera.

5. The multi-camera and IMU collaborative calibration method according to claim 3, characterized in that: The method of using the feature point matching model to process each of the combined feature point description subsequences to obtain a corresponding similarity sequence includes: S441, using the third convolution layer of the feature point matching model, performing convolution processing on the combined feature point descriptor sequence to obtain third convolution information; S442, using the third normalization layer of the feature point matching model, performing a batch normalization operation on the third convolution information to obtain third normalized information; S443, using the third pooling layer of the feature point matching model, performing a pooling operation on the third normalized information to obtain third pooled information; S444, using the fourth convolution layer of the feature point matching model to perform convolution processing on the third pooling information to obtain fourth convolution information; S445, using the fourth normalization layer of the feature point matching model, performing a batch normalization operation on the fourth convolution information to obtain fourth normalized information; S446, using the fourth pooling layer of the feature point matching model, performing a pooling operation on the fourth normalized information to obtain fourth pooled information; S447, using the third fully connected layer of the feature point matching model, performing fully connected processing on the fourth pooling information to obtain third fully connected information; S448: Use the fourth fully connected layer of the feature point matching model to perform fully connected processing on the third fully connected information to obtain the corresponding similarity sequence.

6. The multi-camera and IMU collaborative calibration method according to claim 3, characterized in that: The processing of the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set to obtain the main camera world coordinates includes: S481, establishing a depth coordinate calculation model using the relative posture matrix set, the relative translation matrix set and the normalized pixel coordinate set; The depth coordinate calculation model is: In the formula, x n,1 、x n,2 、x n,3 、x n,4 are the horizontal components of the four normalized pixel coordinates respectively; n,1 ,y n,2 ,y n,3 and n,4 are the vertical components of the four normalized pixel coordinates respectively; n,1 、z n,1 、z n,1 and z n,4 are respectively the first depth coordinate, the second depth coordinate, the third depth coordinate and the fourth depth coordinate; R1, R2 and R3 are respectively the first relative posture matrix, the second relative posture matrix and the third relative posture matrix; T1, T2 and T3 are respectively the first relative translation matrix, the second relative translation matrix and the third relative translation matrix; S482, solving a depth coordinate calculation model to obtain the first depth coordinate, the second depth coordinate, the third depth coordinate, and the fourth depth coordinate; S483, using a world coordinate calculation model, processing the relative posture matrix set, the relative translation matrix set, the first depth coordinate, the second depth coordinate, the third depth coordinate, and the fourth depth coordinate to obtain three world coordinates; The world coordinate calculation model is: In the formula, (x w,1 ,y w,1 ,z w,1 )、(x w,2 ,y w,2 ,z w,2 ) and (x w,3 ,y w,3 ,z w,3 ) are the three world coordinates respectively; S484, using a main camera world coordinate calculation model, processing the three world coordinates to obtain the main camera world coordinates; The main camera world coordinate calculation model is: In the formula, x w ,y w and z w are respectively the horizontal component, vertical component and depth component of the main camera world coordinates.

7. The multi-camera and IMU collaborative calibration method according to claim 1, characterized in that: The acquiring of the attitude calibration matrix and the translation calibration matrix based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates comprises: S51, determining that the current number of cycles is 1; S52, processing the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence, and the main camera world coordinates to obtain an IMU position sequence, a main camera position sequence, and an IMU attitude sequence, and initialize the attitude calibration matrix and the translation calibration matrix; S53, using the trajectory error calculation model, the IMU position sequence P imu , the attitude calibration matrix R cam_imu , the main camera position sequence P cam , the translation calibration matrix T cam_imu and the IMU attitude sequence R imu Processing is performed to obtain a trajectory error F, a first error matrix G, and a second error matrix H; The trajectory error calculation model is: In the formula, α is the preset weight coefficient, and i is the element number; S54, using the calibration matrix to update the model, processing the attitude calibration matrix and the translation calibration matrix to obtain updated attitude calibration matrix and translation calibration matrix; The calibration matrix update model is: In the formula, and R cam_imu are the values ​​of the attitude calibration matrix before and after updating, and T cam_imu are respectively the values ​​of the translation calibration matrix before and after updating; S55, adding 1 to the current cycle number; S56, repeat S53 to S55 until the trajectory error value is less than a preset target threshold, or the current number of cycles reaches a preset maximum number of cycles.

8. The multi-camera and IMU collaborative calibration method according to claim 7, characterized in that: The processing based on the acceleration acquisition sequence, the angular velocity acquisition sequence, the camera acquisition time sequence, the IMU acquisition time sequence and the main camera world coordinates to obtain the attitude calibration matrix and the translation calibration matrix includes: S521, interpolating the acceleration acquisition sequence and the angular velocity acquisition sequence to obtain an interpolation time series, an acceleration interpolation sequence, and an angular velocity interpolation sequence; S522, establishing a calibration equation using the camera acquisition time series and the IMU acquisition time series; The calibration equation is: T1=T2+δt Wherein, T1, T2 and δ are column vectors composed of the camera acquisition time series, the IMU acquisition time series and the initial time offset sequence respectively; S523, solving the calibration equation to obtain the initial time offset sequence; S524, adding the acceleration acquisition sequence to the elements in the initial time offset sequence correspondingly to obtain the acceleration calibration sequence; S525, adding the angular velocity acquisition sequence and the elements of the initial time offset sequence correspondingly to obtain the angular velocity calibration sequence; S526, performing fusion processing on the angular velocity calibration sequence to obtain an initial IMU attitude matrix; S527, using the acceleration calibration sequence, the angular velocity calibration sequence and the initial IMU attitude matrix, obtain an IMU position sequence, a main camera position sequence and an IMU attitude sequence, as well as initial values ​​of the attitude calibration matrix and the translation calibration matrix.

9. A multi-camera and IMU collaborative calibration data processing device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multi-camera and IMU collaborative calibration method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the multi-camera and IMU collaborative calibration method as described in any one of claims 1-8.

Images