Camera and imu joint calibration method, device and system

Abstract

Embodiments of the present application provide a camera and IMU joint calibration method, device and system, relating to the technical field of artificial intelligence. The method is applied to a control unit of a camera and IMU joint calibration system, and the method comprises: controlling a laser scanner to scan a device to be calibrated to obtain point cloud data of a camera chip and an IMU; performing calculation based on the point cloud data of the camera chip and the IMU to obtain a translation vector parameter; calibrating and correcting the camera under the condition that a physical center of the camera chip is aligned with an optical center of a lens to obtain a millimeter focal length of the camera; and calculating a translation vector of the IMU from the optical center of the lens according to the translation vector parameter and the millimeter focal length of the camera to make the coordinate systems of the IMU and the camera consistent. Embodiments of the present application realize joint calibration of the camera and the IMU in batch production conveniently, and make the coordinate systems of the IMU and the camera consistent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a camera and IMU joint calibration method, device and system. BACKGROUND

[0002] Camera and IMU (Inertial Measurement Unit) are two commonly used sensors in the field of autonomous driving. In actual application, the information provided by the two is often fused to obtain more stable positioning and perception results. The perception and positioning results obtained by a single sensor are in the coordinate system of the sensor itself, and need to be unified in space by relying on the relative pose relationship between the coordinate systems of the sensors, i.e. the extrinsic parameters, and then fused in the unified coordinate system. Therefore, it is necessary to calibrate the camera and IMU. The accuracy of the extrinsic parameters between the camera and IMU during the calibration process will greatly affect the accuracy of the autonomous driving positioning and perception system.

[0003] When calibrating the camera and IMU by comparing the motion trajectories of the camera and IMU or by comparing their acceleration and angular velocity, etc., the IMU needs to be given sufficient excitation of three axes, i.e. sufficient acceleration translation and sufficient angular velocity rotation. Traditional rolling shutter cameras or cameras with long exposure times are prone to image motion blur, which makes the image unable to be recognized by the algorithm, ultimately leading to calibration failure. The method of directly measuring the extrinsic parameters from the drawings will also lead to inaccurate calibration results due to the errors introduced by the processing technology and installation. The above methods are not suitable for use in large-scale production. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a camera and IMU joint calibration method, device and system to realize the joint calibration of the camera and IMU and make the coordinate systems of the IMU and the camera consistent.

[0005] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a camera and IMU joint calibration method applied to a control unit of a camera and IMU joint calibration system, the camera and IMU joint calibration system further comprising a laser scanner, the laser scanner being electrically connected to the control unit, and the method comprising:

[0007] controlling the laser scanner to scan a device to be calibrated to obtain point cloud data of a camera chip and an IMU, wherein the device to be calibrated is installed with an IMU and at least one camera, and the camera comprises a camera chip and a lens;

[0008] calculating, based on the point cloud data of the camera chip and the IMU, a translation vector parameter;

[0009] calibrating and rectifying the camera to obtain a millimeter focal length of the camera, in a case where a physical center of the camera chip and an optical center of the lens are aligned;

[0010] calculating, based on the translation vector parameter and the millimeter focal length of the camera, a translation vector of a center of the IMU from the optical center of the lens, to make coordinate systems of the IMU and the camera consistent.

[0011] In an optional implementation, the step of calculating, based on the point cloud data of the camera chip and the IMU, a translation vector parameter, comprises:

[0012] calculating, based on the point cloud data of the camera chip and the IMU, a translation vector of a center of the IMU from a physical center of the camera chip, and taking the translation vector as a translation vector parameter.

[0013] In an optional implementation, the device to be calibrated is installed with one camera, and the step of calibrating and rectifying the camera to obtain a millimeter focal length of the camera, comprises:

[0014] calibrating the camera to obtain intrinsic parameters and distortion coefficients of the camera;

[0015] rectifying the camera based on the intrinsic parameters and distortion coefficients of the camera to obtain a millimeter focal length of the camera.

[0016] In an optional implementation, the intrinsic parameters of the camera include a pixel focal length of the camera, and the step of rectifying the camera based on the intrinsic parameters and distortion coefficients of the camera to obtain a millimeter focal length of the camera, comprises:

[0017] updating the pixel focal length of the camera based on the pixel focal length and the distortion coefficients of the camera to obtain an updated pixel focal length of the camera as a first to-be-calculated intrinsic parameter;

[0018] obtaining sizes of each phase element of the camera chip in a width direction and a height direction as a first to-be-calculated parameter;

[0019] obtaining a millimeter focal length of the camera based on the first to-be-calculated intrinsic parameter and the first to-be-calculated parameter.

[0020] In an optional implementation, the device to be calibrated is installed with two cameras, and the step of calibrating and rectifying the camera to obtain a millimeter focal length of the camera, comprises:

[0021] The cameras are calibrated respectively to obtain intrinsic parameters, distortion coefficients and extrinsic parameters of the cameras;

[0022] The cameras are rectified respectively based on the intrinsic parameters, distortion coefficients and extrinsic parameters of the cameras to obtain millimeter focal lengths of the cameras.

[0023] In an optional embodiment, the intrinsic parameters of the cameras include pixel focal lengths of the cameras, and the step of rectifying the cameras respectively based on the intrinsic parameters, distortion coefficients and extrinsic parameters of the cameras to obtain millimeter focal lengths of the cameras includes:

[0024] The cameras are rectified respectively based on the pixel focal lengths and distortion coefficients of the cameras to obtain the rectified cameras;

[0025] The rectified cameras are rectified based on the extrinsic parameters of the cameras to obtain pixel focal lengths of the rectified cameras as second to-be-calculated intrinsic parameters;

[0026] The sizes of the pixels corresponding to the camera chips in the width direction and the height direction are obtained as second to-be-calculated parameters;

[0027] The millimeter focal lengths of the cameras are obtained based on the second to-be-calculated intrinsic parameters and the second to-be-calculated parameters.

[0028] In an optional embodiment, the extrinsic parameters of the cameras include an extrinsic translation vector and an extrinsic rotation matrix of the cameras, and the extrinsic translation vector represents a translation vector of an optical center of a lens in one of the cameras to an optical center of a lens in another of the cameras.

[0029] In an optional embodiment, the distortion coefficients are radial distortion coefficients and tangential distortion coefficients.

[0030] In a second aspect, an embodiment of the present application provides a camera and IMU joint calibration device applied to a control unit of a camera and IMU joint calibration system, the camera and IMU joint calibration system further comprising a laser scanner, the laser scanner being electrically connected to the control unit, and the device comprising:

[0031] A parameter acquisition module is configured to control the laser scanner to scan a to-be-calibrated device to obtain point cloud data of camera chips and IMUs, wherein the to-be-calibrated device is installed with IMUs and at least one camera, and the camera includes camera chips and lenses.

[0032] A calculation module is configured to perform calculation based on the point cloud data of the camera chips and IMUs to obtain a translation vector parameter.

[0033] a calibration module, configured to calibrate and correct the camera to obtain a millimeter focal length of the camera in a case that a physical center of the camera chip and an optical center of the lens are aligned, and calculate a translation vector of a center of the IMU from the optical center of the lens according to the translation vector parameter and the millimeter focal length of the camera, so as to make the coordinate systems of the IMU and the camera consistent.

[0034] In a third aspect, an embodiment of the present application provides a camera and IMU joint calibration system, comprising a mechanical arm, a fixing device, a laser scanner and a control unit, the control unit being electrically connected with the mechanical arm and the laser scanner;

[0035] The mechanical arm is configured to drive the laser scanner to move.

[0036] The fixing device is configured to fix a device to be calibrated, the device to be calibrated being installed with an IMU and at least one camera, the camera comprising a camera chip and a lens.

[0037] The laser scanner is configured to scan the device to be calibrated to obtain point cloud data of the camera chip and the IMU.

[0038] The control unit is configured to control the mechanical arm and the laser scanner to work, and is further configured to implement the camera and IMU joint calibration method provided in the first aspect and / or possible implementation manners of the first aspect, so as to make the coordinate systems of the IMU and the camera consistent.

[0039] The beneficial effects of the embodiments of the present application include, for example:

[0040] The camera and IMU joint calibration method, device and system provided by the embodiments of the present application can conveniently calibrate the camera and the IMU in batch production, so as to make the coordinate systems of the IMU and the camera consistent.

[0041] In order to make the above objectives, characteristics and advantages of the present application more apparent and understandable, the following preferred embodiments are described in detail below, and the accompanying drawings are referred to, and the detailed description is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 An exemplary structural block diagram of a camera and IMU joint calibration system provided by the embodiments of the present application is shown;

[0044] Figure 2 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0045] Figure 3 An exemplary structural diagram of a camera chip in a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0046] Figure 4 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0047] Figure 5 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0048] Figure 6 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0049] Figure 7 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown;

[0050] Figure 8 A flowchart of a camera and IMU joint calibration method provided by the embodiments of the present application is shown Figure Six ;

[0051] Figure 9 An exemplary structural block diagram of a camera and IMU joint calibration device provided by the embodiments of the present application is shown.

[0052] Figure legend: 100-camera and IMU joint calibration system; 110-robotic arm; 120-fixing device; 130-laser scanner; 140-control unit; 300-camera and IMU joint calibration device; 301-parameter acquisition module; 302-computation module; 303-calibration module. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0054] Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0055] It should be noted that the relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0056] It should be noted that the features in the embodiments of the present application can be combined with each other without conflict.

[0057] Please refer to Figure 1 , Figure 1 An exemplary structural block diagram of a camera and IMU joint calibration system 100 provided by an embodiment of the present application is shown, as shown in Figure 1 The camera and IMU joint calibration system 100 includes a mechanical arm 110, a fixed device 120, a laser scanner 130, and a control unit 140, which is electrically connected with the mechanical arm and the laser scanner.

[0058] The mechanical arm 110 is used to drive the laser scanner 130 to move, and the fixing device 120 is used to fix the device to be calibrated (not shown in the figure). The laser scanner 130 is generally installed at the end of the mechanical arm 110, and is used to scan the device to be calibrated. The device to be calibrated is generally provided with an IMU and at least one camera (not shown in the figure), and the camera includes an image COMS or CCD chip and a lens (not shown in the figure). Based on this, the laser scanner 130 scans the device to be calibrated, and point cloud data of the camera chip and the IMU is obtained for subsequent processing.

[0059] Further, the control unit 140 of the camera and IMU joint calibration system 100 is used to control the mechanical arm 110 and the laser scanner 130 to work, and is also used to process the point cloud data of the camera chip and the IMU obtained by the laser scanner 130 scanning the device to be calibrated, and calibrate and correct the camera, so as to finally realize the coordinate system of the IMU and the camera, and make the IMU and the camera can be fused in the unified coordinate system, so as to obtain more stable positioning and perception results.

[0060] Next, taking the control unit 140 of the camera and IMU joint calibration system 100 as the execution subject, a camera and IMU joint calibration method provided by the embodiment of the application is exemplarily described. Please refer to Figure 2 , Figure 2 The flowchart of the camera and IMU joint calibration method provided by the embodiment of the application is shown.

[0061] As Figure 2 shown, the camera and IMU joint calibration method is applied to the control unit 140 of the camera and IMU joint calibration system 100. The camera and IMU joint calibration system 100 further includes the laser scanner 130, and the laser scanner 130 is electrically connected with the control unit 140. The camera and IMU joint calibration method can include the following steps:

[0062] S210, controlling the laser scanner to scan the device to be calibrated, so as to obtain the point cloud data of the camera chip and the IMU.

[0063] The device to be calibrated is provided with the IMU and at least one camera, and the camera includes the camera chip and the lens.

[0064] S220, calculating based on the point cloud data of the camera chip and the IMU, so as to obtain the translation vector parameter.

[0065] S230, calibrating and correcting the camera under the condition that the physical center of the camera chip and the optical center of the lens are aligned, so as to obtain the millimeter focal length of the camera.

[0066] S240, according to the translation vector parameter and the millimeter focal length of the camera, the translation vector of the center of the IMU from the optical center of the lens is calculated to make the coordinate systems of the IMU and the camera consistent.

[0067] The above steps realize the process of processing the point cloud data of the camera chip and the IMU obtained by scanning the device to be calibrated by the laser scanner, calibrating and correcting the camera, and finally realizing the coordinate system consistency of the IMU and the camera.

[0068] Specifically, step S210 is a process of controlling the laser scanner to scan, and an exemplary structure of the camera chip in the camera can be as shown in the following figure. Figure 3

[0069] It should be noted that when scanning the device to be calibrated, since the IMU in the device to be calibrated usually includes a gyroscope and an accelerometer, the output results of the two kinds of data of the gyroscope and the accelerometer are usually converted to the same coordinate system, therefore, assuming that the coordinate systems of the output two kinds of data of the gyroscope and the accelerometer are both the coordinate system of the accelerometer, the point cloud data of the IMU obtained by scanning the device to be calibrated in step S210 is the point cloud data of the IMU in the coordinate system of the accelerometer. After the laser scanner scans to obtain the point cloud data of the camera chip and the IMU, step S220 is continued, and the translation vector parameter is obtained based on the point cloud data of the camera chip and the IMU.

[0070] In the embodiment of the application, the above-mentioned translation vector parameter can be, for example, the translation vector of the center position of the IMU from the physical center of the camera chip, and after obtaining the translation vector, step S230 is continued to calibrate and correct the camera to obtain the millimeter focal length of the camera.

[0071] It should be noted that before calibrating and correcting the camera, it is also necessary to ensure that the physical center of the camera chip and the optical center of the lens are aligned. Since the camera is usually composed of an image CMOS or CCD chip and a lens, during the lens assembly process, an active alignment (AA) device is generally used to adjust the attitude and relative position between the lens and the camera chip, align the physical center of the camera chip and the optical center of the lens, and then the subsequent calibration and correction work can be performed.

[0072] Exemplarily, the process of calibrating and correcting the camera in step S230 can be the process of calibrating and distortion correcting the camera. Since the camera lens is generally a lens, the propagation of light in the imaging process will have the following effects: first, the shape of the lens itself affects the propagation of light, and second, during the mechanical assembly process, the lens and the imaging plane cannot be completely parallel, so that the position of the light passing through the lens and projected onto the imaging plane changes.

[0073] ​Based on this, the above-mentioned distortion caused by the lens shape is called radial distortion, which can make a straight line in the real environment become a curve in the picture. At the same time, during the camera assembly process, the tangential distortion will also be introduced because the lens and the imaging surface cannot be strictly parallel. Therefore, the above-mentioned radial distortion and tangential distortion need to be corrected, and the distortion correction process can be based on the camera intrinsic parameters obtained after camera calibration and the radial distortion coefficients (for example, set the radial distortion coefficients as k1, k2, k3), the tangential distortion coefficients (for example, set the tangential distortion coefficients as p1, p2), update the camera intrinsic parameters to obtain the corrected camera intrinsic parameters.

[0074] Exemplarily, if the above-mentioned camera intrinsic parameters are set as K(fx, fy, cx, cy), that is, the above-mentioned camera intrinsic parameters can include the pixel focal length fx, fy of the camera, after the distortion correction of the camera, the pixel focal length of the camera after the distortion correction can be obtained. Since the relationship between the pixel focal length of the camera and the millimeter focal length of the camera can be represented by the following formula:

[0075] fx=af, fy=bf

[0076] (Formula 1)

[0077] Wherein, a and b are the size of each pixel of the camera chip in the width direction and the height direction, and the unit is um, and f is the millimeter focal length.

[0078] Since the camera chip is composed of a plurality of pixels, and the size of each pixel in the width direction and the height direction is the same, the size of each pixel of the camera chip in the width direction and the height direction can be obtained based on the parameters of the camera when it is shipped. Finally, based on the obtained pixel focal length of the camera after the distortion correction, and the size of each pixel of the camera chip in the width direction and the height direction, the millimeter focal length of the camera is calculated.

[0079] Further, since the to-be-calibrated device is installed with at least one camera, there is also a case that the to-be-calibrated device is installed with two cameras. At this time, when step S230 is performed, each camera needs to be calibrated and distortion corrected, and then stereo correction needs to be performed to make the final obtained millimeter focal length more accurate.

[0080] Exemplarily, the process of the above-mentioned stereo correction can be performed after the calibration of each camera, the pixel focal length fx, fy, the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2 of the camera are obtained, and the stereo correction of each camera is performed based on the camera extrinsic parameters obtained after the calibration of each camera to obtain the pixel focal length of the camera after the stereo correction.

[0081] The camera extrinsic parameters can be, for example, an extrinsic rotation matrix of each camera, and a translation vector (R, t) of the optical center of the lens of one camera to the optical center of the lens of another camera. When the pixel focal length of the rectified camera is obtained, and the size of each pixel on the camera chip in the width direction and the height direction is obtained, the millimeter focal length of the camera can be calculated based on the formula (1) above.

[0082] In the embodiment of the present application, after the millimeter focal length of the camera is obtained in step S230, step S240 is performed to calculate the translation vector of the center of the IMU to the optical center of the lens based on the translation vector parameter obtained in step S220 and the millimeter focal length of the camera obtained in step S230, that is, to calculate the position of the IMU relative to the camera coordinate system, so that the coordinate systems of the IMU and the camera are consistent, facilitating subsequent fusion in the unified coordinate system.

[0083] The camera and IMU joint calibration method provided by the embodiment of the present application can conveniently calibrate the camera and the IMU in batch production, so that the coordinate systems of the IMU and the camera are consistent.

[0084] Optionally, the specific process of calculating the translation vector parameter based on the point cloud data of the camera chip and the IMU in step S220 can be implemented by the following steps:

[0085] On the basis of Figure 2 , please refer to Figure 4 , Figure 4 Fig. 2 shows a flowchart of the camera and IMU joint calibration method provided by the embodiment of the present application, and the step of calculating the translation vector parameter based on the point cloud data of the camera chip and the IMU in step S220 includes:

[0086] S221, based on the point cloud data of the camera chip and the IMU, the translation vector of the center of the IMU to the physical center of the camera chip is calculated, and the translation vector is taken as the translation vector parameter.

[0087] The above steps realize the process of obtaining the translation vector of the center position of the IMU to the physical center of the camera chip based on the point cloud data of the camera chip and the IMU.

[0088] Optionally, the process of calibrating and correcting the camera in step S230 to obtain the millimeter focal length of the camera can be implemented on the premise that the device to be calibrated is installed with one camera, and the specific process can be implemented by the following steps:

[0089] On the basis of Figure 2 , refer to Figure 5 , Figure 5 Fig. 3 shows a flowchart of a camera and IMU joint calibration method provided by an embodiment of the present application, the device to be calibrated is installed with one camera, and the step of calibrating and correcting the camera in step S230 to obtain the millimeter focal length of the camera comprises:

[0090] S231, calibrate the camera to obtain the intrinsic parameters and distortion coefficients of the camera;

[0091] S232, correct the camera based on the intrinsic parameters and distortion coefficients of the camera to obtain the millimeter focal length of the camera.

[0092] The above steps implement the process of calibrating and correcting the camera to obtain the millimeter focal length of the camera in the case that the device to be calibrated is installed with one camera.

[0093] Exemplarily, based on the camera intrinsic parameters in the foregoing, the pixel focal length fx, fy of the camera and the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2, the camera can be corrected based on the above parameters to obtain the pixel focal length of the camera after distortion correction, and the millimeter focal length of the camera can be obtained based on the formula one in the foregoing.

[0094] Optionally, the process of correcting the camera based on the intrinsic parameters and distortion coefficients of the camera in step S232 to obtain the millimeter focal length of the camera is actually the process of correcting the camera based on the pixel focal length and distortion coefficients of the camera to finally obtain the millimeter focal length of the camera. The specific process can be implemented by the following steps:

[0095] On the basis of Figure 5 , refer to Figure 6 , Figure 6 Fig. 4 shows a flowchart of a camera and IMU joint calibration method provided by an embodiment of the present application, the intrinsic parameters of the camera comprise the pixel focal length of the camera, and the step of correcting the camera based on the intrinsic parameters and distortion coefficients of the camera in step S232 to obtain the millimeter focal length of the camera comprises:

[0096] S2321, update the pixel focal length of the camera based on the pixel focal length and distortion coefficients of the camera to obtain the updated pixel focal length of the camera as the first to-be-calculated intrinsic parameter.

[0097] S2322, obtain the size of each phase element of the camera chip in the width direction and the height direction as the first to-be-calculated parameter.

[0098] S2323, obtaining the millimeter focal length of the camera based on the first to-be-calculated internal parameter and the first to-be-calculated parameter.

[0099] The above steps realize a process of obtaining a new pixel focal length based on a pixel focal length and a distortion coefficient of a camera, and finally obtaining a millimeter focal length of the camera according to sizes of each phase element of a camera chip in a width direction and a height direction.

[0100] Exemplarily, based on the camera internal parameters in the foregoing, the pixel focal length fx, fy of the camera and the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2, if the updated pixel focal length of the camera is fx', fy', and the sizes of each phase element of the camera chip in the width direction and the height direction are a and b respectively, the millimeter focal length of the camera obtained based on the formula one in the foregoing can be fx' / a and fy' / b.

[0101] Optionally, the process of calibrating and correcting the camera to obtain the millimeter focal length of the camera in the S230 step can also be realized under the premise that the to-be-calibrated device is provided with two cameras, and the above specific process can be realized through the following steps.

[0102] On the basis of Figure 2 , please refer to Figure 7 , Figure 7 Fig. 5 shows a flowchart of a camera and IMU joint calibration method provided by an embodiment of the application, and the to-be-calibrated device is provided with two cameras. The step of calibrating and correcting the camera to obtain the millimeter focal length of the camera in the S230 step includes the following steps.

[0103] S233, calibrating each camera respectively to obtain the internal parameter, the distortion coefficient and the external parameter of each camera.

[0104] S234, correcting each camera respectively based on the internal parameter, the distortion coefficient and the external parameter of each camera to obtain the millimeter focal length of each camera.

[0105] The above steps realize a process of calibrating and correcting the camera to obtain the millimeter focal length of the camera under the condition that the to-be-calibrated device is provided with two cameras.

[0106] Exemplarily, based on the camera intrinsic parameters in the foregoing, the pixel focal length of the camera, the intrinsic parameters of each camera in the step S233 can be the pixel focal length of each camera. Further, based on the radial distortion coefficients k1, k2, k3, the tangential distortion coefficients p1, p2, the extrinsic parameters of the camera are the extrinsic rotation matrix, and the translation vector (R, t) of the optical center of the lens of one camera to the optical center of the lens of another camera, each camera can be corrected based on the above parameters to obtain the pixel focal length of each camera after correction, and finally based on the formula one in the foregoing, the millimeter focal length of each camera is obtained.

[0107] Optionally, the process of correcting each camera based on the intrinsic parameters, distortion coefficients and extrinsic parameters of each camera in the step S234 to obtain the millimeter focal length of each camera is actually a process of correcting each camera based on the pixel focal length, distortion coefficients and extrinsic parameters of each camera to finally obtain the millimeter focal length of each camera. The specific process can be realized by the following steps:

[0108] On the basis of Figure 7 , please refer to Figure 8 , Figure 8 Fig. 6 shows a flowchart of a camera and IMU joint calibration method provided by an embodiment of the application, the intrinsic parameters of each camera include the pixel focal length of each camera, the step of correcting each camera based on the intrinsic parameters, distortion coefficients and extrinsic parameters of each camera in the step S234 to obtain the millimeter focal length of each camera, including:

[0109] S2341, based on the pixel focal length and distortion coefficients of each camera, the corresponding camera is corrected for distortion to obtain each camera after distortion correction.

[0110] S2342, based on the extrinsic parameters of each camera, the camera after distortion correction is corrected for stereo to obtain the pixel focal length of each camera after correction as the second to-be-calculated intrinsic parameter.

[0111] S2343, the size of each camera chip corresponding to each camera element in the width direction and the height direction is obtained as the second to-be-calculated parameter.

[0112] S2344, based on the second to-be-calculated intrinsic parameter and the second to-be-calculated parameter, the millimeter focal length of each camera is obtained.

[0113] The above steps realize the process of correcting distortion based on the pixel focal length and distortion coefficients of each camera, correcting the camera after distortion correction for stereo based on the extrinsic parameters of each camera, and finally obtaining the millimeter focal length of each camera according to the size of each camera chip corresponding to each camera element in the width direction and the height direction.

[0114] Exemplarily, if the camera intrinsic parameters are set as the pixel focal lengths fx1, fy1, fx2, fy2 of the cameras, and the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2 are set based on the above, the translation vector (R, t) of the optical center of the lens of one camera to the optical center of the lens of the other camera.

[0115] Further, the corrected pixel focal lengths of the cameras obtained in the setting step S2342 are fx1', fy1', fx2', and fy2', and the sizes of the corresponding pixels of the camera chips in the width direction and the height direction are a1, b1, a2, and b2, respectively. The millimeter focal lengths of the cameras obtained based on the above formula one can be fx1' / a1, fy1' / b1, fx2' / a2, and fy2' / b2.

[0116] Optionally, the camera extrinsic parameters include an extrinsic translation vector and an extrinsic rotation matrix of the cameras, and the extrinsic translation vector represents a translation vector of the optical center of the lens in one camera to the optical center of the lens in the other camera.

[0117] In the embodiment of the application, the extrinsic parameters include an extrinsic translation vector and an extrinsic rotation matrix of the cameras, so that the corrected pixel focal lengths of the cameras obtained when the cameras are rectified are more accurate.

[0118] Optionally, the distortion coefficients are radial distortion coefficients and tangential distortion coefficients.

[0119] In the embodiment of the application, the distortion coefficients are considered as radial distortion coefficients and tangential distortion coefficients that jointly affect, so that the data of the cameras obtained after rectification based on the above distortion coefficients are more accurate.

[0120] Based on the above camera and IMU joint calibration method, a camera and IMU joint calibration device is given below to execute the steps of the above embodiments and achieve the corresponding technical effects.

[0121] Specifically, Figure 9 An exemplary structural block diagram of the camera and IMU joint calibration device 300 provided in the embodiment of the application is shown in FIG. 3. Figure 9 The device includes a parameter acquisition module 301, a calculation module 302, and a calibration module 303.

[0122] The parameter acquisition module 301 is configured to control a laser scanner to scan a device to be calibrated to obtain point cloud data of the camera chip and the IMU, wherein the device to be calibrated is installed with the IMU and at least one camera, and the camera includes a camera chip and a lens.

[0123] The calculation module 302 is configured to perform calculation based on the point cloud data of the camera chip and the IMU to obtain a translation vector parameter.

[0124] The calibration module 303 is configured to calibrate and correct the camera to obtain a millimeter focal length of the camera in a case that a physical center of the camera chip is aligned with an optical center of the lens, and calculate a translation vector of the IMU from the optical center of the lens according to the translation vector parameter and the millimeter focal length of the camera, so as to make the coordinate systems of the IMU and the camera consistent.

[0125] Based on the same inventive concept, the embodiments of the present application further provide a camera and IMU joint calibration system 100, which comprises a mechanical arm 110, a fixing device 120, a laser scanner 130 and a control unit 140, the control unit 140 being electrically connected with the mechanical arm 110 and the laser scanner 130.

[0126] The mechanical arm 110 is configured to drive the laser scanner 130 to move.

[0127] The fixing device 120 is configured to fix a device to be calibrated (not shown in the figure), the device to be calibrated being installed with an IMU and at least one camera (not shown in the figure), the camera comprising a camera chip and a lens (not shown in the figure).

[0128] The laser scanner 130 is configured to scan the device to be calibrated to obtain point cloud data of the camera chip and the IMU.

[0129] The control unit 140 is configured to control the mechanical arm 110 and the laser scanner 130 to work, and is further configured to implement the camera and IMU joint calibration method provided in the above embodiments and / or possible implementation manners in combination with the above embodiments, so as to make the coordinate systems of the IMU and the camera consistent.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed apparatus, system and method can also be implemented by other manners. The apparatus, system embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, program segment or part of code containing one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] In addition, the functional modules in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0132] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0133] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for joint calibration of a camera and an IMU, characterized in that, A control unit applied to a camera and IMU joint calibration system, the camera and IMU joint calibration system further including a laser scanner, the laser scanner being electrically connected to the control unit, the method comprising: The laser scanner is controlled to scan the device to be calibrated to obtain point cloud data of the camera chip and IMU. The device to be calibrated is equipped with an IMU and at least one camera. The camera includes a camera chip and a lens. The IMU includes a gyroscope and an accelerometer. The data of the gyroscope and the data of the accelerometer are converted to the same coordinate system. The point cloud data of the IMU is the point cloud data of the IMU in the coordinate system. Based on the point cloud data of the camera chip and IMU, the translation vector of the distance between the center of the IMU and the physical center of the camera chip is calculated, and the translation vector is used as the translation vector parameter. With the physical center of the camera chip and the optical center of the lens aligned, the camera is calibrated and corrected to obtain the camera's millimeter focal length. Based on the translation vector parameters and the camera's millimeter focal length, the translation vector of the IMU's center distance from the lens's optical center is calculated to make the coordinate systems of the IMU and the camera consistent. When a camera is installed on the device to be calibrated, the step of calibrating and correcting the camera to obtain its millimeter focal length includes: The camera is calibrated to obtain its intrinsic parameters and distortion coefficients, including the camera's pixel focal length. The pixel focal length of the camera is updated based on the pixel focal length and distortion coefficient to obtain the updated pixel focal length of the camera, which is used as the first intrinsic parameter to be calculated. The size of each phase element of the camera chip in the width and height directions is obtained as the first parameter to be calculated; The camera's millimeter focal length is obtained based on the first intrinsic parameter to be calculated and the first parameter to be calculated.

2. The camera and IMU joint calibration method according to claim 1, characterized in that, When the calibration device has two cameras installed, the step of calibrating and correcting the cameras to obtain their millimeter focal length includes: Each camera is calibrated to obtain its intrinsic parameters, distortion coefficients, and extrinsic parameters. The intrinsic parameters, distortion coefficients, and extrinsic parameters of each camera are used to correct the corresponding camera to obtain the millimeter focal length of each camera.

3. The camera and IMU joint calibration method according to claim 2, characterized in that, The intrinsic parameters of each camera include the pixel focal length of each camera. The step of correcting the corresponding camera based on the intrinsic parameters, distortion coefficients, and extrinsic parameters of each camera to obtain the millimeter focal length of each camera includes: Based on the pixel focal length and distortion coefficient of each camera, distortion correction is performed on the corresponding camera to obtain each camera after distortion correction; Based on the extrinsic parameters of each camera, stereo correction is performed on each of the cameras after distortion correction to obtain the pixel focal length of each of the cameras after correction, which is used as the second intrinsic parameter to be calculated. The size of each phase cell corresponding to each camera chip in the width and height directions is obtained as the second parameter to be calculated. The millimeter focal length of each camera is obtained based on the second intrinsic parameter to be calculated and the second parameter to be calculated.

4. The camera and IMU joint calibration method according to claim 3, characterized in that, The extrinsic parameters of each camera include an extrinsic translation vector and an extrinsic rotation matrix of each camera. The extrinsic translation vector represents the translation vector from the optical center of the lens in one camera to the optical center of the lens in another camera.

5. The camera and IMU joint calibration method according to claim 1 or 2, characterized in that, The distortion coefficients are radial distortion coefficients and tangential distortion coefficients.

6. A camera and IMU joint calibration device, characterized in that, A control unit for a camera and IMU joint calibration system, the camera and IMU joint calibration system further comprising a laser scanner electrically connected to the control unit, the device comprising: The parameter acquisition module is used to control the laser scanner to scan the device to be calibrated in order to obtain point cloud data of the camera chip and the IMU. The device to be calibrated is equipped with an IMU and at least one camera. The camera includes a camera chip and a lens. The IMU includes a gyroscope and an accelerometer. The data of the gyroscope and the data of the accelerometer are converted to the same coordinate system. The point cloud data of the IMU is the point cloud data of the IMU in the coordinate system. The calculation module is used to calculate the translation vector of the center of the IMU from the physical center of the camera chip based on the point cloud data of the camera chip and the IMU, and use the translation vector as the translation vector parameter. The calibration module is used to calibrate and correct the camera when the physical center of the camera chip and the optical center of the lens are aligned, so as to obtain the millimeter focal length of the camera; and to calculate the translation vector of the center of the IMU from the optical center of the lens according to the translation vector parameter and the millimeter focal length of the camera, so as to make the coordinate system of the IMU and the camera consistent. The calibration module is specifically used to calibrate a camera when the device to be calibrated is equipped with a camera, in order to obtain the intrinsic parameters and distortion coefficients of the camera, the intrinsic parameters of the camera including the pixel focal length of the camera; to update the pixel focal length of the camera based on the pixel focal length and distortion coefficients, so as to obtain the updated pixel focal length of the camera as the first intrinsic parameter to be calculated; to obtain the size of each cell of the camera chip in the width and height directions as the first parameter to be calculated; and to obtain the millimeter focal length of the camera based on the first intrinsic parameter to be calculated and the first parameter to be calculated.

7. A camera and IMU joint calibration system, characterized in that, It includes a robotic arm, a fixed device, a laser scanner, and a control unit, wherein the control unit is electrically connected to the robotic arm and the laser scanner; The robotic arm is used to move the laser scanner; The fixing device is used to fix the device to be calibrated, the device to be calibrated is equipped with an IMU and at least one camera, the camera includes a camera chip and a lens; The laser scanner is used to scan the device to be calibrated to obtain point cloud data of the camera chip and IMU; The control unit is used to control the operation of the robotic arm and the laser scanner, and is also used to execute the camera and IMU joint calibration method as described in any one of claims 1-5, so as to make the coordinate systems of the IMU and the camera consistent.

Images