Multi-camera global calibration method and device, and electronic equipment

Abstract

The disclosed embodiments relate to the field of multi-camera global calibration and provide a multi-camera global calibration method, apparatus, and electronic device. The method comprises: obtaining an intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and an auxiliary camera; obtaining a rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera based on the intrinsic parameter matrix and distortion coefficient; obtaining three-dimensional point cloud data for the same target area in the coordinate system of each camera to be calibrated; converting the three-dimensional point cloud data to the coordinate system of the auxiliary camera to obtain auxiliary point cloud data; aligning the auxiliary point cloud data to obtain a transformation matrix from each auxiliary point cloud data to the global coordinate system; determining the final transformation matrix from each camera to be calibrated to the global coordinate system based on the rotation matrix, translation vector, and transformation matrix, and performing global calibration on each camera to be calibrated based on the final transformation matrix. The disclosed embodiments can effectively solve the problems of complex practical operations and low calibration accuracy in multi-camera global calibration with small or no overlapping fields of view.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multi-camera global calibration, and in particular to a multi-camera global calibration method and device, and electronic equipment. Background Art

[0002] Multi-camera global calibration is of great significance to multi-view vision measurement systems. In recent years, with the development of modern industrial manufacturing technology, vision measurement has become non-contact and highly efficient. High-precision industrial vision measurement equipment has high accuracy and no blind spots, which usually requires the use of multiple cameras and requires that they be unified into the same coordinate system.

[0003] The multi-camera calibration method in the prior art is usually completed using a calibration plate. This method combines multiple cameras in pairs for binocular calibration to obtain the corresponding transformation relationship, and finally selects one of the cameras as the reference camera to convert all camera coordinate systems to the reference camera coordinate system. However, the problem with the above method is that when the common field of view between the two cameras is small, the common field of view cannot accommodate the complete calibration plate, and the binocular calibration conditions cannot be met, making calibration difficult or even impossible. When there is no common field of view between the two cameras, although the existing multi-camera calibration method can arrange multiple control points in the entire measurement area and use precision measuring equipment such as dual theodolites or laser trackers to unify these control points into the same coordinate system, this method requires the deployment of a large number of control points, which has the disadvantages of high workload and low work efficiency. Summary of the Invention

[0004] The present disclosure aims to solve at least one of the problems existing in the prior art and provide a multi-camera global calibration method and device, and an electronic device.

[0005] In one aspect of the present disclosure, a multi-camera global calibration method is provided, the multi-camera global calibration method comprising:

[0006] Obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera respectively;

[0007] Based on the intrinsic parameter matrix and the distortion coefficient, respectively obtaining the rotation matrix and translation vector between each of the cameras to be calibrated and the auxiliary camera;

[0008] Obtain the three-dimensional point cloud data of the same target area in each camera coordinate system to be calibrated;

[0009] Based on the rotation matrix and the translation vector, the three-dimensional point cloud data in the coordinate system of each camera to be calibrated is converted to the coordinate system of the auxiliary camera to obtain auxiliary point cloud data of each camera to be calibrated;

[0010] Registering the auxiliary point cloud data based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to a global coordinate system;

[0011] Based on the rotation matrix, the translation vector and the transformation matrix, a final transformation matrix of each camera to be calibrated to the global coordinate system is determined respectively, and based on the final transformation matrix, a global calibration is performed on each camera to be calibrated.

[0012] Optionally, respectively obtaining the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera includes:

[0013] Placing the auxiliary camera at a target position so that the field of view of the auxiliary camera at the target position can cover the fields of view of all cameras to be calibrated;

[0014] Based on a preset target, each of the cameras to be calibrated and the auxiliary camera is used to take a set of calibration images, wherein the preset targets in each set of calibration images are located at different positions in the corresponding camera field of view, and the sum of the positions of the preset targets in each set of calibration images covers the corresponding camera field of view; wherein the preset target includes a plurality of feature points arranged according to a preset arrangement rule, and the plurality of feature points form a non-centrally symmetrical pattern;

[0015] Based on the calibration images, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera.

[0016] Optionally, based on the calibration image, obtaining the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera using Zhang Zhengyou's camera calibration method respectively includes:

[0017] Determining a world coordinate system based on the preset target, and determining the world coordinates of each of the feature points in the calibration image based on the world coordinate system;

[0018] Based on the calibration image and the world coordinates of each of the feature points, respectively determining the image coordinates of each of the feature points in the calibration image in a corresponding camera coordinate system;

[0019] Based on the world coordinates and the image coordinates, the Zhang Zhengyou camera calibration method is used to respectively obtain the intrinsic parameter matrix and the distortion coefficient of each camera to be calibrated and the auxiliary camera.

[0020] Optionally, determining a world coordinate system based on the preset target, and determining the world coordinates of each of the feature points in the calibration image based on the world coordinate system, respectively, includes:

[0021] Taking the target feature point in the preset target as the origin of the world coordinate system, establishing the world coordinate system;

[0022] Based on the world coordinate system and according to the preset arrangement rule, the world coordinates of each of the feature points in the calibration image are determined respectively.

[0023] Optionally, the acquiring, based on the intrinsic parameter matrix and the distortion coefficient, a rotation matrix and a translation vector between each of the cameras to be calibrated and the auxiliary camera, respectively, includes:

[0024] Using each of the cameras to be calibrated and the auxiliary camera to simultaneously shoot the preset target in different postures, to obtain multiple image groups corresponding to each of the cameras to be calibrated;

[0025] Based on the multiple image groups corresponding to each camera to be calibrated, the intrinsic parameter matrix, and the distortion coefficient, the rotation matrix and the translation vector between each camera to be calibrated and the auxiliary camera are respectively determined.

[0026] Optionally, the step of respectively obtaining three-dimensional point cloud data for the same target area in the coordinate systems of the cameras to be calibrated includes:

[0027] Each of the cameras to be calibrated is combined with a laser to form a three-dimensional scanning system;

[0028] Utilizing each of the three-dimensional scanning systems to sequentially perform three-dimensional scanning on the preset step gauge blocks to obtain three-dimensional image data corresponding to each of the cameras to be calibrated;

[0029] The preset step block is three-dimensionally reconstructed based on the three-dimensional image data to obtain the three-dimensional point cloud data of the preset step block in each camera coordinate system to be calibrated.

[0030] Optionally, based on the rotation matrix and the translation vector, converting the three-dimensional point cloud data in the coordinate system of each camera to be calibrated to the coordinate system of the auxiliary camera to obtain the auxiliary point cloud data of each camera to be calibrated includes:

[0031] According to the following formula (1), the auxiliary point cloud data in each of the camera coordinate systems to be calibrated is determined respectively:

[0032] orientationf=R i *orientation i +T i (1)

[0033] Among them, orientation i Represents the 3D point cloud data in the coordinate system of the i-th camera to be calibrated, Ri Represents the rotation matrix between the i-th camera to be calibrated and the auxiliary camera, T i Represents the translation vector between the i-th camera to be calibrated and the auxiliary camera, and orientationf represents orientation i Corresponding auxiliary point cloud data.

[0034] Optionally, determining the final transformation matrix of each camera to be calibrated to the global coordinate system based on the rotation matrix, the translation vector, and the transformation matrix includes:

[0035] According to the following formula (2), the final transformation matrix of each camera to be calibrated to the global coordinate system is determined respectively:

[0036]

[0037] Among them, Tran iq Represents the transformation matrix from the auxiliary point cloud data corresponding to the i-th camera to be calibrated to the global coordinate system, SRT i Represents the final transformation matrix from the i-th camera to be calibrated to the global coordinate system.

[0038] Another aspect of the present disclosure provides a multi-camera global calibration device, the multi-camera global calibration device comprising:

[0039] The first acquisition module is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera respectively;

[0040] A second acquisition module is configured to respectively acquire a rotation matrix and a translation vector between each of the cameras to be calibrated and the auxiliary camera based on the intrinsic parameter matrix and the distortion coefficient;

[0041] The third acquisition module is used to respectively obtain the three-dimensional point cloud data of the same target area in the coordinate system of each camera to be calibrated;

[0042] a conversion module, configured to convert the three-dimensional point cloud data in the coordinate system of each camera to be calibrated into the coordinate system of the auxiliary camera based on the rotation matrix and the translation vector, thereby obtaining auxiliary point cloud data of each camera to be calibrated;

[0043] A registration module, configured to register the auxiliary point cloud data based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to a global coordinate system;

[0044] A global calibration module is used to determine the final transformation matrix of each camera to be calibrated to the global coordinate system based on the rotation matrix, the translation vector and the transformation matrix, and perform global calibration on each camera to be calibrated based on the final transformation matrix.

[0045] Another aspect of the present disclosure provides an electronic device, including:

[0046] at least one processor; and,

[0047] a memory communicatively connected to at least one processor; wherein,

[0048] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the multi-camera global calibration method described above.

[0049] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-camera global calibration method described above.

[0050] Compared with the prior art, the embodiments of the present disclosure, by using an auxiliary camera, can unify the coordinate systems of each camera to be calibrated based on the coordinate system of the auxiliary camera. The operation process is simple and easy, and can effectively solve the problems of complex actual operation and low calibration accuracy of multi-camera global calibration with small overlapping fields of view or no overlapping fields of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0052] Figure 1 A flowchart of a multi-camera global calibration method provided in one embodiment of the present disclosure;

[0053] Figure 2 A graphical representation of a target provided for another embodiment of the present disclosure;

[0054] Figure 3 A schematic diagram of the pose conversion relationship between the camera to be calibrated and the auxiliary camera provided in another embodiment of the present disclosure;

[0055] Figure 4 A schematic diagram of a step gauge block provided in another embodiment of the present disclosure;

[0056] Figure 5A schematic structural diagram of a multi-camera global calibration device provided in another embodiment of the present disclosure;

[0057] Figure 6 A schematic structural diagram of an electronic device provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present disclosure, many technical details are provided to enable readers to better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present disclosure can be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present disclosure. The various embodiments can be combined and referenced with each other under the premise that there is no contradiction.

[0059] One embodiment of the present disclosure relates to a multi-camera global calibration method, the process of which is as follows: Figure 1 Shown, including:

[0060] Step S110 , respectively obtaining the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera.

[0061] Exemplarily, step S110 includes: placing the auxiliary camera at the target position so that the field of view of the auxiliary camera at the target position can cover the field of view of all cameras to be calibrated. Based on the preset target, a set of calibration pictures are taken using each camera to be calibrated and the auxiliary camera respectively, and each preset target in each set of calibration pictures is located at a different position in the field of view of the corresponding camera, and the sum of the positions of each preset target in each set of calibration pictures covers the field of view of the corresponding camera. The preset target includes a plurality of feature points arranged according to a preset arrangement rule, and the plurality of feature points form a non-centrally symmetric figure. Based on the calibration pictures, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera respectively.

[0062] Specifically, such as Figure 2 As shown, the preset target can use multiple dots arranged in an array as feature points. Multiple feature points form a non-centrally symmetrical figure, which means that the figure composed of each feature point cannot overlap with the figure before rotation after being rotated 180 degrees to the left or right. It should be noted that the preset target can also be Figure 2 Targets of other styles other than the targets shown can be used as long as they include multiple feature points arranged according to a preset arrangement rule and the figure composed of the multiple feature points is a non-centrally symmetrical figure.

[0063] When obtaining the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera, the auxiliary camera can be placed in a position where its field of view covers the field of view of all cameras to be calibrated, and the aperture and focal length of the auxiliary camera and the camera to be calibrated are adjusted to meet the clear imaging requirements. The auxiliary camera and each camera to be calibrated are used as follows: Figure 2 A set of 15 to 20 calibration images is taken with the preset target shown. In each set of calibration images, each preset target is located at a different position in the corresponding camera field of view, and the sum of the positions of each preset target in each set of calibration images covers the corresponding camera field of view. Then, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera using each set of calibration images.

[0064] This embodiment uses the same preset target to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera, which can improve the accuracy of the intrinsic parameter calibration of each camera to be calibrated and the auxiliary camera.

[0065] Exemplarily, based on the calibration image, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera, including: determining a world coordinate system based on a preset target, and determining the world coordinates of each feature point in the calibration image based on the world coordinate system. Based on the calibration image and the world coordinates of each feature point, the image coordinates of each feature point in the calibration image in the corresponding camera coordinate system are determined. Based on the world coordinates and image coordinates, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera.

[0066] Specifically, for example, in the preset target such as Figure 2 As shown, this embodiment can use the detectCircleGridPoints function in MATLAB to detect each feature point in the calibration image, and return the image coordinates of each feature point in the corresponding camera coordinate system based on the world coordinates corresponding to the center of each detected feature point.

[0067] This embodiment uses the world coordinates and image coordinates of the feature points to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera, which can further improve the accuracy of the intrinsic parameter calibration of each camera to be calibrated and the auxiliary camera.

[0068] Exemplarily, determining a world coordinate system based on a preset target and determining the world coordinates of each feature point in the calibration image based on the world coordinate system includes: establishing the world coordinate system using the target feature point in the preset target as the origin of the world coordinate system. Based on the world coordinate system, determining the world coordinates of each feature point in the calibration image according to a preset arrangement rule.

[0069] Specifically, for example, in the preset target such as Figure 2 As shown, the first dot in the upper left corner of the preset target, that is, the first dot in the first row, can be used as the target feature point, the center of the target feature point can be used as the origin of the world coordinate system, and the upward direction perpendicular to the plane where the target feature point is located is used as the Z-axis direction. The Z-axis coordinate of the target feature point is marked as 0, thereby obtaining the world coordinate system. Then, according to the arrangement rule of each feature point in the preset target, the center coordinates of each feature point are calculated respectively, and the center coordinates are the world coordinates of the corresponding feature point.

[0070] This embodiment establishes a world coordinate system with the target feature point in the preset target as the origin, which can facilitate calculation using the world coordinate system in subsequent steps.

[0071] Step S120 : Based on the intrinsic parameter matrix and the distortion coefficient, respectively obtain the rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera.

[0072] Exemplarily, step S120 includes: using each camera to be calibrated and the auxiliary camera to simultaneously shoot a preset target in different postures, to obtain multiple image groups corresponding to each camera to be calibrated.

[0073] Specifically, for example, Figure 3 As shown, when the cameras to be calibrated are the first camera to be calibrated 101, the second camera to be calibrated 102, and the third camera to be calibrated 103, the auxiliary camera is 120, and the first target 131, the second target 132, and the third target 133 are preset targets of different postures corresponding to the first camera to be calibrated 101, the second camera to be calibrated 102, and the third camera to be calibrated 103, respectively, the first target 131 in different postures can be photographed simultaneously by the first camera to be calibrated 101 and the auxiliary camera 120 to obtain the first image group corresponding to the first camera to be calibrated 101. Wherein, t=1, 2, ..., N represents the shooting order, N represents the number of shots and N≥3, the first image group The image of the first target 131 taken by the first camera 101 for the tth time and the image of the first target 131 taken by the auxiliary camera 120 for the tth time are included. Afterwards, the second target 132 in different postures is taken by the second camera 102 for the tth time and the auxiliary camera 120 for the tth time, and the second image group corresponding to the second camera 102 for the tth time is obtained. The second image group The third image group corresponding to the third camera 103 to be calibrated is obtained by simultaneously shooting the third target 133 in different postures with the third camera 103 to be calibrated and the auxiliary camera 120. The third group of images It includes the image of the third target 133 taken by the third camera 103 for the tth time and the image of the third target 133 taken by the auxiliary camera 120 for the tth time.

[0074] Based on the multiple image groups, intrinsic parameter matrix and distortion coefficient corresponding to each camera to be calibrated, the rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera are determined respectively.

[0075] Specifically, for example, this step can be to obtain the first image group Input the binocular positioning toolbox in MATLAB, use the fixed intrinsic parameter matrix estimation option in the binocular positioning toolbox to import the intrinsic parameter matrix and distortion coefficient of the first camera 101 to be calibrated, and make the binocular positioning toolbox based on the first image group The rotation matrix R1 and translation vector T1 between the first camera 101 to be calibrated and the auxiliary camera 120 are obtained by using the internal parameter matrix and distortion coefficient of the first camera 101 to be calibrated. Then, the same method is used to use the dual-target calibration toolbox in MATLAB to calibrate the image based on the second image group. And the internal parameter matrix and distortion coefficient of the second camera 102 to be calibrated, the third image group As well as the intrinsic parameter matrix and distortion coefficient of the third camera to be calibrated 103, the rotation matrix R2 and translation vector T2 between the second camera to be calibrated 102 and the auxiliary camera 120, and the rotation matrix R3 and translation vector T3 between the third camera to be calibrated 103 and the auxiliary camera 120 are obtained.

[0076] Step S130 , respectively obtaining three-dimensional point cloud data for the same target area in the coordinate systems of the cameras to be calibrated.

[0077] Exemplarily, step S130 includes: assembling each camera to be calibrated with a laser to form a 3D scanning system. Using each 3D scanning system, a preset step gauge block is sequentially scanned in 3D to obtain 3D image data corresponding to each camera to be calibrated. Based on the 3D image data, the preset step gauge block is 3D reconstructed to obtain 3D point cloud data of the preset step gauge block in the coordinate system of each camera to be calibrated.

[0078] Specifically, for example, combined with Figure 3 When the cameras to be calibrated include a first camera 101, a second camera 102, and a third camera 103, the first camera 101, the second camera 102, and the third camera 103 can be respectively combined with a line laser to form a three-dimensional scanning system, and each three-dimensional scanning system is used to sequentially calibrate the cameras. Figure 4The step block shown is subjected to three-dimensional scanning to obtain the three-dimensional scanning data corresponding to the first camera to be calibrated 101, the second camera to be calibrated 102, and the third camera to be calibrated 103 respectively. Based on the three-dimensional scanning data, the step block is three-dimensionally reconstructed to obtain the three-dimensional point cloud data PC1 of the step block in the coordinate system of the first camera to be calibrated 101, the three-dimensional point cloud data PC2 in the coordinate system of the second camera to be calibrated 102, and the three-dimensional point cloud data PC3 in the coordinate system of the third camera to be calibrated 103.

[0079] It should be noted that the preset step gauge block can be Figure 4 In addition to the step gauge blocks shown, other heights or step shapes can be used. All that is required is to perform 3D scanning on the same step gauge block using each 3D scanning system. When performing 3D scanning on the preset step gauge block using each 3D scanning system, the position and posture of the step gauge block must remain unchanged to prevent any impact on the final calibration accuracy.

[0080] In step S140 , based on the rotation matrix and the translation vector, the three-dimensional point cloud data in the coordinate system of each camera to be calibrated is converted to the coordinate system of the auxiliary camera to obtain the auxiliary point cloud data of each camera to be calibrated.

[0081] Exemplarily, step S140 includes: determining the auxiliary point cloud data of each camera to be calibrated according to the following formula (1):

[0082] orientationf=R i *orientation i +T i (1)

[0083] Among them, orientation i Represents the 3D point cloud data in the i-th camera coordinate system to be calibrated, and is used to indicate the position and posture of the 3D point cloud data in the i-th camera coordinate system to be calibrated. i Represents the rotation matrix between the i-th camera to be calibrated and the auxiliary camera. i Represents the translation vector between the i-th camera to be calibrated and the auxiliary camera. orientationf represents orientation i Corresponding auxiliary point cloud data, used to indicate orientation i The position and posture of the corresponding auxiliary point cloud data.

[0084] In step S150 , the auxiliary point cloud data are registered based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to the global coordinate system.

[0085] Specifically, the global coordinate system here refers to the preset camera coordinate system that is applicable to all cameras to be calibrated. Figure 3 The global coordinate system may be the coordinate system of the first camera 101 to be calibrated, the coordinate system of the second camera 102 to be calibrated, or the coordinate system of the third camera 103 to be calibrated.

[0086] When registering the auxiliary point cloud data, you can select any auxiliary point cloud data corresponding to the camera to be calibrated as the target point cloud, and use the auxiliary point cloud data corresponding to other cameras to be calibrated as the registration point cloud. Use the preset point cloud registration algorithm such as the scaled iterative closest point (ICP) algorithm, the normal distribution transform (NDT) algorithm, etc. to complete the registration between the auxiliary point cloud data corresponding to each camera to be calibrated. The registration result can be used as the transformation matrix of each auxiliary point cloud data to the global coordinate system.

[0087] For example, after converting all the 3D point cloud data into the coordinate system of the auxiliary camera and obtaining each auxiliary point cloud data, when registering the auxiliary point cloud data, select any auxiliary point cloud data PC of the camera to be calibrated. m (1≤m≤I), the auxiliary point cloud data PC m As the target point cloud, and the auxiliary point cloud data PC m The coordinate system of the camera to be calibrated is used as the global coordinate system, and the auxiliary point cloud data PC m Other auxiliary point cloud data PC n (1≤n≤I and n≠m) is used as the registration point cloud, and the preset point cloud registration algorithm is used to complete the registration of the registration point cloud to the target point cloud. The registration result can be used as the transformation matrix of the auxiliary point cloud data to the global coordinate system, where I represents the total number of cameras to be calibrated, and m and n are both positive integers. Specifically, in this embodiment, the auxiliary point cloud data PC1 for the target area in the first camera coordinate system to be calibrated is selected as the target point cloud, the auxiliary point cloud data PC2 for the same target area in the second camera coordinate system to be calibrated, and the auxiliary point cloud data PC3 for the same target area in the third camera coordinate system to be calibrated are selected as the registration point cloud, and the scaled iterative closest point algorithm is used to align the three-dimensional point cloud data to obtain the transformation matrix Tran of each auxiliary point cloud data iq , among which, Tran iq represents the transformation matrix from the i-th registration point cloud, i.e., the i-th auxiliary point cloud data to the target point cloud, i = 1, 2, 3. In particular, the transformation matrix from the auxiliary point cloud data to the target point cloud in the first camera coordinate system to be calibrated can be expressed as E is a 3*3 identity matrix.

[0088] Step S160 : Based on the rotation matrix, the translation vector, and the transformation matrix, respectively determine the final transformation matrix of each camera to be calibrated to the global coordinate system, and perform global calibration on each camera to be calibrated based on the final transformation matrix.

[0089] Exemplarily, based on the rotation matrix, translation vector, and transformation matrix, the final transformation matrix of each camera to be calibrated to the global coordinate system is determined, including:

[0090] According to the following formula (2), the final transformation matrix of each camera to be calibrated to the global coordinate system is determined separately:

[0091]

[0092] Among them, Tran iq Represents the transformation matrix from the auxiliary point cloud data corresponding to the i-th camera to be calibrated to the global coordinate system, SRT i Represents the final transformation matrix of the i-th camera to be calibrated to the global coordinate system. At this point, based on the final transformation matrix of each camera to be calibrated to the global coordinate system, the calibration of all cameras to be calibrated and the global calibration of each camera to be calibrated can be completed.

[0093] Compared with the existing technology, the embodiments of the present disclosure can unify the coordinate systems of each camera to be calibrated based on the coordinate system of the auxiliary camera by using an auxiliary camera. The operation process is simple and easy, and can effectively solve the problems of complex actual operation and low calibration accuracy of multi-camera global calibration with small overlapping fields of view or no overlapping fields of view.

[0094] Another embodiment of the present disclosure relates to a multi-camera global calibration device, such as Figure 5 Shown, including:

[0095] The first acquisition module 501 is used to respectively obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera;

[0096] The second acquisition module 502 is used to respectively acquire the rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera based on the intrinsic parameter matrix and the distortion coefficient;

[0097] The third acquisition module 503 is used to respectively acquire the three-dimensional point cloud data of the same target area in the coordinate system of each camera to be calibrated;

[0098] A conversion module 504 is configured to convert the three-dimensional point cloud data in the coordinate system of each camera to be calibrated into the coordinate system of the auxiliary camera based on the rotation matrix and the translation vector, thereby obtaining auxiliary point cloud data of each camera to be calibrated;

[0099] The registration module 505 is used to register the auxiliary point cloud data based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to the global coordinate system;

[0100] The global calibration module 506 is used to determine the final transformation matrix of each camera to be calibrated to the global coordinate system based on the rotation matrix, the translation vector and the transformation matrix, and perform global calibration on each camera to be calibrated based on the final transformation matrix.

[0101] The specific implementation method of the multi-camera global calibration device provided in the embodiment of the present disclosure can be found in the multi-camera global calibration method provided in the embodiment of the present disclosure, and will not be repeated here.

[0102] Compared with the existing technology, the embodiments of the present disclosure can unify the coordinate systems of each camera to be calibrated based on the coordinate system of the auxiliary camera by using an auxiliary camera, which can effectively solve the problems of complex actual operation and low calibration accuracy of multi-camera global calibration under small overlapping fields of view or no overlapping fields of view.

[0103] Another embodiment of the present disclosure relates to an electronic device, such as Figure 6 Shown, including:

[0104] at least one processor 601; and,

[0105] A memory 602 in communication with at least one processor 601; wherein,

[0106] The memory 602 stores instructions that can be executed by the at least one processor 601. The instructions are executed by the at least one processor 601 so that the at least one processor 601 can perform the multi-camera global calibration method described in the above embodiment.

[0107] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0108] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0109] Another embodiment of the present disclosure relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the multi-camera global calibration method described in the above embodiment is implemented.

[0110] That is, those skilled in the art will understand that all or part of the steps in the methods described in the above embodiments can be implemented by instructing related hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0111] Those skilled in the art will appreciate that the above-mentioned embodiments are specific embodiments for implementing the present disclosure, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present disclosure.

Claims

1. A multi-camera global calibration method, characterized in that: The multi-camera global calibration method includes: Obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera respectively; Based on the intrinsic parameter matrix and the distortion coefficient, respectively obtaining the rotation matrix and translation vector between each of the cameras to be calibrated and the auxiliary camera; Obtain the three-dimensional point cloud data of the same target area in each camera coordinate system to be calibrated; Based on the rotation matrix and the translation vector, the three-dimensional point cloud data in the coordinate system of each camera to be calibrated is converted to the coordinate system of the auxiliary camera to obtain auxiliary point cloud data of each camera to be calibrated; Registering the auxiliary point cloud data based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to a global coordinate system; Based on the rotation matrix, the translation vector and the transformation matrix, a final transformation matrix of each camera to be calibrated to the global coordinate system is determined respectively, and based on the final transformation matrix, a global calibration is performed on each camera to be calibrated.

2. The multi-camera global calibration method according to claim 1, characterized in that: The step of respectively obtaining the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera includes: Placing the auxiliary camera at a target position so that the field of view of the auxiliary camera at the target position can cover the fields of view of all cameras to be calibrated; Based on a preset target, each of the cameras to be calibrated and the auxiliary camera is used to take a set of calibration images, wherein the preset targets in each set of calibration images are located at different positions in the corresponding camera field of view, and the sum of the positions of the preset targets in each set of calibration images covers the corresponding camera field of view; wherein the preset target includes a plurality of feature points arranged according to a preset arrangement rule, and the plurality of feature points form a non-centrally symmetrical pattern; Based on the calibration images, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera.

3. The multi-camera global calibration method according to claim 2, characterized in that: Based on the calibration image, the Zhang Zhengyou camera calibration method is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera, including: Determining a world coordinate system based on the preset target, and determining the world coordinates of each of the feature points in the calibration image based on the world coordinate system; Based on the calibration image and the world coordinates of each of the feature points, respectively determining the image coordinates of each of the feature points in the calibration image in a corresponding camera coordinate system; Based on the world coordinates and the image coordinates, the Zhang Zhengyou camera calibration method is used to respectively obtain the intrinsic parameter matrix and the distortion coefficient of each camera to be calibrated and the auxiliary camera.

4. The multi-camera global calibration method according to claim 3, characterized in that: Determining a world coordinate system based on the preset target, and determining the world coordinates of each of the feature points in the calibration image based on the world coordinate system, respectively, includes: Taking the target feature point in the preset target as the origin of the world coordinate system, establishing the world coordinate system; Based on the world coordinate system and according to the preset arrangement rule, the world coordinates of each of the feature points in the calibration image are determined respectively.

5. The multi-camera global calibration method according to claim 4, characterized in that: The step of respectively obtaining the rotation matrix and the translation vector between each of the cameras to be calibrated and the auxiliary camera based on the intrinsic parameter matrix and the distortion coefficient includes: Using each of the cameras to be calibrated and the auxiliary camera to simultaneously shoot the preset target in different postures, to obtain multiple image groups corresponding to each of the cameras to be calibrated; Based on the multiple image groups corresponding to each camera to be calibrated, the intrinsic parameter matrix, and the distortion coefficient, the rotation matrix and the translation vector between each camera to be calibrated and the auxiliary camera are respectively determined.

6. The multi-camera global calibration method according to any one of claims 1 to 5, characterized in that: The step of respectively obtaining three-dimensional point cloud data for the same target area in the coordinate systems of the cameras to be calibrated includes: Each of the cameras to be calibrated is combined with a laser to form a three-dimensional scanning system; Utilizing each of the three-dimensional scanning systems to sequentially perform three-dimensional scanning on the preset step gauge blocks to obtain three-dimensional image data corresponding to each of the cameras to be calibrated; The preset step block is three-dimensionally reconstructed based on the three-dimensional image data to obtain the three-dimensional point cloud data of the preset step block in each camera coordinate system to be calibrated.

7. The multi-camera global calibration method according to any one of claims 1 to 5, characterized in that: The step of converting the three-dimensional point cloud data in the coordinate system of each camera to be calibrated to the coordinate system of the auxiliary camera based on the rotation matrix and the translation vector to obtain the auxiliary point cloud data of each camera to be calibrated includes: According to the following formula (1), the auxiliary point cloud data in each of the camera coordinate systems to be calibrated is determined respectively: orientationf=R i *orientation i +T i (1) Among them, orientation i Represents the 3D point cloud data in the coordinate system of the i-th camera to be calibrated, R i Represents the rotation matrix between the i-th camera to be calibrated and the auxiliary camera, T i Represents the translation vector between the i-th camera to be calibrated and the auxiliary camera, and orientationf represents orientation i Corresponding auxiliary point cloud data.

8. The multi-camera global calibration method according to claim 7, characterized in that: The step of determining the final transformation matrix of each camera to be calibrated to the global coordinate system based on the rotation matrix, the translation vector, and the transformation matrix includes: According to the following formula (2), the final transformation matrix of each camera to be calibrated to the global coordinate system is determined respectively: Among them, Tran iq Represents the transformation matrix from the auxiliary point cloud data corresponding to the i-th camera to be calibrated to the global coordinate system, SRT i Represents the final transformation matrix from the i-th camera to be calibrated to the global coordinate system.

9. A multi-camera global calibration device, characterized in that: The multi-camera global calibration device comprises: The first acquisition module is used to obtain the intrinsic parameter matrix and distortion coefficient of each camera to be calibrated and the auxiliary camera respectively; A second acquisition module is configured to respectively acquire a rotation matrix and a translation vector between each of the cameras to be calibrated and the auxiliary camera based on the intrinsic parameter matrix and the distortion coefficient; The third acquisition module is used to respectively obtain the three-dimensional point cloud data of the same target area in the coordinate system of each camera to be calibrated; a conversion module, configured to convert the three-dimensional point cloud data in the coordinate system of each camera to be calibrated into the coordinate system of the auxiliary camera based on the rotation matrix and the translation vector, thereby obtaining auxiliary point cloud data of each camera to be calibrated; A registration module, configured to register the auxiliary point cloud data based on a preset point cloud registration algorithm to obtain a transformation matrix from each auxiliary point cloud data to a global coordinate system; A global calibration module is used to determine the final transformation matrix of each camera to be calibrated to the global coordinate system based on the rotation matrix, the translation vector and the transformation matrix, and perform global calibration on each camera to be calibrated based on the final transformation matrix.

10. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-camera global calibration method according to any one of claims 1 to 8.

Images