Camera group calibration method, device, equipment and medium

Abstract

Embodiments of the application disclose a camera group calibration method, device, equipment and medium, and relate to the technical field of camera calibration. The method comprises: acquiring a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determining candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image; determining whether the candidate camera coordinate system is a reference camera coordinate system, and if not, converting the candidate calibration object coordinates in the candidate camera coordinate system into reference calibration object coordinates in the reference camera coordinate system according to a preset camera group transformation matrix; determining corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determining target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data; wherein the calibration error function is determined based on a spherical image of a calibration sphere at different acquisition positions in a field of view overlap region of the camera group and a preset sphere radius. The spatial consistency and robustness of camera group calibration are improved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of camera calibration, in particular to a camera group calibration method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of robot vision perception technology and human-computer cooperation system, due to the ability of simultaneously acquiring color and depth information, the depth perception camera (i.e. RGB-D camera) has been widely applied in three-dimensional target recognition, grasping positioning, path planning and other tasks. In order to realize the spatial fusion and high-precision cooperation between multiple sensors, it is usually necessary to calibrate objects based on multiple depth perception cameras. Therefore, how to improve the spatial consistency and robustness of camera group calibration is crucial. SUMMARY

[0003] The present application provides a camera group calibration method, device, equipment and medium to improve the spatial consistency and robustness of camera group calibration.

[0004] According to an aspect of the present application, a camera group calibration method is provided, comprising:

[0005] acquiring a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determining candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image;

[0006] determining whether the candidate camera coordinate system is a reference camera coordinate system, and if not, converting the candidate calibration object coordinates in the candidate camera coordinate system into reference calibration object coordinates in the reference camera coordinate system according to a preset camera group transformation matrix; wherein the camera group transformation matrix is determined based on a group of calibration board images of a calibration board in different acquisition postures in the field of view overlap region of the camera group;

[0007] determining corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determining target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data; wherein the calibration error function is determined based on a group of sphere images of a calibration sphere in different acquisition positions in the field of view overlap region of the camera group and a preset sphere radius.

[0008] According to another aspect of the present application, a camera group calibration device is provided, comprising:

[0009] a candidate calibration object coordinate determination module configured to acquire a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determine candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image;

[0010] The reference calibration object coordinate determination module is configured to determine whether the candidate camera coordinate system is a reference camera coordinate system, and if not, convert candidate calibration object coordinates in the candidate camera coordinate system into reference calibration object coordinates in the reference camera coordinate system according to a preset camera group transformation matrix, wherein the camera group transformation matrix is determined based on a plurality of calibration board images of the calibration board in different acquisition postures in the field-of-view overlap region of the camera group.

[0011] The target calibration object coordinate determination module is configured to determine corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determine target calibration object coordinates of the calibration object to be calibrated according to the reference calibration object coordinates and the calibration compensation data, wherein the calibration error function is determined based on a plurality of sphere images of the calibration sphere in different acquisition positions in the field-of-view overlap region of the camera group and a preset sphere radius.

[0012] According to another aspect of the present application, an electronic device is provided, comprising:

[0013] one or more processors;

[0014] a memory configured to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform any one of the camera group calibration methods provided by the embodiments of the present application.

[0016] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement any one of the camera group calibration methods provided by the embodiments of the present application when the computer instructions are executed by the processor.

[0017] The embodiment of the present application provides a camera group calibration scheme, the calibration object image of the candidate calibration object in the candidate camera coordinate system is obtained, and the candidate calibration object coordinates in the candidate camera coordinate system are determined according to the calibration object image; whether the candidate camera coordinate system is the reference camera coordinate system is judged, if not, the candidate calibration object coordinates in the candidate camera coordinate system are converted into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix; wherein the camera group transformation matrix is determined based on the calibration board image group under different acquisition postures of the calibration board in the field of view overlap area of the camera group; the corresponding calibration compensation data is determined according to the reference calibration object coordinates and the preset calibration error function, and the target calibration object coordinates of the calibration object to be calibrated are determined according to the reference calibration object coordinates and the calibration compensation data; wherein the calibration error function is determined based on the ball image under different acquisition positions of the calibration ball in the field of view overlap area of the camera group and the preset ball radius. The above scheme converts the candidate calibration object coordinates in the non-reference camera coordinate system into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix, then determines the calibration compensation data corresponding to the reference calibration object coordinates based on the reference calibration object coordinates and the preset calibration error function, finally compensates the reference calibration object coordinates based on the calibration compensation data, and obtains the target calibration object coordinates of the calibration object to be calibrated, thereby improving the accuracy of the determined target calibration object coordinates, and improving the spatial consistency and robustness of the camera group calibration.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flow chart of a camera group calibration method provided by the first embodiment of the present application;

[0021] Figure 2 is a flow chart of a camera group calibration method provided by the second embodiment of the present application;

[0022] Figure 3A is a flow chart of a double camera calibration method provided by the third embodiment of the present application;

[0023] Figure 3B is a schematic diagram of a double chessboard used for double camera rigid calibration provided by the third embodiment of the present application;

[0024] Figure 3C is a schematic diagram of a dual-camera extraction sphere center provided by embodiment three of the present application;

[0025] Figure 3D is a schematic diagram of coordinate system transformation in a hand-eye calibration scene provided by embodiment three of the present application;

[0026] Figure 4 is a structural schematic diagram of a camera group calibration device provided by embodiment four of the present application;

[0027] Figure 5 is a structural schematic diagram of an electronic device for implementing a camera group calibration method provided by embodiment five of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0029] Embodiment one

[0030] Figure 1 is a flowchart of a camera group calibration method provided by embodiment one of the present application. The embodiment can be applicable to the calibration of an object in a coordinate system corresponding to any camera in a camera group. The method can be executed by a camera group calibration device, which can be implemented in software and / or hardware, and can be configured in an electronic device carrying a camera group calibration function.

[0031] Referring to Figure 1 , a camera group calibration method is shown, which includes:

[0032] S110, acquiring a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determining a candidate calibration object coordinate in the candidate camera coordinate system according to the calibration object image.

[0033] Wherein, the to-be-calibrated object refers to an object that needs to be calibrated. The candidate camera coordinate system refers to the coordinate system corresponding to the camera in the camera group used to collect the calibration object image. The camera group includes at least two cameras. The type of camera is not limited in the embodiment of the present application, which can be set by the technician according to experience or needs. For example, the camera in the camera group can be a depth perception camera.

[0034] Wherein, the calibration object image refers to the image of the to-be-calibrated object. The candidate calibration object coordinate refers to the coordinate of the to-be-calibrated object in the candidate camera coordinate system.

[0035] Specifically, for any camera in the camera group, an image of the calibration object under the corresponding candidate camera coordinate system is acquired by the camera; and candidate calibration object coordinates of the calibration object under the camera coordinate system of the camera are determined according to the image of the calibration object.

[0036] In S120, it is determined whether the candidate camera coordinate system is the reference camera coordinate system. If not, the candidate calibration object coordinates under the candidate camera coordinate system are converted into reference calibration object coordinates under the reference camera coordinate system according to a preset camera group transformation matrix.

[0037] The reference camera coordinate system can be understood as a reference camera coordinate system adopted by the camera group. For example, the reference camera coordinate system can be a coordinate system corresponding to any camera in the camera group. The reference camera coordinate system can be determined in advance based on the experience or needs of those skilled in the art.

[0038] The camera group transformation matrix can be used to represent the coordinate correspondence relationship between the cameras in the camera group. The reference calibration object coordinates refer to the coordinates obtained by converting the candidate calibration object coordinates into the reference camera coordinate system.

[0039] For example, the camera group transformation matrix is determined based on a group of calibration board images of the calibration board under different acquisition poses in the field of view overlap region of the camera group. The calibration board refers to a template used to determine the camera group transformation matrix. For example, the calibration board can be a standard double-sided chessboard target. The field of view overlap region refers to the intersection region of the cameras in the camera group for image acquisition. The acquisition pose can be understood as the pose of the calibration board in the field of view overlap region. The group of calibration board images refers to a combination of calibration board images acquired by the cameras in the camera group under any acquisition pose. It should be noted that the calibration board images in the group of calibration board images can be color images of the calibration board.

[0040] In an optional embodiment, the camera group transformation matrix is determined based on the following manner: a group of calibration board images of the calibration board under different acquisition poses in the field of view overlap region of the camera group are acquired, and a corresponding group of candidate extrinsic parameter matrices and a group of candidate re-projection errors are determined according to the group of calibration board images; a group of target extrinsic parameter matrices is determined from the group of candidate extrinsic parameter matrices according to the group of candidate re-projection errors, and the camera group transformation matrix of the camera group is determined according to the group of target extrinsic parameter matrices and a preset relative pose transformation matrix corresponding to the calibration board.

[0041] The candidate extrinsic matrix group refers to a combination of candidate extrinsic matrices corresponding to each calibration board image in the calibration board image group in any acquisition posture. The candidate extrinsic matrix refers to an extrinsic matrix corresponding to any calibration board image. The extrinsic matrix can be understood as a coordinate correspondence relationship between a camera and a calibration board corresponding to any calibration board image. Specifically, the extrinsic matrix can be understood as a coordinate correspondence relationship between a camera and a captured surface of a calibration board corresponding to any calibration board image. For example, if the calibration board is a standard double-sided chessboard target, the extrinsic matrix can be understood as a coordinate correspondence relationship between a camera A in the camera group and a standard double-sided chessboard target a captured by the camera A. The extrinsic matrix can include a rotation matrix and a translation vector.

[0042] The candidate extrinsic matrix group refers to a combination of candidate extrinsic matrices corresponding to each calibration board image in the calibration board image group in any acquisition posture. The candidate extrinsic matrix refers to an extrinsic matrix corresponding to any calibration board image. The extrinsic matrix can be understood as a coordinate correspondence relationship between a camera and a calibration board corresponding to any calibration board image. Specifically, the extrinsic matrix can be understood as a coordinate correspondence relationship between a camera and a captured surface of a calibration board corresponding to any calibration board image. For example, if the calibration board is a standard double-sided chessboard target, the extrinsic matrix can be understood as a coordinate correspondence relationship between a camera A in the camera group and a standard double-sided chessboard target a captured by the camera A. The extrinsic matrix can include a rotation matrix and a translation vector.

[0043] The target extrinsic matrix group refers to a candidate extrinsic matrix group used to determine the camera group transformation matrix. The preset relative pose transformation matrix refers to a pre-set relative pose transformation matrix between different surfaces of a calibration board. For example, if the calibration board is a standard double-sided chessboard target, the preset relative pose transformation matrix refers to a relative pose transformation matrix between the two surfaces of the standard double-sided chessboard target.

[0044] It can be understood that by determining the target extrinsic matrix group from the candidate extrinsic matrix group according to the candidate re-projection error group, and determining the camera group transformation matrix of the camera group according to the target extrinsic matrix group and the preset relative pose transformation matrix corresponding to the calibration board, the accuracy of the determined camera group transformation matrix is improved.

[0045] In an optional embodiment, determining the target extrinsic matrix group from the candidate extrinsic matrix group according to the candidate re-projection error group comprises: determining a candidate group mean error corresponding to the candidate re-projection error group, and taking the smallest candidate group mean error as a target group mean error; and taking the target extrinsic matrix group corresponding to the target group mean error as the target extrinsic matrix group.

[0046] The candidate group mean error refers to a mean value of each candidate re-projection error in any candidate re-projection error group. For example, for any candidate re-projection error group, the mean value of each candidate re-projection error in the candidate re-projection error group is calculated to obtain the candidate group mean error; or, according to the candidate re-projection error corresponding to any calibration board image in the candidate re-projection error group, the mean value of the single face re-projection error corresponding to the calibration board image is determined, and then the mean value of the single face re-projection error corresponding to each calibration board image in the candidate re-projection error group is determined to obtain the candidate group mean error corresponding to the candidate re-projection error group.

[0047] The target group mean error is the minimum candidate group mean error. For example, the candidate group mean errors are compared to determine the minimum candidate group mean error, which is taken as the target group mean error.

[0048] It can be understood that by comparing the candidate group mean errors corresponding to the determined candidate re-projection error groups to obtain the target group mean error, and then taking the candidate extrinsic parameter matrix group corresponding to the target group mean error as the target extrinsic parameter matrix group, the accuracy of the determined target extrinsic parameter matrix group is improved.

[0049] For example, if the camera group includes two depth perception cameras, the dual-camera rigid calibration process (i.e., determining the camera group transformation matrix) can be as follows: sequentially place the double-sided chessboard in the field of view of the two depth perception cameras (i.e., place the standard double-sided chessboard target in the field of view overlap region of the camera group), collect multiple groups of images under different poses (i.e., obtain a group of calibration board images of the standard double-sided chessboard target under different acquisition poses), and respectively use Zhang's calibration method to obtain the extrinsic parameter matrix and the re-projection error of each camera relative to the front and back of the double-sided chessboard (i.e., determine the corresponding candidate extrinsic parameter matrix group and candidate re-projection error group according to the group of calibration board images). Since the spatial pose between the two sides of the double-sided chessboard (i.e., the preset relative pose transformation matrix) is known, the rigid transformation matrix T between camera 1 and camera 2 (i.e., the camera group transformation matrix) can be derived as follows:

[0050] ;

[0051] wherein T represents the camera group transformation matrix; T1 represents the transformation matrix from the b1 side of the double-sided chessboard to the camera 1 coordinate system, i.e., the candidate extrinsic parameter matrix between the b1 side of the double-sided chessboard and the camera 1; T2 represents the transformation matrix from the b2 side of the double-sided chessboard to the camera 2 coordinate system, i.e., the candidate extrinsic parameter matrix between the b2 side of the double-sided chessboard and the camera 2; and B represents the preset relative pose transformation matrix. It should be noted that, in order to improve the robustness of the dual-camera coordinate system transformation, the result with the minimum average re-projection error is taken as the final extrinsic parameter (i.e., the target extrinsic parameter matrix group) to form the initial coordinate unification relationship.

[0052] It should be noted that if the candidate camera coordinate system is the reference camera coordinate system, the candidate calibration object coordinates are taken as the target calibration object coordinates.

[0053] S130, according to the reference calibration object coordinates and the preset calibration error function, determine the corresponding calibration compensation data, and according to the reference calibration object coordinates and the calibration compensation data, determine the target calibration object coordinates of the to-be-calibrated object.

[0054] The calibration error function can be used to characterize the spatial error distribution of the camera group. For example, the calibration error function can be an error function obtained through polynomial regression fitting. Calibration compensation data can be used to compensate for the coordinates of the reference calibration object. The target calibration object coordinates refer to the actual coordinates of the object to be calibrated in the reference camera coordinate system.

[0055] For example, the calibration error function is determined based on the sphere images of the calibration sphere at different acquisition positions within the overlapping field of view of the camera group and a preset sphere radius. Here, the calibration sphere refers to the sphere used to determine the calibration error function. For instance, the calibration sphere can be a red sphere calibration object. The acquisition position can be understood as the location of the calibration sphere within the overlapping field of view. The sphere image refers to the image of the calibration sphere acquired by each camera within the camera group at any acquisition position. The preset sphere radius refers to the radius of the calibration sphere.

[0056] For example, for candidate calibration object coordinates in a non-reference camera coordinate system, the candidate calibration object coordinates are first transformed into reference calibration object coordinates in the reference camera coordinate system based on the camera group transformation matrix. Then, calibration compensation data is used for compensation to obtain high-precision consistent coordinates, i.e., target calibration object coordinates.

[0057] ;

[0058] in, Indicates the coordinates of the target calibration object to be calibrated; (x trans ,y trans ,z trans () represents the coordinates of the reference calibration object to be calibrated, where x trans This represents the coordinates of the object to be calibrated in the x-direction within the reference camera coordinate system, and the y-direction... trans This represents the coordinates of the object to be calibrated in the y-direction within the reference camera coordinate system, and the z-direction... trans This represents the coordinates of the object to be calibrated in the z-direction within the reference camera coordinate system; This represents the calibration compensation data of the object to be calibrated in the x-direction; This represents the calibration compensation data of the object to be calibrated in the y direction; This represents the calibration compensation data of the object to be calibrated in the z-direction.

[0059] The embodiment of the present application provides a camera group calibration scheme, which comprises the following steps: acquiring a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determining candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image; judging whether the candidate camera coordinate system is a reference camera coordinate system, and if not, converting the candidate calibration object coordinates in the candidate camera coordinate system into reference calibration object coordinates in the reference camera coordinate system according to a preset camera group transformation matrix; wherein the camera group transformation matrix is determined based on a group of calibration board images of a calibration board in a field of view overlap area of the camera group under different acquisition postures; determining corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determining target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data; wherein the calibration error function is determined based on a group of sphere images of a calibration sphere in the field of view overlap area of the camera group under different acquisition positions and a preset sphere radius. The above scheme converts the candidate calibration object coordinates in the non-reference camera coordinate system into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix, and then determines the calibration compensation data corresponding to the reference calibration object coordinates based on the reference calibration object coordinates and the preset calibration error function, and finally compensates the reference calibration object coordinates based on the calibration compensation data to obtain the target calibration object coordinates of the to-be-calibrated object, thereby improving the accuracy of the determined target calibration object coordinates, and improving the spatial consistency and robustness of the camera group calibration.

[0060] Embodiment two

[0061] Figure 2 It is a flowchart of a camera group calibration method provided by the embodiment two of the present application. Based on the above embodiments, the embodiment further adds the following steps: acquiring a group of sphere images of a calibration sphere in a field of view overlap area of the camera group under different acquisition positions, and determining candidate sphere center coordinates corresponding to the sphere images according to the sphere images and a preset sphere radius of the calibration sphere; determining a group of sphere center coordinates under the same acquisition position according to the candidate sphere center coordinates, and determining sphere center coordinate residuals of the corresponding group of sphere center coordinates according to the group of sphere center coordinates and the camera group transformation matrix; and determining a calibration error function of the camera group according to the group of sphere center coordinates and the corresponding sphere center coordinate residuals, so as to improve the determination mechanism of the calibration error function. It should be noted that the parts not described in the embodiment of the present application can be referred to the descriptions of other embodiments.

[0062] Referring to Figure 2 The camera group calibration method shown in the figure comprises the following steps:

[0063] S210, acquiring a group of sphere images of a calibration sphere in a field of view overlap area of the camera group under different acquisition positions, and determining candidate sphere center coordinates corresponding to the sphere images according to the sphere images and a preset sphere radius of the calibration sphere.

[0064] The candidate ball center coordinates refer to three-dimensional ball center coordinates of a calibration ball corresponding to any ball image.

[0065] Specifically, for any camera in the camera set, a ball image of the calibration ball at different acquisition positions in the field of view overlap region is acquired by the camera; and according to the ball image corresponding to each camera and the preset ball radius, the candidate ball center coordinates corresponding to the corresponding ball image are determined.

[0066] In an optional embodiment, the candidate ball center coordinates corresponding to the ball image are determined according to the ball image and the preset ball radius of the calibration ball, including: for any ball image, a circle center two-dimensional coordinate corresponding to the ball image is determined according to a ball color image in the ball image, and a circle center depth value is determined according to the circle center two-dimensional coordinate and a ball depth image in the ball image; and the candidate ball center coordinates corresponding to the ball image are determined according to the circle center two-dimensional coordinate, the circle center depth value, camera attribute data of the camera corresponding to the ball image, and the preset ball radius.

[0067] The ball color image refers to a color image of the calibration ball. The circle center two-dimensional coordinate refers to a two-dimensional circle center coordinate of a circle presented by the calibration ball in the ball color image. The ball depth image refers to a depth image of the calibration ball. For example, when the cameras in the camera set acquire the ball image, the ball color image and the ball depth image can be acquired at the same time, that is, any ball image includes a ball color image and a ball depth image.

[0068] The circle center depth value can be understood as the distance between the circle center and the corresponding camera. For example, the depth value of the pixel point corresponding to the circle center two-dimensional coordinate in the ball depth image is the circle center depth value. The camera attribute data refers to the basic attribute data of the camera. For example, the camera attribute data can include the focal length data of the camera and the imaging offset data of the camera.

[0069] It can be understood that by determining the circle center two-dimensional coordinate corresponding to the ball color image in the ball image, determining the circle center depth value according to the ball depth image in the ball image, and determining the candidate ball center coordinates corresponding to the ball image according to the circle center two-dimensional coordinate, the circle center depth value, the camera attribute data of the camera corresponding to the ball image, and the preset ball radius, the accuracy of the determined candidate ball center coordinates is improved.

[0070] In an optional embodiment, the candidate ball center coordinates corresponding to the ball image are determined according to the circle center two-dimensional coordinate, the circle center depth value, the camera attribute data of the camera corresponding to the ball image, and the preset ball radius, including: the circle center three-dimensional coordinate corresponding to the ball image is determined according to the circle center two-dimensional coordinate, the circle center depth value, and the camera attribute data; and the candidate ball center coordinates corresponding to the ball image are determined according to the circle center three-dimensional coordinate and the preset ball radius.

[0071] The three-dimensional coordinates of the center of the circle refer to the three-dimensional coordinates of the center of the circle presented in the ball image on the surface of the calibration ball.

[0072] For example, if the camera set is a combination of two depth perception cameras, after the preliminary registration of the two cameras is completed, a standard red ball is placed in the overlapping field of view of the two cameras, and the corresponding ball color image and ball depth image at different acquisition positions are obtained. Then, each frame of image is de-distorted and cropped, the red region mask is extracted in the HSV (Hue-Saturation-Value, hue-saturation-value) space, and the main contour is detected, and the projection center of the circle is obtained by using the minimum circumscribed circle fitting. Combined with the depth value of the corresponding pixel in the ball depth image, the three-dimensional coordinates of the point on the surface of the ball are recovered, and the coordinates of the center of the ball are inversely deduced according to the radius of the ball. The three-dimensional coordinates of the center of the circle corresponding to any ball image can be determined based on the following formula:

[0073] ;

[0074] wherein (x, y, z) represents the three-dimensional coordinates of the center of the circle corresponding to the ball image; (u, v) represents the two-dimensional coordinates of the center of the circle corresponding to the ball image; depth (u, v) represents the depth value of the center of the circle corresponding to the ball image; f x represents the focal length of the camera corresponding to the ball image in the x direction; f y represents the focal length of the camera corresponding to the ball image in the y direction; c x represents the imaging offset data of the camera corresponding to the ball image in the x direction; c y represents the imaging offset data of the camera corresponding to the ball image in the y direction.

[0075] Further, since the calibration ball has undergone perspective transformation, in order to make the calculation of the center of the ball more accurate, the center of the ball is perspective corrected, and the candidate center of the ball coordinates of the ball image can be determined by the following formula:

[0076] ;

[0077] wherein O represents the candidate center of the ball coordinates corresponding to the ball image; C represents the three-dimensional coordinates of the center of the circle corresponding to the ball image; and r represents the preset ball radius of the calibration ball.

[0078] It should be noted that the process of determining the candidate center of the ball coordinates in the embodiments of the present application is carried out in the coordinate system of the camera corresponding to the ball image.

[0079] It can be understood that the three-dimensional coordinates of the center of the circle corresponding to the ball image are determined according to the two-dimensional coordinates of the center of the circle, the depth value of the center of the circle and the camera attribute data, and then the candidate center of the ball coordinates corresponding to the ball image is determined according to the three-dimensional coordinates of the center of the circle and the preset ball radius, thereby improving the accuracy of the determined candidate center of the ball coordinates.

[0080] S220, determine a ball center coordinate group under the same acquisition position according to the candidate ball center coordinates, and determine a ball center coordinate residual of the corresponding ball center coordinate group according to the ball center coordinate group and the camera group transformation matrix.

[0081] The ball center coordinate group refers to a combination of candidate ball center coordinates corresponding to each camera in the camera group under the same acquisition position. For example, for any acquisition position, the candidate ball center coordinates determined by each camera in the camera group under the acquisition position are summarized to obtain the ball center coordinate group under the acquisition position.

[0082] In an optional embodiment, determining the ball center coordinate residual of the corresponding ball center coordinate group according to the ball center coordinate group and the camera group transformation matrix comprises: for the candidate ball center coordinates in any ball center coordinate group, taking the candidate ball center coordinates in the reference camera coordinate system in the ball center coordinate group as the reference ball center coordinates; converting the candidate ball center coordinates in the non-reference camera coordinate system in the ball center coordinate group into the reference ball center coordinates in the reference camera coordinate system according to the camera group transformation matrix; and determining the ball center coordinate residual corresponding to the ball center coordinate group according to the reference ball center coordinates and the reference ball center coordinates.

[0083] The ball center coordinate residual refers to the three-dimensional residual between the reference ball center coordinates and the reference ball center coordinates in the ball center coordinate group.

[0084] The reference ball center coordinates refer to the candidate ball center coordinates in the reference camera coordinate system in the ball center coordinate group. The non-reference camera coordinate system refers to the non-reference camera coordinate system. The reference ball center coordinates refer to the coordinates obtained by converting the candidate ball center coordinates in the non-reference camera coordinate system to the reference camera coordinate system.

[0085] For example, if the camera group includes two depth perception cameras, the candidate ball center coordinates in the non-reference camera coordinate system are mapped to the reference camera coordinate system using the rigid transformation matrix (i.e. the camera group transformation matrix), and the three-dimensional residual (i.e. the ball center coordinate residual) between the reference ball center coordinates is calculated, which can be determined based on the following formula:

[0086] ;

[0087] Wherein, represents the ball center coordinate residual corresponding to the i-th ball center coordinate group; represents the reference ball center coordinates in the i-th ball center coordinate group; T represents the camera group transformation matrix; represents the candidate ball center coordinates in the i-th ball center coordinate group in the non-reference camera coordinate system; represents the ball center coordinate residual in the x direction of the i-th ball center coordinate group; represents the spherical center coordinate residual of the i-th spherical center coordinate group in the y direction; represents the spherical center coordinate residual of the i-th spherical center coordinate group in the z direction.

[0088] It can be understood that by taking the candidate spherical center coordinates in the reference camera coordinate system as the reference spherical center coordinates, converting the candidate spherical center coordinates in the non-reference camera coordinate system into the reference spherical center coordinates, updating the corresponding spherical center coordinate group based on the reference spherical center coordinates and the reference spherical center coordinates, and determining the spherical center coordinate residual corresponding to the corresponding updated spherical center coordinate group, the accuracy of the determined spherical center coordinate residual is improved.

[0089] S230, determining the calibration error function of the camera group according to the spherical center coordinate group and the corresponding spherical center coordinate residual.

[0090] Exemplarily, a polynomial regression model can be constructed with the reference spherical center coordinates and the reference spherical center coordinates in the spherical center coordinate group as independent variables, respectively. , and The spatial error distribution function, i.e. the calibration error function, is obtained. The calibration error function can be determined by the following formula:

[0091] ;

[0092] Wherein, X i represents the i-th spherical center coordinate group, and the candidate spherical center coordinates in the spherical center coordinate group in the non-reference camera coordinate system have been converted into the reference spherical center coordinates; g x represents the calibration error function of the camera group in the x direction; g y represents the calibration error function of the camera group in the y direction; g z represents the calibration error function of the camera group in the z direction.

[0093] S240, obtaining the calibration object image of the calibration object in the candidate camera coordinate system, and determining the candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image.

[0094] S250, judging whether the candidate camera coordinate system is the reference camera coordinate system, if not, converting the candidate calibration object coordinates in the candidate camera coordinate system into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix.

[0095] Wherein, the camera group transformation matrix is determined based on the calibration board image group of the calibration board in the field of view overlap region of the camera group under different acquisition postures.

[0096] S260, determine the corresponding calibration compensation data according to the reference calibration object coordinates and the preset calibration error function, and determine the target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data.

[0097] The calibration error function is determined based on the ball images of the calibration ball at different acquisition positions in the field-of-view overlap region of the camera set and the preset ball radius. The camera set calibration scheme provided by the embodiment of the application adds the acquisition of the ball images of the calibration ball at different acquisition positions in the field-of-view overlap region of the camera set, determines the candidate ball center coordinates corresponding to the ball images according to the ball images and the preset ball radius of the calibration ball, determines the ball center coordinate sets at the same acquisition position according to the candidate ball center coordinates, determines the ball center coordinate residuals of the corresponding ball center coordinate sets according to the ball center coordinate sets and the camera set transformation matrix, determines the calibration error function of the camera set according to the ball center coordinate sets and the corresponding ball center coordinate residuals, and improves the determination mechanism of the calibration error function. The above scheme determines the corresponding candidate ball center coordinates according to the ball images and the preset ball radius, generates the ball center coordinate sets at the same acquisition position according to the candidate ball center coordinates, determines the ball center coordinate residuals of the corresponding ball center coordinate sets based on the ball center coordinate sets and the camera set transformation matrix, and finally determines the calibration error function of the camera set based on the ball center coordinate sets and the corresponding ball center coordinate residuals, thereby improving the accuracy of the determined calibration error function.

[0098] Embodiment three

[0099] The embodiment of the application provides an alternative example on the basis of the above-mentioned embodiments. It should be noted that the parts not described in detail in the embodiment of the application can refer to the description of other embodiments.

[0100] The application relates to the technical field of robot vision perception and space registration, in particular to a two-stage multi-depth perception camera and hand-eye high-precision calibration method, which is suitable for a multi-camera-robot integrated system requiring high-precision space fusion in agricultural robots, industrial detection, intelligent manufacturing and the like.

[0101] With the rapid development of robot vision perception technology and human-machine cooperation systems, depth perception cameras have been widely applied to three-dimensional target recognition, grasping positioning, path planning and the like due to the ability of simultaneously acquiring color and depth information. In order to realize space fusion and high-precision cooperation among multiple sensors, the extrinsic parameters of multiple depth perception cameras need to be calibrated, and hand-eye calibration between the cameras and the robot end is further completed. The precision and robustness of the above calibration process directly affect the three-dimensional perception and operation performance of the system in a complex environment.

[0102] In the prior art, current depth perception multi-camera systems and hand-eye calibration methods have made certain progress in modeling accuracy, optimization strategy and partial algorithm performance, but in practical applications, they still face many technical difficulties and bottlenecks, which seriously restrict their stability and universality in complex environments. First, existing methods generally rely on ideal scenes with high image quality, sufficient field of view overlap or uniform illumination. Under complex conditions such as strong light interference, texture loss and target occlusion, the system performance is easily reduced, and it is difficult to achieve a stable and reliable calibration process. Second, most calibration methods construct ideal rigid geometric models, and usually only correct the distortion of RGB images, while ignoring the widespread structural distortion and noise interference in depth maps. Such nonlinear errors are difficult to avoid in multi-camera fusion, and if there is no systematic residual modeling and compensation mechanism, it is easy to cause spatial alignment deviation, three-dimensional reconstruction distortion and end control error. Further, some high-precision calibration methods still rely on specific equipment support (such as high-precision mechanical turntable, laser scanner or auxiliary camera array, etc.), which not only increases the deployment cost and operation complexity, but also limits its applicability in resource-limited or mobile platforms. Most of the current multi-camera and hand-eye calibration methods regard camera calibration and hand-eye calibration as two independent processes, lack a unified spatial fusion framework, and ignore the residual error modeling problem in traditional rigid calibration models. In complex environments, these unmodeled errors will significantly reduce the three-dimensional positioning and execution accuracy, restricting the further improvement of system performance. Therefore, there is an urgent need for a multi-camera and hand-eye integrated calibration method with simple structure, high precision, strong robustness and residual compensation mechanism to meet the actual needs of high-precision robot operation systems.

[0103] The purpose of the embodiment of the present application is to propose a two-stage multi-depth perception camera and hand-eye high-precision calibration method, which has the characteristics of simple structure, high precision and strong robustness. This method unifies the spatial calibration framework between camera-camera and camera-robot, and introduces a residual modeling mechanism to compensate for the unmodeled errors in the traditional rigid calibration model, thereby improving the overall three-dimensional positioning and execution accuracy of the system.

[0104] The overall method framework of the embodiment of the present application includes a double-station depth perception camera, a standard double-sided checkerboard target, a red spherical calibration object, and a robot arm and an end effector. This method sequentially completes the rigid calibration between the two cameras, the spherical center extraction, the residual modeling and compensation, and can be extended for hand-eye calibration between the camera and the robot arm.

[0105] For example, see Figure 3AA flowchart of the dual-camera calibration method is shown. A double-sided chessboard (i.e., a standard double-sided chessboard target) is placed in any position in the field of view overlap region between the two cameras, and the two cameras respectively capture the chessboard images (i.e., the calibration board images in the calibration board image group) of the double-sided chessboard in different capture poses, as shown in Figure 3B A schematic diagram of the double-sided chessboard for dual-camera rigid calibration is shown, in which camera 1 captures the chessboard image of the b1 side of the double-sided chessboard, and camera 2 captures the chessboard image of the b2 side of the double-sided chessboard; the intrinsic parameters of camera 1 (i.e., the basic attribute data of camera 1) and the intrinsic parameters of camera 2 (i.e., the basic attribute data of camera 2) are obtained; for a group of chessboard images (i.e., a calibration board image group) in any capture pose, based on Zhang's calibration method, the candidate extrinsic parameter matrix T1 of camera 1 and the candidate re-projection error in this capture pose, and the candidate extrinsic parameter matrix T2 of camera 2 and the candidate re-projection error in this capture pose are determined; according to the candidate re-projection error of different chessboards in this capture pose, the average re-projection error in this capture pose is determined (i.e., the candidate group mean error in any capture pose is determined); the two candidate extrinsic parameter matrices corresponding to the smallest average re-projection error are taken as the target extrinsic parameter matrix (i.e., the candidate extrinsic parameter matrix group corresponding to the smallest candidate group mean error is taken as the target extrinsic parameter matrix group); according to the target extrinsic parameter matrix and the preset relative pose transformation matrix, the rigid transformation matrix T between the two cameras (i.e., the camera group transformation matrix) is determined.

[0106] Further, a standard red sphere is placed in the field of view overlap region of the two cameras; the two cameras respectively acquire candidate sphere images (including sphere color images and sphere depth images) of the standard red sphere at different capture positions in the field of view overlap region; for a sphere image at any capture position, the center two-dimensional coordinates are determined according to the sphere color image in the aforementioned sphere image; the distance between the center and the corresponding camera (i.e., the center depth value) is determined according to the sphere depth image in the aforementioned sphere image; the center three-dimensional coordinates of the center on the sphere surface are determined according to the distance between the center and the corresponding camera, the basic attribute data of the corresponding camera (including the focal length and the imaging offset), and the center two-dimensional coordinates; the candidate sphere center coordinates corresponding to the aforementioned sphere image are determined according to the preset sphere radius of the standard red sphere and the center three-dimensional coordinates. Through the above process, the candidate sphere center coordinates of camera 1 at different capture positions , and the candidate sphere center coordinates of camera 2 at different capture positions .

[0107] Further, the candidate sphere center coordinates corresponding to the two cameras at the same capture position are taken as a set of paired sample point sets (i.e., a sphere center coordinate group); for any set of paired sample point sets, the candidate sphere center coordinates Mapping to the reference camera coordinate system (i.e. the coordinate system corresponding to camera 1) to obtain the reference sphere center coordinates ; according to the reference sphere center coordinates in the reference camera coordinate system , determine the three-dimensional residual error of the set of paired sample points (i.e. the sphere center coordinate residual error); through polynomial fitting of the error model, based on the three-dimensional residual error and the candidate sphere center coordinates that have undergone coordinate conversion, determine the calibration error function between the two cameras.

[0108] For example, referring to the schematic diagram of extracting the sphere center of the double camera shown in Figure 3C . The diagram shows a model of the double camera system (C1 and C2) imaging the calibration sphere. The calibration sphere has a sphere center O, a preset sphere radius r, and the calibration sphere is projected onto the image plane of the two cameras as P C1 and P C2 ; P 1nr and P 1nw represent the boundary projection of the sphere in the direction of camera C1 (i.e. camera 1); P 2nr and P 2nw represent the boundary projection of the sphere in the direction of camera C2 (i.e. camera 2); the ray between Q 1R and the sphere represents the depth distance d (i.e. the center depth value) from camera C1 to the sphere of the calibration sphere; the three-dimensional coordinates of the corresponding point on the sphere (i.e. the three-dimensional center coordinates) can be deduced in combination with the image projection position, and the three-dimensional sphere center coordinates (i.e. the candidate sphere center coordinates) can be deduced in combination with the preset sphere radius r; X 1R , Y 1R , and Z 1C are the reference coordinate axes of camera C1; X 2R , Y 2R , and Z 2C are the reference coordinate axes of camera C2; Q 1R is the origin of the reference coordinate axis of camera C1; Q 2R is the origin of the reference coordinate axis of camera C2; m and n represent the pixel coordinate axes.

[0109] For example, in the sphere center extraction process of fusing color and depth information, the sphere color image is preprocessed by de-distortion, image cropping, color segmentation, etc., and then the circle center two-dimensional coordinates are determined based on the preprocessed sphere color image. The sphere depth image is preprocessed by de-distortion and image cropping, etc., and then the circle center two-dimensional coordinates corresponding to the circle center depth value are determined based on the preprocessed sphere depth image; according to the circle center two-dimensional coordinates and the circle center depth value, the candidate sphere center coordinates are determined.

[0110] ​The camera group calibration method provided in this invention can be applied to hand-eye calibration scenarios. For example, after completing residual compensation between two cameras, this idea is further extended to hand-eye calibration tasks. A double-sided checkerboard pattern is fixed to the end effector of a robotic arm. The extrinsic parameter matrix of the checkerboard pattern in the camera coordinate system and the pose of the robotic arm end effector in the world coordinate system are obtained. Based on the fixed connection relationship, the hand-eye calibration matrix from the camera to the robotic arm base is derived. Subsequently, referring to the multi-camera residual modeling concept, by acquiring the three-dimensional center coordinates of the calibration sphere under different end effector poses, a residual compensation model from the camera coordinates to the robotic arm base coordinates is established to achieve error correction in hand-eye calibration.

[0111] For example, see Figure 3D The diagram illustrates the coordinate system transformation in a hand-eye calibration scenario. It shows the coordinate transformation relationships between the camera, calibration plate, robotic arm end effector, and base. W, E, F, and H represent the world coordinate system (robotic arm base), robotic arm end effector coordinate system, calibration plate coordinate system, and camera coordinate system, respectively. The camera's extrinsic parameter matrix T, from the calibration plate to the camera, can be obtained through checkerboard calibration. hf Combined with the calibration plate rigidly mounted on the end effector, the end transformation matrix T fe And the base transformation matrix T from the end to the base obtained from the forward kinematics. ew The total transformation matrix T of the camera relative to the world coordinate system can be derived. wh =T ew T fe T hf Among them, X h ,Y h Z h The coordinate axes in the camera coordinate system; X f ,Y f Indicates the coordinate axes in the calibration plate coordinate system; X e ,Y e Z e The coordinate axes in the end-effector coordinate system; X w ,Y w Z w This represents the coordinate axes in the coordinate system of the robot arm base.

[0112] For example, in Figure 3D In the schematic diagram shown, the end effector of the robotic arm can be analogized to one of the cameras in the dual-camera calibration method provided in this embodiment of the invention. Specifically, the calibration plate can be fixed to the end effector of the robotic arm first. Based on the coordinates of any point in the coordinate system corresponding to the end effector of the robotic arm and the coordinate system of the calibration plate, the end effector transformation matrix T between the end effector of the robotic arm and the calibration plate can be determined. feAccording to the method for determining the candidate extrinsic parameter matrix between the camera and the calibration board provided in the embodiments of the present invention, the camera extrinsic parameter matrix T between the camera and the calibration board is obtained. hf Determine the base transformation matrix T between the end effector of the robotic arm and the base in the robot. ew According to the terminal transformation matrix T fe Camera extrinsic matrix T hf and base transformation matrix T ew Determine the total transformation matrix T between the camera and the base. wh .

[0113] The end-effector transformation matrix represents the coordinate correspondence between the robotic arm's end effector and the calibration plate. The camera extrinsic parameter matrix represents the coordinate correspondence between the camera and the calibration plate. The base transformation matrix represents the coordinate correspondence between the robotic arm's end effector and the base. The overall transformation matrix represents the coordinate correspondence between the camera and the base.

[0114] For example, in this embodiment of the invention, a calibration ball is fixed at the end of a robotic arm, and the coordinates of the end-point center of the calibration ball at different positions are determined. Based on the base transformation matrix, the coordinates of the end-point center are converted to the coordinates of the base center. According to the method for determining candidate center coordinates of the calibration ball provided in this embodiment of the invention, the camera center coordinates of the calibration ball at different positions are determined. Using the base center coordinates as a reference, the camera center coordinates are converted to transformed center coordinates in the robotic arm base coordinate system based on the total transformation matrix. Based on the base center coordinates and the transformed center coordinates at different positions, the three-dimensional residual between the robotic arm base and the camera is determined. Based on the three-dimensional residual, the base center coordinates, and the transformed center coordinates, the coordinate error function between the robotic arm base and the camera is determined.

[0115] Among them, the end-point sphere center coordinates are the coordinates of the index sphere in the robot arm's end-point coordinate system. The base sphere center coordinates are the coordinates of the index sphere in the robot arm's base coordinate system. The camera sphere center coordinates are the coordinates of the index sphere in the camera coordinate system. Sphere center transformation coordinates refer to converting the camera sphere center coordinates to coordinates in the robot arm's base coordinate system. The sphere center three-dimensional residual refers to the three-dimensional residual between the base sphere center coordinates and the sphere center transformation coordinates at the same location. The coordinate error function can characterize the spatial error distribution pattern between the robot arm's base and the camera.

[0116] For example, the initial object coordinates of the candidate object are obtained by the camera; the initial object coordinates are converted into reference object coordinates according to the total transformation matrix; the object compensation data is determined according to the reference object coordinates and the coordinate error function; and the target object coordinates of the candidate object are determined according to the reference object coordinates and the object compensation data.

[0117] Wherein, the candidate object refers to an object that needs to be calibrated. The initial object coordinate refers to the coordinate of the candidate object in the camera coordinate system. The reference object coordinate refers to the coordinate of the candidate object in the robot base coordinate system. The object compensation data refers to compensation data for the reference object coordinate. The target object coordinate refers to the actual coordinate of the candidate object in the robot base coordinate system.

[0118] The embodiment of the application provides a two-stage multi-depth perception camera and hand-eye high-precision calibration method, introduces a red standard sphere as an auxiliary calibration tool, extracts the three-dimensional coordinates of the sphere center through color and depth combination, and constructs a multi-dimensional residual sample set. A unified spatial registration framework is proposed, which can not only solve the extrinsic parameter solving between cameras, but also can be extended to the hand-eye calibration task between cameras and robots. Without introducing complex hardware, the full-link calibration accuracy is improved. The residual compensation mechanism is extended to the hand-eye calibration task, and the sphere center observation value under different poses of the robot is combined to establish a hand-eye residual compensation model, which effectively eliminates the non-ideal transformation error between mechanical motion and perception.

[0119] The embodiment of the application effectively compensates for the nonlinear error under the traditional rigid model through the method of double-sided chessboard combined with sphere center residual modeling, and improves the spatial consistency and robustness of the system. The method only relies on conventional depth perception cameras and standard targets, has low hardware cost, flexible deployment, and is suitable for agricultural picking robots, industrial quality inspection, warehouse logistics and other practical scenes.

[0120] Embodiment four

[0121] Figure 4 is a structural schematic diagram of a camera group calibration device provided by the fourth embodiment of the application. The embodiment can be applicable to the case of calibrating an object in the coordinate system corresponding to any camera in the camera group. The method can be executed by a camera group calibration device, which can be realized in software and / or hardware, and can be configured in an electronic device that carries out the camera group calibration function.

[0122] As Figure 4 shown, the device includes a candidate calibration object coordinate determination module 410, a reference calibration object coordinate determination module 420, and a target calibration object coordinate determination module 430. Among them,

[0123] The candidate calibration object coordinate determination module 410 is configured to acquire a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determine a candidate calibration object coordinate in the candidate camera coordinate system according to the calibration object image.

[0124] The reference calibration object coordinate determination module 420 is used to determine whether the candidate camera coordinate system is a reference camera coordinate system. If not, it converts the candidate calibration object coordinates in the candidate camera coordinate system into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix. The camera group transformation matrix is ​​determined based on the calibration board image group under different acquisition postures in the field of view overlap area of ​​the camera group.

[0125] The target calibration object coordinate determination module 430 is used to determine the corresponding calibration compensation data based on the reference calibration object coordinates and the preset calibration error function, and to determine the target calibration object coordinates of the object to be calibrated based on the reference calibration object coordinates and the calibration compensation data; wherein, the calibration error function is determined based on the sphere image of the calibration sphere at different acquisition positions in the overlapping field of view of the camera group and the preset sphere radius.

[0126] This invention provides a camera group calibration scheme. It involves acquiring an image of the object to be calibrated in a candidate camera coordinate system, and determining the coordinates of the candidate object in the candidate camera coordinate system based on the image. The scheme then determines whether the candidate camera coordinate system is a reference camera coordinate system. If not, it converts the candidate object coordinates in the candidate camera coordinate system to reference object coordinates in the reference camera coordinate system based on a preset camera group transformation matrix. The camera group transformation matrix is ​​determined based on a set of calibration board images taken at different acquisition postures within the overlapping field of view of the camera group. Corresponding calibration compensation data is determined based on the reference object coordinates and a preset calibration error function. Finally, the target object coordinates of the object to be calibrated are determined based on the reference object coordinates and the calibration compensation data. The calibration error function is determined based on the sphere image of the calibration sphere at different acquisition positions within the overlapping field of view of the camera group and a preset sphere radius. The above scheme transforms the candidate calibration object coordinates in the non-reference camera coordinate system into the reference calibration object coordinates in the reference camera coordinate system according to the preset camera group transformation matrix. Then, based on the reference calibration object coordinates and the preset calibration error function, the calibration compensation data corresponding to the reference calibration object coordinates is determined. Finally, the reference calibration object coordinates are compensated based on the calibration compensation data to obtain the target calibration object coordinates of the object to be calibrated. This improves the accuracy of the determined target calibration object coordinates and enhances the spatial consistency and robustness of the camera group calibration.

[0127] Optionally, the calibration error function is determined based on the following means:

[0128] The candidate sphere center coordinate determination module is used to acquire images of the calibration sphere at different acquisition positions within the overlapping field of view of the camera group, and determine the candidate sphere center coordinates corresponding to the sphere image based on the sphere image and the preset sphere radius of the calibration sphere.

[0129] a ball center coordinate residual determining module, configured to determine a ball center coordinate group under a same acquisition position according to the candidate ball center coordinate, and determine a ball center coordinate residual of a corresponding ball center coordinate group according to the ball center coordinate group and the camera group transformation matrix;

[0130] a calibration error function determining module, configured to determine a calibration error function of the camera group according to the ball center coordinate group and the corresponding ball center coordinate residual.

[0131] Optionally, the candidate ball center coordinate determining module comprises:

[0132] a circle center data determining unit, configured to determine a circle center two-dimensional coordinate of a corresponding ball image according to a ball color image in the ball image, and determine a circle center depth value according to the circle center two-dimensional coordinate and a ball depth image in the ball image;

[0133] a candidate ball center coordinate determining unit, configured to determine a candidate ball center coordinate corresponding to the ball image according to the circle center two-dimensional coordinate, the circle center depth value, camera attribute data of a camera corresponding to the ball image and the preset ball radius.

[0134] Optionally, the candidate ball center coordinate determining unit is specifically configured to:

[0135] determine a circle center three-dimensional coordinate corresponding to the ball image according to the circle center two-dimensional coordinate, the circle center depth value and the camera attribute data;

[0136] determine a candidate ball center coordinate corresponding to the ball image according to the circle center three-dimensional coordinate and the preset ball radius.

[0137] Optionally, the ball center coordinate residual determining module comprises:

[0138] a reference ball center coordinate determining unit, configured to take a candidate ball center coordinate in a reference camera coordinate system in the ball center coordinate group as a reference ball center coordinate according to the candidate ball center coordinate in any ball center coordinate group;

[0139] a reference ball center coordinate determining unit, configured to take a candidate ball center coordinate in a reference camera coordinate system in the ball center coordinate group as a reference ball center coordinate according to the candidate ball center coordinate in any ball center coordinate group;

[0140] a ball center coordinate residual determining unit, configured to determine a ball center coordinate residual corresponding to the ball center coordinate group according to the reference ball center coordinate and the reference ball center coordinate.

[0141] Optionally, the camera group transformation matrix is determined based on the following device:

[0142] The image group data determination module is configured to acquire a plurality of images of the calibration board in different acquisition postures of the camera group in the field of view overlap region of the camera group, and determine a plurality of candidate extrinsic parameter matrixes and a plurality of candidate re-projection error groups according to the plurality of images of the calibration board.

[0143] The camera group transformation matrix determination module is configured to determine a target extrinsic parameter matrix group from the plurality of candidate extrinsic parameter matrixes according to the plurality of candidate re-projection error groups, and determine a camera group transformation matrix of the camera group according to the target extrinsic parameter matrix group and a preset relative pose transformation matrix corresponding to the calibration board.

[0144] Optionally, the camera group transformation matrix determination module comprises:

[0145] The target group mean error determination unit is configured to determine a candidate group mean error corresponding to the plurality of candidate re-projection error groups, and take the smallest candidate group mean error as a target group mean error.

[0146] The target extrinsic parameter matrix group determination unit is configured to take the candidate extrinsic parameter matrix group corresponding to the target group mean error as a target extrinsic parameter matrix group.

[0147] The camera group calibration device provided by the embodiment of the present application can execute the camera group calibration method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing each camera group calibration method.

[0148] In the technical solution of the present application, the collection, storage, use, processing, transmission, provision and disclosure of the calibration object images, the sphere images and the calibration board image groups and the like all conform to the relevant legal regulations and do not violate public order and good customs.

[0149] Embodiment five

[0150] Figure 5 is a structural schematic diagram of an electronic device for implementing the camera group calibration method provided by Embodiment five of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections, and relationships, and their functions, are merely examples and are not intended to limit the implementations described and / or claimed in this document.

[0151] As Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0152] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0153] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the camera set calibration method.

[0154] In some embodiments, the camera set calibration method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the camera set calibration method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the camera set calibration method by any other appropriate means, such as by means of firmware.

[0155] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0156] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0157] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0159] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0160] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0161] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0162] The above detailed description does not limit the scope of the present disclosure. It is understood that various modifications, combinations, sub-combinations, and alternatives can be made to the detailed disclosure without departing from the spirit and principles of the present disclosure. Any modifications, equivalent substitutions, improvements, and the like that are made within the spirit and principles of the present disclosure are included in the scope of the present disclosure.

Claims

1. A camera set calibration method, characterized in that, The method comprises: acquiring a calibration object image of a to-be-calibrated object in a candidate camera coordinate system, and determining candidate calibration object coordinates in the candidate camera coordinate system according to the calibration object image; determining whether the candidate camera coordinate system is a reference camera coordinate system, and if not, converting the candidate calibration object coordinates in the candidate camera coordinate system into reference calibration object coordinates in the reference camera coordinate system according to a preset camera group transformation matrix, wherein the camera group transformation matrix is determined based on a group of calibration board images of a calibration board in a field-of-view overlap region of a camera group under different acquisition postures; determining corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determining target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data, wherein the calibration error function is determined based on a group of sphere images of a calibration sphere in the field-of-view overlap region of the camera group under different acquisition positions and a preset sphere radius; wherein the calibration error function is determined based on the following method: acquiring a group of sphere images of the calibration sphere in the field-of-view overlap region of the camera group under different acquisition positions, and determining candidate sphere center coordinates corresponding to the group of sphere images according to the group of sphere images and the preset sphere radius of the calibration sphere; determining a group of sphere center coordinates under the same acquisition position according to the candidate sphere center coordinates, and determining sphere center coordinate residuals of the corresponding group of sphere center coordinates according to the group of sphere center coordinates and the camera group transformation matrix; determining the calibration error function of the camera group according to the group of sphere center coordinates and the corresponding sphere center coordinate residuals.

2. The method of claim 1, wherein, The method of determining the candidate sphere center coordinates corresponding to the group of sphere images according to the group of sphere images and the preset sphere radius of the calibration sphere comprises: for any sphere image, determining corresponding circle center two-dimensional coordinates according to a sphere color image in the sphere image, and determining a circle center depth value according to the circle center two-dimensional coordinates and a sphere depth image in the sphere image; determining the candidate sphere center coordinates corresponding to the sphere image according to the circle center two-dimensional coordinates, the circle center depth value, camera attribute data of a camera corresponding to the sphere image, and the preset sphere radius.

3. The method of claim 2, wherein, The method of determining the candidate sphere center coordinates corresponding to the sphere image according to the circle center two-dimensional coordinates, the circle center depth value, the camera attribute data of the camera corresponding to the sphere image, and the preset sphere radius comprises: determining circle center three-dimensional coordinates corresponding to the sphere image according to the circle center two-dimensional coordinates, the circle center depth value, and the camera attribute data; determining the candidate sphere center coordinates corresponding to the sphere image according to the circle center three-dimensional coordinates and the preset sphere radius.

4. The method of claim 1, wherein, The method of determining the sphere center coordinate residuals of the corresponding group of sphere center coordinates according to the group of sphere center coordinates and the camera group transformation matrix comprises: for any candidate sphere center coordinates in a group of sphere center coordinates, taking the candidate sphere center coordinates in the group of sphere center coordinates in the reference camera coordinate system as reference sphere center coordinates; converting the candidate sphere center coordinates in the group of sphere center coordinates in the non-reference camera coordinate system into reference sphere center coordinates in the reference camera coordinate system according to the camera group transformation matrix; According to the reference sphere center coordinates and the reference sphere center coordinates, a sphere center coordinate residual corresponding to the sphere center coordinate group is determined.

5. The method of claim 1, wherein, The camera group transformation matrix is determined based on the following manner: A plurality of calibration board images of the calibration board under different acquisition postures in a field of view overlap region of the camera group are acquired, and a corresponding candidate extrinsic parameter matrix group and a candidate re-projection error group are determined according to the calibration board image group; According to the candidate re-projection error group, a target extrinsic parameter matrix group is determined from the candidate extrinsic parameter matrix group, and a camera group transformation matrix of the camera group is determined according to the target extrinsic parameter matrix group and a preset relative pose transformation matrix corresponding to the calibration board.

6. The method of claim 5, wherein, The target extrinsic parameter matrix group is determined from the candidate extrinsic parameter matrix group according to the candidate re-projection error group, and includes: A candidate group mean error corresponding to the candidate re-projection error group is determined, and a smallest candidate group mean error is taken as a target group mean error; A candidate extrinsic parameter matrix group corresponding to the target group mean error is taken as the target extrinsic parameter matrix group.

7. A camera set calibration apparatus characterized by comprising: It includes: A candidate calibration object coordinate determination module is configured to acquire a calibration object image of a to-be-calibrated object under a candidate camera coordinate system, and determine candidate calibration object coordinates under the candidate camera coordinate system according to the calibration object image; A reference calibration object coordinate determination module is configured to determine whether the candidate camera coordinate system is a reference camera coordinate system, and if not, convert the candidate calibration object coordinates under the candidate camera coordinate system into reference calibration object coordinates under the reference camera coordinate system according to a preset camera group transformation matrix; wherein the camera group transformation matrix is determined based on a plurality of calibration board images of a calibration board under different acquisition postures in a field of view overlap region of the camera group; A target calibration object coordinate determination module is configured to determine corresponding calibration compensation data according to the reference calibration object coordinates and a preset calibration error function, and determine target calibration object coordinates of the to-be-calibrated object according to the reference calibration object coordinates and the calibration compensation data; wherein the calibration error function is determined based on a plurality of sphere images of a calibration sphere under different acquisition positions in a field of view overlap region of the camera group and a preset sphere radius; The calibration error function is determined based on the following device: A candidate sphere center coordinate determination module is configured to acquire a plurality of sphere images of a calibration sphere under different acquisition positions in a field of view overlap region of the camera group, and determine candidate sphere center coordinates corresponding to the sphere images according to the sphere images and a preset sphere radius of the calibration sphere; A sphere center coordinate residual determination module is configured to determine a sphere center coordinate group under the same acquisition position according to the candidate sphere center coordinates, and determine a sphere center coordinate residual of a corresponding sphere center coordinate group according to the sphere center coordinate group and the camera group transformation matrix; A calibration error function determination module is configured to determine the calibration error function of the camera group according to the sphere center coordinate group and the corresponding sphere center coordinate residual.

8. An electronic device, comprising: It includes: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a camera group calibration method as claimed in any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements a camera set calibration method as claimed in any one of claims 1-6.

Images