A method and device for calibrating external parameters of a depth camera

Abstract

The present application discloses a method for calibrating the external parameters of a depth camera, characterized in that the method includes: acquiring depth image data, wherein the depth image data includes pixel coordinates and depth values; based on the acquired depth image data, converting the pixel points in the depth image into spatial three-dimensional points in the camera coordinate system; based on the three-dimensional points, acquiring the fitting plane of the three-dimensional points; and obtaining the parameters between the depth camera and the calibration plane according to the current posture relationship between the calibration plane parallel to or overlapping with the fitting plane and the camera coordinate system; wherein the calibration plane includes any current plane parallel to or perpendicular to the bearing surface of the mobile robot body where the depth camera is located. The present application does not require any external tools during the calibration process, and only needs to collect one image to give the calibration result. Calibration can be performed using natural horizontal and vertical ground and wall surfaces, providing an application basis with good robustness and strong real-time performance for applications based on depth image data.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a method for calibrating extrinsic parameters of a depth camera. Background Art

[0002] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Camera calibration is the process of calculating camera parameters through a certain method.

[0003] During the image capture process, the camera's position is not fixed, which means that the camera coordinate system is not a stable coordinate system. The origin and directions of the coordinate axes of the camera coordinate system will change as the camera moves. This requires the introduction of a stable and invariant coordinate system: the world coordinate system, which is an absolute coordinate system. The transformation from the camera coordinate system to the world coordinate system requires rotation and translation, which become the external parameters of the camera.

[0004] A depth camera is a camera that can directly obtain depth image data. Any pixel data in the depth image data obtained is data based on the pixel coordinates using the pixel grayscale value to represent the distance value (depth value) of the spatial point, which can be expressed as p(u, v, d), where u and v are the coordinates of the pixel in the image coordinate system, and d is the depth value of the three-dimensional point in space corresponding to the pixel.

[0005] According to the composition principle of the camera, the depth camera includes passive binocular camera, active binocular camera, time-of-flight (TOF) camera, monocular structured light, etc. At present, the external parameter calibration of the depth camera usually calibrates the external parameters inside the depth camera. For example, for a depth camera composed of binocular cameras, the relative rotation and translation between the two cameras are calibrated. This calibration method is not suitable for calibrating the external parameter relationship between the depth camera and the external coordinate system. Summary of the invention

[0006] The present invention provides a method for calibrating the extrinsic parameters of a depth camera, so as to calibrate the transformation relationship between a camera coordinate system and a calibration plane based on depth image data.

[0007] The present invention provides a method for calibrating the extrinsic parameters of a depth camera, the method comprising:

[0008] Acquire depth image data, wherein the depth image data includes pixel coordinates and depth values;

[0009] Based on the acquired depth image data, pixel points in the depth image are converted into spatial three-dimensional points in the camera coordinate system;

[0010] Based on the three-dimensional points, obtaining a fitting plane of the three-dimensional points;

[0011] According to the current pose relationship between the calibration plane parallel to or coincident with the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained;

[0012] The calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

[0013] Preferably, the depth image data includes at least one depth image, and the step of converting the pixel points in the depth image into spatial three-dimensional points in the camera coordinate system includes:

[0014] According to the mapping geometric model between the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system, the three-dimensional point coordinates corresponding to the pixel point are obtained.

[0015] Preferably, the mapping geometric model between the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system to obtain the three-dimensional point coordinates corresponding to the pixel point includes:

[0016] For any pixel in the depth image,

[0017] The depth value of the pixel point is used as the z coordinate value of the three-dimensional point;

[0018] Obtain a first difference between the x coordinate of the pixel point and the x-direction offset, multiply the ratio of the z coordinate value to the x-direction focal length in the camera intrinsic parameter by the first difference, and use the result as the x coordinate value of the three-dimensional point; the x-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the x direction;

[0019] Obtain a second difference between the y coordinate of the pixel point and the y direction offset, multiply the ratio of the z coordinate value to the y direction focal length in the camera intrinsic parameter by the second difference, and obtain the result as the y coordinate value of the three-dimensional point; the y direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y direction;

[0020] Among them, the x-direction offset of the origin of the camera coordinate system relative to the image coordinate system and the y-direction offset of the origin of the camera coordinate system relative to the image coordinate system are obtained according to the camera intrinsic parameters;

[0021] Convert all pixels in the depth image into three-dimensional point coordinates to obtain a three-dimensional point set.

[0022] Preferably, the method further comprises screening the three-dimensional points in the three-dimensional point set according to a screening strategy to obtain a screened three-dimensional point set; wherein the screening strategy comprises any one of the following conditions or any combination thereof:

[0023] (1) Remove 3D points within a certain range above the depth image according to the camera orientation;

[0024] (2) When there is an initial estimate of the external parameter, the three-dimensional points in the camera coordinate system are transformed into the world coordinate system through the camera external parameter, and the three-dimensional points whose height in the height direction is greater than the preset height threshold are removed;

[0025] (3) For a binocular stereoscopic depth camera, remove three-dimensional points whose depth values ​​are greater than a preset depth threshold;

[0026] (4) For a time-of-flight depth camera, based on the distance difference between a pixel and its adjacent pixels, remove the three-dimensional points whose distance difference is greater than a preset distance threshold.

[0027] Preferably, obtaining a fitting plane of the three-dimensional points based on the three-dimensional points includes:

[0028] According to the screened three-dimensional point set, the random sampling consistent RANSAC algorithm is used to obtain the fitting plane equation of the three-dimensional points, where the number of three-dimensional points is greater than or equal to 3.

[0029] Preferably, the method of obtaining the fitting plane equation of the three-dimensional points by using the random sampling consistent RANSAC algorithm includes:

[0030] Based on the screened three-dimensional point set, a random selection is performed to obtain a current subset consisting of the randomly selected three-dimensional points, wherein the number of three-dimensional points in the subset includes at least 3;

[0031] Based on the 3D points in the current subset, obtain a fitted plane estimate for the subset;

[0032] According to the obtained fitting plane estimate, obtaining the conformity degree of all three-dimensional points in the filtered three-dimensional point set with respect to the fitting plane estimate;

[0033] If the degree of conformity is insufficient, returning to the step of performing random selection based on the screened three-dimensional point set;

[0034] If the degree of conformity is achieved, the fitting plane equation is solved using the estimated interior points of the fitting plane with the best degree of conformity;

[0035] The inner points include three-dimensional points in the screened three-dimensional point set whose distance from the three-dimensional point to the fitting plane with the best conformity is less than a preset distance threshold.

[0036] Preferably, obtaining a fitting plane estimate of the subset based on the three-dimensional points in the current subset includes:

[0037] If the number of 3D points in the current subset is equal to 3, substitute the coordinate values ​​of the 3D points into the fitting plane equation, solve the unknowns in the fitting plane equation, and obtain the fitting plane estimate of the current subset;

[0038] If the number of 3D points in the current subset is greater than 3, the coordinate values ​​of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset.

[0039] Preferably, the step of obtaining the conformity of all three-dimensional points in the filtered three-dimensional point set with respect to the fitted plane estimate based on the obtained fitted plane estimate comprises:

[0040] Calculate the distance value from each three-dimensional point in the filtered three-dimensional point set to the estimated fitting plane,

[0041] The 3D points whose calculated distance values ​​are less than the set distance threshold are regarded as internal points.

[0042] Count the number of inliers,

[0043] Calculate the ratio of the number of inliers to the filtered three-dimensional point cloud to obtain the inlier rate;

[0044] Determining the degree of compliance according to an inlier rate;

[0045] The method of using the inliers of the best fitting plane estimate to solve the fitting plane equation includes taking the fitting plane estimate with the highest inliers rate as the best fitting plane estimate, and resolving the unknowns in the fitting plane equation by the least squares method based on the inliers of the best fitting plane estimate to obtain the fitting plane equation.

[0046] Preferably, determining the degree of conformity according to the inlier rate includes:

[0047] The degree of compliance is determined according to the number of iterations, wherein the number of iterations satisfies:

[0048]

[0049] Where m is the number of 3D points in the subset, η is the set confidence level, ε is the inlier rate, which is the proportion of inliers under the worst condition, or is set to the proportion under the worst condition in the initial state, and is updated to the current maximum inlier rate as the number of iterations increases.

[0050] Preferably, determining the degree of conformity according to the inlier rate includes determining the degree of conformity according to whether the probability that all v subsets are inliers satisfies a set confidence level, wherein the probability that all v subsets are inliers is:

[0051]

[0052] Where λ is the expected number of selections in which all subsets are inliers in the current iteration.

[0053] Preferably, the parameters between the depth camera and the calibration plane are obtained according to the current pose relationship between the calibration plane parallel to or coincident with the fitting plane and the camera coordinate system, including:

[0054] According to the fitting plane equation, the fitting plane normal vector is obtained.

[0055] Obtaining an equation of a calibration plane that is parallel to or coincident with the fitting plane according to the normal vector of the fitting plane;

[0056] Calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane;

[0057] Calculate the rotation of the camera coordinate system relative to the calibration plane, and the rotation transformation of the camera coordinate system relative to the calibration plane.

[0058] The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

[0059] Preferably, the depth image data includes image data of the bearing surface of the mobile robot body.

[0060] The step of obtaining an equation of a calibration plane associated with the fitting plane according to the normal vector of the fitting plane comprises:

[0061] According to the normal vector of the fitting plane, an equation of a calibration plane which is parallel to the fitting plane and has a set distance is obtained; wherein the calibration plane is the supporting ground of the mobile robot body in the world coordinate system;

[0062] The calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane includes:

[0063] According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the calibration plane is calculated to obtain the height transformation of the camera coordinate system relative to the calibration plane;

[0064] The calculation of the rotation of the camera coordinate system relative to the calibration plane to the rotation transformation of the camera coordinate system relative to the calibration plane includes:

[0065] According to the equation parameters of the calibration plane, the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

[0066] Preferably, the depth image data includes elevation image data perpendicular to the bearing surface of the mobile robot body.

[0067] The step of obtaining an equation of a calibration plane associated with the fitting plane according to the normal vector of the fitting plane comprises:

[0068] According to the normal vector of the fitting plane, an equation of a calibration plane which is parallel to the fitting plane and has a set distance is obtained; wherein the calibration plane is perpendicular to the supporting ground of the mobile robot body in the world coordinate system;

[0069] The calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane includes:

[0070] According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the calibration plane is calculated to obtain the distance transformation from the camera coordinate system to the calibration plane;

[0071] The calculation of the rotation of the camera coordinate system relative to the calibration plane to the rotation transformation of the camera coordinate system relative to the calibration plane includes:

[0072] According to the equation parameters of the calibration plane, the rotation of the z-axis of the camera coordinate system relative to the normal vector of the calibration plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

[0073] The present invention provides an electronic device for calibrating external parameters of a depth camera, the electronic device comprising a memory and a processor, wherein:

[0074] The memory is used to store computer programs;

[0075] The processor is used to execute the program stored in the memory to implement any of the above-mentioned depth camera extrinsic parameter calibration methods.

[0076] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to perform any of the above-mentioned methods for calibrating the extrinsic parameters of a depth camera.

[0077] The present application provides a method for calibrating the extrinsic parameters of a depth camera. Based on the three-dimensional points in space under the camera coordinate system corresponding to the depth image data, a fitting plane of the calibration plane is obtained, thereby obtaining the distance transformation and rotation change between the camera coordinate system and the calibration plane according to the current relative posture of the camera coordinate system and the fitting plane, and obtaining the extrinsic parameters of the camera relative to the calibration plane. The calibration process of the present application does not require the use of any external tools in advance. Only one image needs to be collected to give the calibration result. In this way, when an image is collected in real time, the calibration can be performed in real time with the help of a natural horizontal and vertical ground or wall. It is simple and easy to use, has strong real-time performance and strong applicability, and provides a robust and real-time application foundation for applications based on depth image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of the principle of extrinsic calibration of a depth camera based on depth image data.

[0079] Figure 2 A schematic diagram of a process for calibrating the extrinsic parameters of a depth camera according to an embodiment of the present application.

[0080] Figure 3 A schematic diagram of calibration using a fitted plane.

[0081] Figure 4 A schematic diagram of calibration using a second fitting plane (wall) perpendicular to the fitting plane.

[0082] Figure 5 A schematic diagram of data association of the calibration method according to an embodiment of the present application.

[0083] Figure 6 A schematic diagram of a calibration device according to an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0085] See also Figure 1 As shown, Figure 1 A schematic diagram of the principle of extrinsic calibration of a depth camera based on depth image data. The depth camera installed on the mobile robot body acquires depth image data. A presentation of the depth image is shown in the figure. The depth image corresponding to the physical space ground (represented by the thick solid line in the figure) is the dotted frame area in the image. A fitting plane Ax+By+Cz+D=0 is solved based on the depth image data of the space ground. The fitting plane can be regarded as representing the plane where the space ground is located. Since the world coordinate system is usually based on the ground, the plane where the xy axis is located is parallel to the ground plane, and the z axis is perpendicular to the plane where the xy axis is located. The translation trajectory of the camera (x, y coordinate changes) belongs to the same plane. In this way, the rotation of the camera relative to the ground is calibrated. and height (distance), which is equivalent to solving the rotation and height (distance) of the camera relative to the fitting plane, and thus obtaining the rotation transformation and z-axis transformation of the camera relative to the world coordinate system, thereby obtaining the parameters of the imaging model between the ground depth image and the ground plane (horizontal plane) in the physical space; in a broad sense, the imaging model parameters between the depth image and any desired plane in the physical space parallel or perpendicular to the bearing surface of the mobile robot body, that is, the imaging model parameters between the depth image and the calibration plane, which reflect an extrinsic relationship between the depth camera and the external coordinate system, can be regarded as an extrinsic parameter of the depth camera, and are referred to as depth camera extrinsics in this application.

[0086] Based on the above calibration principle, the depth camera extrinsic calibration of the present application is based on the depth image captured by the depth camera, which is converted into three-dimensional points in the camera coordinate system. The three-dimensional points are used to fit a fitting plane associated with the calibration plane, and the extrinsic parameters of the depth camera are obtained through the current posture relationship between the fitting plane and the depth camera.

[0087] See also Figure 2 As shown, Figure 2 A schematic diagram of a process for calibrating the extrinsic parameters of a depth camera according to an embodiment of the present application.

[0088] Step 201 , obtain any depth image data output by a depth camera, and project the depth image into a three-dimensional point cloud through camera internal parameters, that is, the depth image is projected into a set of three-dimensional points.

[0089] In view of the calibration principle, a fitting plane associated with the calibration plane can be fitted using the three-dimensional points of the depth image. The external parameters of the depth camera can be obtained by fitting the current posture relationship between the plane and the depth camera. In this way, the depth image output by the depth camera only needs one at least. For any pixel point p(u, v, d) on the depth image, u and v are the coordinates of the pixel point in the image coordinate system, and d is the gray value of the pixel point, which is also the depth value of the actual three-dimensional point:

[0090] According to the mapping geometric model between the three-dimensional coordinate point and the two-dimensional image plane (in pixels), such as the pinhole model, the relationship between the pixel point in the depth image and the three-dimensional point pc (x, y, z) in the camera coordinate system is obtained:

[0091] The depth value of the pixel point is used as the z coordinate value of the three-dimensional point; a first difference between the x coordinate of the pixel point and the x-direction offset is obtained, wherein the x-direction offset is the offset of the origin of the camera coordinate system (the optical center of the camera) relative to the image coordinate system in the x direction, and the ratio of the z coordinate value to the x-direction focal length in the camera intrinsic parameter is multiplied by the first difference, and the result is used as the x coordinate value of the three-dimensional point;

[0092] Obtain a second difference between the y coordinate of the pixel point and the y direction offset, wherein the y direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y direction, and multiply the ratio of the z coordinate value to the y direction focal length in the camera intrinsic parameter by the second difference, and obtain the result as the y coordinate value of the three-dimensional point.

[0093] Among them, the x-direction offset of the origin of the camera coordinate system relative to the image coordinate system and the y-direction offset of the origin of the camera coordinate system relative to the image coordinate system are obtained according to the camera intrinsic parameters;

[0094] The mathematical expression is:

[0095]

[0096] Among them, f x 、f y are the camera internal parameters, c x 、c y is the offset of the origin of the camera coordinate system in the camera coordinate system relative to the image coordinate system. These parameters correspond to the data in the camera intrinsic parameter matrix K:

[0097]

[0098] According to the above method, each pixel in the depth image can be transformed into a three-dimensional point in the camera coordinate system, and the obtained three-dimensional point set is also called a point cloud.

[0099] Step 202 , screening the three-dimensional point cloud to remove three-dimensional points with large errors, so as to improve the accuracy and success rate of obtaining the fitting plane.

[0100] Taking the ground as an example, in this step, the three-dimensional point cloud calculated in step 201 is screened, with the purpose of preliminarily removing the three-dimensional points that are obviously not on the ground and the three-dimensional points with relatively large errors.

[0101] Point cloud screening strategies can add point cloud screening conditions and standards according to different usage scenarios. Here are several point cloud screening conditions, each of which has its own applicable situation. In actual use, it can be one or any combination of the following screening conditions:

[0102] (1) Remove 3D points within a certain range above the depth image according to the camera orientation.

[0103] For example, when the camera is close to horizontal, the 3D points in the upper half of the depth image will not appear on the ground. Therefore, 1 / 2 of the image can be used as a threshold to remove the 3D points converted from the pixels in the upper half of the depth image.

[0104] (2) When there is an initial estimate of the external parameter, the three-dimensional points in the camera coordinate system can be transferred to the world coordinate system through the camera external parameter, and the points whose height in the height direction is greater than the preset height threshold are removed.

[0105] (3) If the principle of the depth camera is to use binocular stereo vision, then due to the inherent characteristics of the binocular measurement principle, the depth measurement accuracy will decrease with the increase of distance. In other words, the depth accuracy of distant three-dimensional points will deteriorate. Therefore, a certain depth direction distance threshold (depth threshold) can be set to remove points that exceed the depth threshold.

[0106] (4) If the principle of the depth camera is to adopt the TOF method, then due to the characteristics of the TOF sensor itself, the camera is more susceptible to the influence of multipath reflection, and there may be more flying points in the depth image. In this case, the points whose distance difference is greater than the preset distance threshold can be removed based on the distance difference between the pixel point and its adjacent pixel points.

[0107] Step 203, obtaining a fitting plane based on the screened three-dimensional point cloud;

[0108] In this step, one implementation method is to use the Random Sample Consensus (RANSAC) algorithm to obtain the fitting plane. Specifically, it includes:

[0109] Step 2031, based on the screened 3D point cloud, random selection is performed to obtain a subset of randomly selected 3D points, wherein the number of 3D points is equal to the unknown number for determining the fitting plane. Since the fitting plane equation can be expressed as: Ax+By+Cz+D=0, it can be transformed into ax+by+cz=1.

[0110] Therefore, it includes three unknowns a, b, and c, so the number of three-dimensional points in the subset is at least 3.

[0111] Step 2032, based on the three-dimensional points in the subset, generate an estimate of the fitted plane of the subset: use the three-dimensional points in the selected subset to substitute into the plane equation, solve the unknowns in the fitted plane estimation equation, and obtain an estimate of the fitted plane;

[0112] If the number m of three-dimensional points in the selected subset is greater than the number of unknowns, an overdetermined system of equations needs to be solved. Regression analysis can be used to solve the unknowns in the fitting plane estimation equations. For example, a linear least squares method is used, as follows:

[0113] Transform the fitting plane equation to: ax+by+cz=1, then:

[0114]

[0115] When the matrix non-singular, then the unknowns a, b, c have a unique solution, where x m ,y m 、z m is the coordinate of a 3D point.

[0116] Step 2033, based on the estimation of the fitting plane, obtaining the degree of conformity of all the filtered three-dimensional points with respect to the estimation of the fitting plane;

[0117] One implementation method is to calculate the inlier rate of the estimated fitted plane and determine the degree of conformity based on the inlier rate, specifically:

[0118] Calculate the distance from each point in the filtered 3D point cloud to the estimated fitted plane.

[0119] The points whose calculated distance values ​​are less than the set distance threshold are regarded as internal points.

[0120] Count the number of inliers,

[0121] The ratio of the number of inliers to the filtered three-dimensional point cloud is calculated to obtain the inlier rate. The larger the ratio, the higher the inlier rate, which means the higher the degree of conformity and the better the estimation of the fitting plane.

[0122] Step 2034, determining whether the degree of conformity is satisfied, if yes, executing step 2035, otherwise, returning to step 2031, so as to randomly select a subset again to perform fitting plane estimation, thereby performing an estimation-confirmation cycle;

[0123] In this step, one implementation method is to determine whether the inlier rate meets the preset conditions.

[0124] In the second implementation mode, under the condition of confidence η, at least one random selection is made during the iterative cycle, so that the selected m points are all interior points, which is conducive to obtaining the best value of the fitted plane estimate at least once during the cycle. Therefore, the number of iterations i should meet the following conditions:

[0125]

[0126] Among them, m is the size of the subset, that is, the number of 3D points in the subset; the confidence level is generally set in the range of 0.95 to 0.99. ε is the inlier rate. In general, ε is usually unknown, so the proportion of inliers under the worst conditions can be taken, or it can be set to the proportion under the worst conditions in the initial state, and then continuously updated to the current maximum inlier rate as the number of iterations increases.

[0127] The third implementation method is to determine whether the probability that all subsets are inliers meets the required confidence level. Specifically, the selected subset is considered as a binomial distribution of the two results of "all are inliers" or "not all are inliers", and the probability of the former is p = 1 / ε m When p is small enough, it can be regarded as a Poisson distribution. Therefore, in i cycles, the probability that there are v "subsets are all interior points" can be expressed as:

[0128]

[0129] Among them, λ represents the expected number of selections of “all subsets are interior points” in i cycles.

[0130] For example, we hope that the probability that none of the selected subsets in this i-th iteration cycle is all internal points is less than a certain confidence level, that is, p(0, λ) = e -λ <1-η, taking the confidence level of 95% as an example, λ is approximately equal to 3, which means that under the confidence level of 95%, in i cycles, an average of 3 “good” subsets can be selected.

[0131] Step 2035, solving the fitting plane equation using the interior points of the best fitting plane estimate;

[0132] Specifically, based on the inlier points of the estimated fitting plane with the highest inlier rate, the estimated fitting plane is re-solved by the least squares method to obtain the final fitting plane equation.

[0133] Step 204 : Calculate the distance from the origin of the camera coordinate system to the fitting plane and the rotation of the camera based on the fitting plane parameters.

[0134] See also Figure 3 As shown, Figure 3 This is a schematic diagram of calibration using the fitting plane as the calibration plane. The symbol in the figure is represents the cross product of the vectors, and the fitting plane is the supporting ground of the mobile robot body in the world coordinate system. The distance from the origin of the camera coordinate system to the fitting plane is the height transformation of the camera, and the rotation of the camera coordinate system relative to the fitting plane corresponds to the rotation of the Y axis of the camera coordinate system relative to the normal vector of the fitting plane; therefore, according to the fitting plane equation Ax+By+Cz+D=0, we can get:

[0135] The normal vector of the fitted plane (ground) is n = (A, B, C);

[0136] The distance from the camera coordinate system to the fitting plane

[0137] Camera coordinate system rotation transformation

[0138] Here, the symbol × represents the cross product of vectors.

[0139] The above implementation can be expanded to obtain the equation of any calibration plane parallel to the fitting plane and at a set distance from the fitting plane according to the normal vector of the fitting plane. Thus, the distance transformation and rotation transformation of the camera coordinate system relative to the calibration plane can be obtained, thereby obtaining the parameters between the depth camera and the calibration plane.

[0140] If the depth image is related to a vertical surface (e.g., a wall) perpendicular to the supporting surface of the mobile robot body, for example, if a depth image of the wall is obtained, calibration can be performed based on the vertical surface. Figure 4 As shown, Figure 4This is a schematic diagram of calibration with the fitting plane as the vertical plane. In the figure, the normal vector n is opposite to the normal view direction of the camera coordinate system relative to the calibration plane, so it is -n. The distance from the origin of the camera coordinate system to the wall is the distance transformation of the camera. The rotation of the camera coordinate system to the normal view of the first calibration plane corresponds to the rotation of the z-axis of the camera coordinate system relative to the wall normal vector; therefore, according to the equation of the calibration plane A′x+B′y+C′z+D′=0, we can get:

[0141] Calibration plane normal vector - n = (A', B', C')

[0142] The distance from the camera coordinate system to the calibration plane

[0143] Camera coordinate system rotation transformation

[0144] Here, × represents the cross product of vectors.

[0145] The first calibration plane equation may be obtained through steps 201 - 203 based on the depth image.

[0146] The above implementation can be expanded to obtain the equation of any calibration plane parallel to the fitting plane and at a set distance from the fitting plane according to the normal vector of the fitting plane. Thus, the distance transformation and rotation transformation of the camera coordinate system relative to the calibration plane can be obtained, thereby obtaining the parameters between the depth camera and the calibration plane.

[0147] The depth camera extrinsic parameter calibration method proposed in the present application can be calibrated with the help of a ground plane according to different needs. For example, when the obtained depth image includes the image of the bearing surface of the mobile robot body, the calibration can be carried out with the help of a ground plane. When the obtained depth image includes the image data of a wall perpendicular to the bearing surface of the mobile robot body, the calibration can be carried out with the help of the wall without the help of any external tools. There are no requirements on the principle of the depth camera itself, and it can be applicable to the extrinsic parameter calibration of depth cameras based on binocular, structured light or TOF principles. The calibration method is simple to operate, has high calibration accuracy and strong robustness. Only one depth image is required for calibration, the calibration method has low computational complexity, and can be calibrated in real time. Through the calibration of the extrinsic parameters, the three-dimensional information in the camera coordinate system can be converted into the world coordinate system, and operations such as ground removal and three-dimensional obstacle perception can be easily performed, which has strong application value.

[0148] See also Figure 5 As shown, Figure 5A schematic diagram of data association in the calibration method of the embodiment of the present application. Based on the depth image data, the pixel points in the depth image are converted into spatial three-dimensional points in the camera coordinate system; in order to improve the accuracy and success rate of obtaining the fitting plane, the spatial three-dimensional points can be screened; when using the RANSAC algorithm, three-dimensional points are randomly selected as a subset, and the fitting plane estimate of the subset is obtained based on the subset. Through the multiple randomly selected subsets of the algorithm and the fitting plane estimates of each subset, the best fitting plane estimate is selected according to the condition of the maximum inlier rate; based on the inliers of the best fitting plane, the fitting plane is recalculated; if the fitting plane is used as the calibration plane, the height transformation and rotation transformation of the camera coordinate system relative to the fitting plane are calculated according to the fitting plane.

[0149] See also Figure 6 As shown, Figure 6 This is a schematic diagram of a calibration device according to an embodiment of the present application. The device includes:

[0150] A depth image acquisition module is used to acquire depth image data, wherein the depth image data includes pixel coordinates and depth values;

[0151] A conversion module, based on the acquired depth image data, converts the pixel points in the depth image into spatial three-dimensional points in the camera coordinate system;

[0152] A fitting plane acquisition module, which acquires a fitting plane of the three-dimensional points based on the three-dimensional points;

[0153] The transformation calculation module obtains the parameters between the depth camera and the calibration plane according to the current posture relationship between the calibration plane parallel to or coincident with the fitting plane and the camera coordinate system;

[0154] The calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

[0155] The device also includes a screening module, which screens the three-dimensional points in the three-dimensional point set according to the screening strategy to obtain a screened three-dimensional point set.

[0156] The fitting plane acquisition module includes:

[0157] A random selection module, performing random selection based on the screened three-dimensional point set to obtain a current subset consisting of the randomly selected three-dimensional points, wherein the number of three-dimensional points in the subset includes at least 3;

[0158] A fitting plane estimation module obtains a fitting plane estimation of the subset based on the three-dimensional points in the current subset;

[0159] A conformity acquisition module, which acquires the conformity of all three-dimensional points in the filtered three-dimensional point set relative to the fitted plane estimate according to the acquired fitted plane estimate;

[0160] If the degree of compliance is insufficient, return to the random selection module;

[0161] If the compliance level is reached, return to the re-decomposition module;

[0162] The inner points include three-dimensional points in the screened three-dimensional point set whose distance from the three-dimensional point to the fitting plane with the best conformity is less than a preset distance threshold.

[0163] The re-solution module solves the equation of the fitted plane using the interior points estimated by the best fitting plane.

[0164] The fitting plane estimation module further includes, if the number of three-dimensional points in the current subset is equal to 3, substituting the coordinate values ​​of the three-dimensional points into the fitting plane equation, solving the unknown number in the fitting plane equation, and obtaining the fitting plane estimation of the current subset;

[0165] If the number of 3D points in the current subset is greater than 3, the coordinate values ​​of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset.

[0166] The conformity degree acquisition module further includes calculating the distance value from each three-dimensional point in the screened three-dimensional point set to the estimated fitting plane.

[0167] The 3D points whose calculated distance values ​​are less than the set distance threshold are regarded as internal points.

[0168] Count the number of inliers,

[0169] Calculate the ratio of the number of inliers to the filtered three-dimensional point cloud to obtain the inlier rate;

[0170] Determining the degree of compliance according to an inlier rate;

[0171] or,

[0172] The degree of compliance is determined according to the number of iterations, wherein the number of iterations satisfies:

[0173]

[0174] Where m is the number of 3D points in the subset, η is the set confidence level, ε is the inlier rate, which is the ratio of inliers under the worst condition, or is set to the ratio under the worst condition in the initial state, and is updated to the current maximum inlier rate as the number of iterations increases;

[0175] or,

[0176] The degree of conformity is determined according to whether the probability that all v subsets are inliers meets the set confidence level, wherein the probability that all v subsets are inliers is:

[0177]

[0178] Where λ is the expected number of selections in which all subsets are inliers in the current iteration.

[0179] The re-solution module also includes re-solving unknowns in the fitting plane equation by a least square method according to the estimated inliers of the fitting plane with the highest inliers rate, so as to obtain the fitting plane equation.

[0180] The transformation calculation module includes obtaining a normal vector of a fitting plane according to a fitting plane equation,

[0181] Obtaining an equation of a calibration plane that is parallel to or coincident with the fitting plane according to the normal vector of the fitting plane;

[0182] Calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane;

[0183] Calculate the rotation of the camera coordinate system relative to the calibration plane, and the rotation transformation of the camera coordinate system relative to the calibration plane.

[0184] The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

[0185] The depth image data includes image data of the mobile robot body bearing surface, and the transformation calculation module also includes:

[0186] According to the normal vector of the fitting plane, an equation of a calibration plane which is parallel to the fitting plane and has a set distance is obtained; wherein the calibration plane is the supporting ground of the mobile robot body in the world coordinate system;

[0187] According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the fitting plane is calculated to obtain the height transformation of the camera coordinate system relative to the calibration plane; the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

[0188] The height transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

[0189] The depth image data includes elevation image data perpendicular to the supporting surface of the mobile robot body, and the transformation calculation module further includes, according to the normal vector of the fitting plane, obtaining an equation of a calibration plane parallel to the fitting plane and having a set distance; wherein the calibration plane is perpendicular to the supporting surface of the mobile robot body in the world coordinate system;

[0190] According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the second fitting plane is calculated to obtain the distance transformation of the camera coordinate system relative to the calibration plane; the rotation of the z-axis of the camera coordinate system relative to the normal vector of the second fitting plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

[0191] The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

[0192] The present application also provides an electronic device for calibrating the external parameters of a depth camera, the electronic device comprising a memory and a processor, wherein:

[0193] The memory is used to store computer programs;

[0194] The processor is used to execute the program stored in the memory to implement the calibration step of any depth camera extrinsic parameter.

[0195] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0196] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0197] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0198] Acquire depth image data, wherein the depth image data includes pixel coordinates and depth values;

[0199] Based on the acquired depth image data, pixel points in the depth image are converted into spatial three-dimensional points in the camera coordinate system;

[0200] Based on the three-dimensional points, obtaining a fitting plane associated with the calibration plane;

[0201] According to the current pose relationship between the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained;

[0202] The calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

[0203] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0204] It should be noted that the embodiments of the depth camera extrinsic calibration method provided in the present application may not be limited to the above-mentioned implementation. For example, when obtaining the fitting plane, other fitting algorithms besides the RANSAC algorithm may be used. When solving the overdetermined equation, a solution method besides the least squares method may also be used, for example, regression calculation, etc. In addition, in order to reduce the computational complexity of the overdetermined equation, the accuracy of the overdetermined equation may be improved by selectively selecting reliable three-dimensional points, for example, with the aid of feature points of the image.

[0205] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for calibrating the extrinsic parameters of a depth camera, characterized in that: The method includes, Acquire depth image data, wherein the depth image data includes pixel coordinates and depth values; Based on the acquired depth image data, pixel points in the depth image are converted into spatial three-dimensional points in the camera coordinate system; According to the screening strategy, the spatial three-dimensional points transformed into the camera coordinate system are screened to obtain a screened three-dimensional point set; Based on the screened three-dimensional point set, a random selection is performed to obtain a current subset consisting of the randomly selected three-dimensional points, wherein the number of three-dimensional points in the subset includes at least 3; Based on the 3D points in the current subset, obtain a fitted plane estimate for the subset; According to the obtained fitting plane estimate, obtaining the conformity degree of all three-dimensional points in the filtered three-dimensional point set with respect to the fitting plane estimate; If the degree of conformity is insufficient, returning to the step of performing random selection based on the screened three-dimensional point set; If the degree of conformity is reached, the fitting plane equation is solved using the inliers estimated by the fitting plane with the best degree of conformity; wherein the inliers include the three-dimensional points in the screened three-dimensional point set whose distance to the fitting plane with the best degree of conformity is less than a preset distance threshold, and the degree of conformity is determined according to whether the probability that all the current subsets are inliers meets the set confidence level; According to the current pose relationship between the calibration plane parallel to or coincident with the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained, wherein the calibration plane includes any current plane parallel to or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

2. The method according to claim 1, characterized in that The depth image data includes at least one depth image, The step of converting the pixel points in the depth image into three-dimensional points in the camera coordinate system includes: According to the mapping geometric model between the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system, the three-dimensional point coordinates corresponding to the pixel point are obtained.

3. The method according to claim 2, characterized in that The method of obtaining the three-dimensional point coordinates corresponding to the pixel point according to the mapping geometric model between the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system includes: For any pixel in the depth image, The depth value of the pixel point is used as the z coordinate value of the three-dimensional point; Obtain a first difference between the x coordinate of the pixel point and the x-direction offset, multiply the ratio of the z coordinate value to the x-direction focal length in the camera intrinsic parameter by the first difference, and use the result as the x coordinate value of the three-dimensional point; the x-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the x direction; Obtain a second difference between the y coordinate of the pixel point and the y direction offset, multiply the ratio of the z coordinate value to the y direction focal length in the camera intrinsic parameter by the second difference, and obtain the result as the y coordinate value of the three-dimensional point; the y direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y direction; Among them, the x-direction offset of the origin of the camera coordinate system relative to the image coordinate system and the y-direction offset of the origin of the camera coordinate system relative to the image coordinate system are obtained according to the camera intrinsic parameters; Convert all pixels in the depth image into three-dimensional point coordinates to obtain a three-dimensional point set.

4. The method according to claim 3, characterized in that The screening strategy includes any one of the following conditions or any combination thereof: (1) Remove the 3D points within a set range above the depth image according to the camera orientation; (2) When there is an initial estimate of the external parameter, the 3D points in the camera coordinate system are transformed into the world coordinate system through the camera external parameter, and the 3D points whose height in the height direction is greater than the preset height threshold are removed; (3) For a binocular stereo depth camera, remove three-dimensional points whose depth values ​​are greater than a preset depth threshold; (4) For a time-of-flight depth camera, based on the distance difference between a pixel and its adjacent pixels, remove the three-dimensional points whose distance difference is greater than a preset distance threshold.

5. The method according to claim 1, characterized in that The step of obtaining a fitted plane estimate of the subset based on the three-dimensional points in the current subset includes: If the number of 3D points in the current subset is equal to 3, substitute the coordinate values ​​of the 3D points into the fitting plane equation, solve the unknowns in the fitting plane equation, and obtain the fitting plane estimate of the current subset; If the number of 3D points in the current subset is greater than 3, the coordinate values ​​of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset.

6. The method according to claim 1, characterized in that The method of solving the fitting plane equation by using the interior points estimated by the fitting plane with the best conformity degree comprises: The fitting plane estimate with the highest inlier rate is taken as the best fitting plane estimate. According to the inlier points of the best fitting plane estimate, the unknowns in the fitting plane equation are re-solved by the least squares method to obtain the fitting plane equation.

7. The method according to claim 1, characterized in that The probability that the current subset is all inliers is determined as follows: Among them, λ is the expected number of selections in which all the current subsets are interior points in the current iteration, is the number of subsets formed by randomly selected 3D points.

8. The method according to any one of claims 1 to 7, characterized in that: The method of obtaining the parameters between the depth camera and the calibration plane according to the current pose relationship between the calibration plane parallel to or coincident with the fitting plane and the camera coordinate system includes: According to the fitting plane equation, the fitting plane normal vector is obtained. Obtaining an equation of a calibration plane that is parallel to or coincident with the fitting plane according to the normal vector of the fitting plane; Calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane; Calculate the rotation of the camera coordinate system relative to the calibration plane, and the rotation transformation of the camera coordinate system relative to the calibration plane. The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

9. The method according to claim 8, characterized in that The depth image data includes image data of the mobile robot body bearing surface. The step of obtaining an equation of a calibration plane associated with the fitting plane according to the fitting plane normal vector comprises: According to the normal vector of the fitting plane, an equation of a calibration plane which is parallel to the fitting plane and has a set distance is obtained; wherein the calibration plane is the supporting ground of the mobile robot body in the world coordinate system; The calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane includes: According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the calibration plane is calculated to obtain the height transformation of the camera coordinate system relative to the calibration plane; The calculation of the rotation of the camera coordinate system relative to the calibration plane to the rotation transformation of the camera coordinate system relative to the calibration plane includes: According to the equation parameters of the calibration plane, the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

10. The method according to claim 8, characterized in that The depth image data includes elevation image data perpendicular to the bearing surface of the mobile robot body. The step of obtaining an equation of a calibration plane associated with the fitting plane according to the fitting plane normal vector comprises: According to the normal vector of the fitting plane, an equation of a calibration plane which is parallel to the fitting plane and has a set distance is obtained; wherein the calibration plane is perpendicular to the supporting ground of the mobile robot body in the world coordinate system; The calculating the distance from the origin of the camera coordinate system to the calibration plane to obtain the distance transformation of the camera coordinate system relative to the calibration plane includes: According to the equation parameters of the calibration plane, the distance from the origin of the camera coordinate system to the calibration plane is calculated to obtain the distance transformation from the camera coordinate system to the calibration plane; The calculation of the rotation of the camera coordinate system relative to the calibration plane to the rotation transformation of the camera coordinate system relative to the calibration plane includes: According to the equation parameters of the calibration plane, the rotation of the z-axis of the camera coordinate system relative to the normal vector of the calibration plane is calculated to obtain the rotation transformation of the camera coordinate system relative to the calibration plane.

11. An electronic device for calibrating external parameters of a depth camera, characterized in that: The electronic device includes a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the program stored in the memory to implement the depth camera extrinsic parameter calibration method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is executed by a processor to implement the method for calibrating the extrinsic parameters of a depth camera according to any one of claims 1 to 10.

Images