Depth sensor calibration method, electronic device and system

Abstract

The invention provides a depth sensor calibration method, electronic equipment and a system. The method comprises the following steps: receiving a jig plate image and a calibration plate image; determining a plurality of first mark point coordinates in the calibration plate image and a plurality of second mark point coordinates in the jig plate image; fitting a calibration plate plane and determining a first center coordinate of the calibration plate; fitting the plane of the jig plate and a second center coordinate of the jig plate; calibrating the plane parallelism of the depth sensor according to the calibration plate plane and the calibration plate plane, and calibrating the center of the depth sensor according to the first center coordinate and the second center coordinate; thus, the binocular camera can obtain three-dimensional point cloud data of the jig plate and the calibration plate through stereoscopic vision, complete postures of the two fitting planes are directly solved, global three-dimensional posture comparison is achieved, and the calibration precision of the plane parallelism is improved; the offset of the depth sensor relative to the center of the calibration plate can be accurately calculated, and the calibration precision of the center of the depth sensor is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical measurement and automatic calibration, and in particular to a depth sensor calibration method, electronic device and system. BACKGROUND

[0002] A direct time of flight sensor (dToF) is a depth sensor that calculates the distance between the sensor and a target object by measuring the time difference of a light signal from emission to reflection by an object and return to the receiver. The measurement accuracy of the depth sensor directly depends on the relative position and attitude of its own optical system and the target object (or calibration plate), so plane parallelism calibration and centering calibration are necessary before the depth sensor is put into use.

[0003] In the prior art, laser ranging is generally used to calibrate the plane parallelism and centering of the depth sensor, but the laser ranging method requires human intervention (such as manual collection of 4-6 discrete points), takes a long time to calibrate and introduces human error, affecting the calibration accuracy of the depth sensor. SUMMARY

[0004] To solve or partially solve the technical problem that the calibration accuracy cannot be ensured when calibrating a dTOF sensor in the prior art, embodiments of the present application provide a depth sensor calibration method, electronic device and system.

[0005] In a first aspect of the present application, a depth sensor calibration method is provided, which comprises:

[0006] Receiving a jig plate image and a calibration plate image collected by a binocular camera; the jig plate and the depth sensor are loaded on the same bearing panel of a test jig in the shooting direction of the binocular camera; the calibration plate is located behind the jig plate and has a preset distance from the jig plate;

[0007] Determining a plurality of first marker point coordinates in the calibration plate image using a hybrid feature detection algorithm and determining a plurality of second marker point coordinates in the jig plate image using a circle detection algorithm;

[0008] Fitting a calibration plate plane according to the plurality of first marker point coordinates and determining a first center coordinate of the calibration plate; fitting a jig plate plane according to the plurality of second marker points and determining a second center coordinate of the jig plate;

[0009] Calibrating the plane parallelism of the depth sensor according to the calibration plate plane and the jig plate plane, and calibrating the center of the depth sensor according to the first center coordinate and the second center coordinate.

[0010] In the above scheme, the calibration board is a checkerboard background board, and the step of determining the plurality of first marker point coordinates in the calibration board image by using a hybrid feature detection algorithm comprises:

[0011] Determining the gray scale gradient of each pixel in the calibration board image in the X-axis direction and the gray scale gradient in the Y direction;

[0012] According to the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction, a corresponding second-order matrix is determined, and the second-order matrix is smoothed to obtain a second-order smoothed matrix;

[0013] According to the second-order smoothed matrix, a response value is determined, and the response value is normalized to obtain a normalized response value;

[0014] A plurality of target normalized response values are obtained by screening a plurality of normalized response values;

[0015] A plurality of first candidate corner points are obtained by performing non-maximum suppression processing on the plurality of target normalized response values;

[0016] The plurality of candidate corner points are subjected to secondary screening by using boundary constraint conditions and minimum spacing constraint conditions to obtain a plurality of second candidate corner points;

[0017] Sub-pixel level coordinate calculation is performed on the plurality of second candidate corner points to obtain sub-pixel level coordinates of the plurality of second candidate corner points;

[0018] The plurality of second candidate corner points are sorted in descending order of response value, and the second candidate corner points corresponding to the first N response values are selected, and the sub-pixel level coordinates of the first N second candidate corner points are determined as the plurality of first marker point coordinates.

[0019] In the above scheme, a plurality of circular holes with consistent diameters are arranged on the jig board, and the step of determining the plurality of second marker point coordinates in the jig board image by using a circle detection algorithm comprises:

[0020] The jig board image is subjected to gray scale processing to obtain a gray scale image;

[0021] The gray scale image is subjected to edge extraction to obtain a plurality of edge contours;

[0022] A space accumulator (x0, y0, r) is established, the accumulator is used to record the number of votes for each three-dimensional array (x0, y0, r), and the (x0, y0) is a candidate center, and the r is any radius within a predetermined radius range;

[0023] Traverse all edge pixels on each edge contour, judge whether the edge pixel and the candidate center satisfy the voting condition, if yes, accumulate the space accumulator;

[0024] The three-dimensional array greater than the accumulator threshold is reserved, and a de-duplication operation is performed on the reserved three-dimensional array to obtain a plurality of center coordinates; the plurality of center coordinates are the plurality of second marker point coordinates.

[0025] In the above scheme, the fitting of the calibration board plane according to the plurality of first marker point coordinates comprises:

[0026] The plurality of first marker points are converted into initial three-dimensional point cloud coordinates by using the parameters of the binocular camera, and the initial three-dimensional point cloud coordinates are subjected to outlier suppression to obtain target three-dimensional point cloud coordinates;

[0027] The initial parameters of the calibration board plane equation are determined by using the target three-dimensional point cloud coordinates;

[0028] On the basis of the initial parameters, the calibration board plane equation is subjected to iterative reweighted least squares processing;

[0029] When the iteration termination condition is met, the target parameters of the calibration board plane equation are output, and the plane equation of the fitted calibration board plane is determined based on the target parameters.

[0030] In the above scheme, the calibration of the planar parallelism of the depth sensor according to the jig board plane and the calibration board plane comprises:

[0031] A first plane equation of the calibration board plane and a second plane equation of the jig board plane are obtained;

[0032] The offset angle between the jig board plane and the calibration board plane is determined according to the first plane equation of the calibration board plane and the second plane equation of the jig board plane;

[0033] The planar parallelism of the depth sensor is calibrated by using the offset angle.

[0034] In the above scheme, the calibration of the center of the depth sensor according to the first center coordinate and the second center coordinate comprises:

[0035] A first position offset between the jig board center and the calibration board center is determined according to the first center coordinate and the second center coordinate;

[0036] A second position offset between the jig board center and the depth sensor center is determined;

[0037] determine a target position offset of the depth sensor center relative to the calibration plate center according to the first position offset and the second position offset;

[0038] calibrate the center of the depth sensor by using the target position offset.

[0039] In the above scheme, the second position offset between the jig plate center and the depth sensor center is determined by:

[0040] determine a horizontal offset x_offset of the depth sensor center relative to the jig plate center according to the formula x_offset=AO-D1;

[0041] determine a vertical offset y_offset of the depth sensor center relative to the jig plate center according to the formula y_offset=BO-D2; wherein,

[0042] The AO is a theoretical distance between an edge of the bearing panel and the center of the depth sensor in the horizontal direction, the BO is a theoretical distance between another edge of the bearing panel and the center of the depth sensor in the vertical direction, the D1 is an actual distance between the edge of the bearing panel and the center of the depth sensor in the horizontal direction, and the D2 is an actual distance between the other edge of the bearing panel and the center of the depth sensor in the vertical direction.

[0043] In a second aspect of the present application, a depth sensor calibration electronic device is provided, which comprises:

[0044] An acquisition unit is configured to receive jig plate images and calibration plate images collected by a binocular camera; the jig plate and the depth sensor are relatively loaded on the same bearing panel of a test jig along a shooting direction of the binocular camera; and the calibration plate is located behind the jig plate and has a preset distance from the jig plate.

[0045] A first determination unit is configured to determine a plurality of first marker point coordinates in the calibration plate image by using a hybrid feature detection algorithm and determine a plurality of second marker point coordinates in the jig plate image by using a circle detection algorithm.

[0046] A second determination unit is configured to respectively fit a calibration plate plane according to the plurality of first marker point coordinates and determine a first center coordinate of the calibration plate; and respectively fit a jig plate plane according to the plurality of second marker points and determine a second center coordinate of the jig plate.

[0047] The calibration unit is configured to calibrate the depth sensor according to the planar parallelism of the calibration board plane and the calibration board plane, and calibrate the center of the depth sensor according to the first center coordinates and the second center coordinates.

[0048] In the above scheme, the first determination unit is specifically configured to:

[0049] Determine the gray scale gradient of each pixel in the jig board image in the X-axis direction and the gray scale gradient in the Y direction;

[0050] According to the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction, a corresponding second-order matrix is determined, and the second-order matrix is smoothed to obtain a second-order smoothed matrix;

[0051] According to the second-order smoothed matrix, a response value is determined, and the response value is normalized to obtain a normalized response value;

[0052] A plurality of normalized response values are screened to obtain a plurality of target normalized response values;

[0053] The plurality of target normalized response values are subjected to non-maximum suppression processing to obtain a plurality of first candidate corner points;

[0054] The plurality of candidate corner points are subjected to secondary screening using boundary constraint conditions and minimum spacing constraint conditions to obtain a plurality of second candidate corner points;

[0055] The plurality of second candidate corner points are subjected to sub-pixel level coordinate calculation using quadratic function fitting to obtain sub-pixel level coordinates of the plurality of second candidate corner points;

[0056] The plurality of second candidate corner points are sorted in descending order of response value, and the first N response values corresponding to the second candidate corner points are selected, and the sub-pixel level coordinates of the first N second candidate corner points are determined as the plurality of first marker point coordinates.

[0057] In a third aspect of the present application, a calibration system for a depth sensor is provided, which comprises a binocular camera, a test jig, a jig board, a calibration board, and the calibration electronic device of the second aspect.

[0058] Along the shooting direction of the binocular camera, the jig board and the depth sensor are loaded on the same bearing panel of the test jig; the calibration board is located behind the jig board and has a preset distance from the jig board.

[0059] The calibration electronic device is specifically configured to: acquire a jig board image and a calibration board image collected by the binocular camera, determine a plurality of first marker point coordinates in the calibration board image by using a hybrid feature detection algorithm, and determine a plurality of second marker point coordinates in the jig board image by using a circle detection algorithm; fit a calibration board plane according to the plurality of first marker point coordinates respectively, and determine a first center coordinate of the calibration board; fit the jig board plane according to the plurality of second marker points respectively, and determine a second center coordinate of the jig board; calibrate a plane parallelism of the depth sensor according to the calibration board plane, and calibrate a center of the depth sensor according to the first center coordinate and the second center coordinate.

[0060] The application provides a depth sensor calibration method, an electronic device and a system. The method comprises the following steps: receiving a jig board image and a calibration board image collected by a binocular camera; the jig board and the depth sensor are loaded on the same bearing panel of a test jig in a relative manner along a shooting direction of the binocular camera; the calibration board is located behind the jig board and has a preset distance from the jig board; a plurality of first marker point coordinates in the calibration board image are determined by using a hybrid feature detection algorithm, and a plurality of second marker point coordinates in the jig board image are determined by using a circle detection algorithm; a calibration board plane is fitted according to the plurality of first marker point coordinates respectively, and a first center coordinate of the calibration board is determined; the jig board plane is fitted according to the plurality of second marker points respectively, and a second center coordinate of the jig board is determined; a plane parallelism of the depth sensor is calibrated according to the calibration board plane, and a center of the depth sensor is calibrated according to the first center coordinate and the second center coordinate. In this way, the binocular camera can acquire three-dimensional point cloud data of the jig board and the calibration board through stereo vision, directly solve the complete attitude of the two fitted planes, realize global three-dimensional attitude comparison, and thus improve the calibration accuracy of the plane parallelism. Similarly, when the center of the depth sensor is calibrated, the first center coordinate and the second center coordinate determined by the binocular camera through the parallax principle are also three-dimensional, so that the offset of the depth sensor relative to the center of the calibration board can be accurately calculated, thereby improving the calibration accuracy of the center of the depth sensor. BRIEF DESCRIPTION OF DRAWINGS

[0061] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings.

[0062] Figure 1 A calibration system structure schematic diagram of a depth sensor according to an embodiment of the application is shown;

[0063] Figure 2 A flowchart of a calibration method of a depth sensor according to an embodiment of the present application is shown;

[0064] Figure 3 A structural diagram of a calibration electronic device of a depth sensor according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0066] In order to better understand the technical solutions of the present application, a calibration system of a depth sensor is first introduced as follows. Figure 1 As shown, the system comprises a binocular camera 1, a test fixture 2, a fixture plate 3, a calibration plate 4, and an electronic device (not shown in the figure).

[0067] Along the shooting direction of the binocular camera 1, the fixture plate 3 and the depth sensor 5 are loaded on the same bearing panel of the test fixture 2, and the calibration plate 4 is located behind the fixture plate 3 and has a preset distance from the fixture plate 3. The calibration plate 4 can be a checkerboard background plate, and the fixture plate 3 can be a plate with multiple circular holes of the same size and arranged at equal intervals on the plate.

[0068] When the fixture plate 3 and the depth sensor 5 are loaded on the bearing panel of the test fixture 2, the fixture plate 3 and the depth sensor 5 are coaxial. The test fixture 2 can be a six-axis platform that can control the bearing panel to move, rotate, etc. along different degrees of freedom.

[0069] During calibration, the binocular camera 1 is used to simultaneously shoot the fixture plate 3 and the calibration plate 4 to obtain fixture plate images and calibration plate images, and transmit the fixture plate images and the calibration plate images to the electronic device.

[0070] The calibration electronic device is specifically used for: acquiring a jig plate image and a calibration plate image collected by the binocular camera 1, determining a plurality of first marker point coordinates in the calibration plate image by using a hybrid feature detection algorithm and determining a plurality of second marker point coordinates in the jig plate image by using a circle detection algorithm; fitting a calibration plate plane according to the plurality of first marker point coordinates and determining a first center coordinate of the calibration plate; fitting the jig plate plane according to the plurality of second marker points and determining a second center coordinate of the jig plate; calibrating a plane parallelism of the depth sensor according to the calibration plate plane and the plane parallelism of the depth sensor, and calibrating a center of the depth sensor according to the first center coordinate and the second center coordinate.

[0071] In actual applications, the calibration electronic device can be an industrial computer, a computer or the like.

[0072] The specific implementation method of the calibration electronic device for calibrating the plane parallelism of the depth sensor and calibrating the center of the depth sensor can refer to the specific implementation of the subsequent calibration electronic device side, and thus will not be described here.

[0073] Based on the same inventive concept as the foregoing embodiments, the present application provides a calibration method of a depth sensor, applied to a calibration electronic device side, as shown in the accompanying drawings, the method comprising the following steps: Figure 2 The method comprises the following steps:

[0074] S210, receiving a jig plate image and a calibration plate image collected by a binocular camera; along a shooting direction of the binocular camera, a jig plate and a depth sensor are relatively loaded on a same bearing panel of a test jig; a calibration plate is located behind the jig plate and has a preset distance from the jig plate.

[0075] As described above, along the shooting direction of the binocular camera, the jig plate and the depth sensor are relatively loaded on the same bearing panel of the test jig; the calibration plate is located behind the jig plate and has the preset distance from the jig plate. The calibration plate and the jig plate both face the binocular camera, and after the binocular camera photographs the calibration plate and the jig plate, the jig plate image and the calibration plate image are obtained. The jig plate image and the calibration plate image are transmitted to the calibration electronic device, so that the calibration electronic device receives the jig plate image and the calibration plate image.

[0076] S211, determining a plurality of first marker point coordinates in the calibration plate image by using a hybrid feature detection algorithm and determining a plurality of second marker point coordinates in the jig plate image by using a circle detection algorithm.

[0077] The present application needs to fit the jig plate plane and the calibration plate plane, so firstly needs to determine a plurality of first mark point coordinates in the calibration plate image by using a hybrid feature detection algorithm and determine a plurality of second mark point coordinates in the jig plate image by using a circle detection algorithm. The first mark point is an angle point, and the second mark point is the center of a circular hole.

[0078] In an embodiment, the hybrid feature detection algorithm is used to determine a plurality of first mark point coordinates in the calibration plate image, comprising:

[0079] Determine the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction in the calibration plate image;

[0080] According to the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction, determine the corresponding second order matrix, and smooth the second order matrix to obtain the second order smooth matrix;

[0081] According to the second order smooth matrix, determine the response value, and perform normalization processing on the response value to obtain the normalized response value;

[0082] Screen a plurality of normalized response values to obtain a plurality of target normalized response values;

[0083] Perform non-maximum suppression processing on a plurality of target normalized response values to obtain a plurality of first candidate angle points;

[0084] Use the boundary constraint condition and the minimum spacing constraint condition to perform secondary screening on a plurality of candidate angle points to obtain a plurality of second candidate angle points;

[0085] Perform sub-pixel level coordinate calculation on a plurality of second candidate angle points to obtain sub-pixel level coordinates of the plurality of second candidate angle points;

[0086] Sort a plurality of second candidate angle points in descending order of response value, select the second candidate angle points corresponding to the first N response values, and determine the sub-pixel level coordinates of the first N second candidate angle points as a plurality of first mark point coordinates.

[0087] Specifically, since the gradient can reflect the change rate of the pixel gray value, the angle point region exists significant gradient change in multiple directions, so the Sobel operator can be used to calculate the gray scale gradient I x of each pixel in the X direction (horizontal) of the calibration plate image and the gray scale gradient I y in the Y direction (vertical direction).

[0088] For each pixel, determine the second order matrix based on the gray scale gradient, and smooth the second order matrix to obtain the second order smooth matrix

[0089] According to formula (1), calculate the response value S:

[0090] S = det(M) - k x (trace(M)) 2 (1)

[0091] wherein k is an empirical constant, det(M) is a matrix determinant, and trace(M) is a matrix trace.

[0092]

[0093] All pixels are processed in the same way, and multiple response values are obtained, which are normalized to obtain multiple normalized response values.

[0094] The multiple normalized response values are screened according to a preset screening condition, and the normalized response values greater than c x max(S) are retained, and the retained normalized response values are target normalized response values. Wherein c is a proportionality coefficient, and max(S) is the maximum normalized response value.

[0095] In order to further improve the accuracy of corner positioning and avoid errors caused by redundant points, a non-maximum suppression algorithm is also needed to perform non-maximum suppression processing on the multiple target normalized response values to obtain multiple first candidate corner points. Since the edge region of the image is easily affected by imaging distortion, the feature reliability is low, so it is necessary to eliminate the candidate corner points close to the edge of the calibration plate image, and set a minimum pixel distance threshold (such as 5-10px) between the corner points, and eliminate the dense candidate points with a spacing less than the threshold, to avoid the corner points being too concentrated, affecting the stability of subsequent matching or three-dimensional reconstruction. After the elimination is completed, multiple second candidate corner points are obtained.

[0096] Sub-pixel level coordinate calculation is performed on the multiple second candidate corner points by a quadratic function fitting or interpolation method, so that when the pixel level corner point coordinate is an integer (100, 200), the actual physical center of the corner point may be located inside the pixel. By fitting a quadratic curve with the multiple second candidate corners, the peak point of the fitted curve is found, and the sub-pixel level coordinate (100.3, 199.8) is obtained. In this way, the sub-pixel level coordinates corresponding to the multiple second candidate corner points are obtained.

[0097] Then, the multiple second candidate corner points are sorted in descending order of response value, and the second candidate corner points corresponding to the first N response values are selected, and the sub-pixel level coordinates of the first N (such as the first 100) second candidate corner points are determined as the first marker point coordinates.

[0098] In one embodiment, a plurality of circular holes with consistent diameters are arranged on the jig plate; and a plurality of second marker point coordinates in the jig plate image are determined by using a circle detection algorithm, comprising:

[0099] The jig plate image is subjected to grayscale processing to obtain a grayscale image;

[0100] Edge extraction is performed on the gray image to obtain a plurality of edge contours;

[0101] A space accumulator (x0, y0, r) is established, and the accumulator is used to record the number of votes for each three-dimensional array (x0, y0, r); (x0, y0) is a candidate center, and r is any radius within a preset radius range;

[0102] All edge pixels on each edge contour are traversed, and it is judged whether the edge pixel and the candidate center satisfy the voting condition, if yes, the space accumulator is accumulated;

[0103] The three-dimensional array greater than the accumulator threshold is retained, and the retained three-dimensional array is sequentially subjected to a de-duplication operation and a radius consistency verification operation to obtain a plurality of center coordinates; the plurality of center coordinates are a plurality of second marker point coordinates.

[0104] Specifically, first, the jig plate image needs to be converted into a gray image, and a Gaussian filter (such as a 5*5 Gaussian kernel) is used to smooth the gray image to suppress noise interference while retaining the gray mutation characteristics of the hole edge (the hole edge appears as a clear light-dark boundary in the gray image).

[0105] The edges of the gray image are extracted using the Canny edge detection algorithm, including: obtaining continuous hole contour edges by double thresholding (high thresholding to screen strong edges, and low thresholding to connect weak edges), to ensure the integrity of the edges.

[0106] The circle detection algorithm of the application is a Hough circle detection algorithm, and the core is to count possible centers and radii through an accumulator. First, the radius range constraint and the accumulator threshold constraint can be determined.

[0107] For the radius range, since the hole sizes are consistent, a fixed radius range (minr1, maxr1) can be set according to the actual size of the hole (or the pixel size in the image), wherein minr = r1-2 and maxr = r1+2; r1 is the hole radius.

[0108] For the accumulator threshold, the accumulator threshold includes two: param1 and param2; param1 can be slightly lower than the high threshold of the Canny edge detection to ensure that the gradient of the hole edge is effectively identified; param2 can be adjusted according to the number of holes and image noise, and is usually set to 30-50 (the higher the value, the more reliable the detected circle, and false positives can be filtered).

[0109] In the Hough circle detection, a space accumulator (x0, y0, r) is established, which is used to record the number of votes for each three-dimensional array (x0, y0, r); (x0, y0) is a candidate center, and r is any radius within a preset radius range; the initial value of the space accumulator is 0.

[0110] For each edge pixel point on each edge contour, all possible radii within the above radius range are traversed, and if the circle equation (x-x0) 2 +(y-y0) 2 =r is satisfied, (x0, y0, r) in the accumulator is voted. Peak detection is performed on the accumulator, and (x0, y0, r) combinations with a number of votes exceeding param2 are retained, which are the most likely circle hole parameters.

[0111] The retained three-dimensional arrays are de-duplicated to obtain multiple three-dimensional arrays, and then multiple center coordinates are obtained. The multiple center coordinates are determined as multiple second marker point coordinates.

[0112] S212, respectively fitting the calibration board plane according to the multiple first marker point coordinates and determining the first center coordinates of the calibration board; respectively fitting the jig board plane according to the multiple second marker points and determining the second center coordinates of the jig board.

[0113] After the multiple first marker point coordinates and the multiple second marker point coordinates are determined, the calibration board plane can be fitted according to the multiple first marker point coordinates, and the first center coordinates of the calibration board can be determined; the jig board plane can be fitted according to the multiple second marker points, and the second center coordinates of the jig board can be determined.

[0114] In an embodiment, fitting the calibration board plane according to the multiple first marker point coordinates comprises:

[0115] Converting the multiple first marker points into initial three-dimensional point cloud coordinates by using the parameters of the binocular camera, and performing outlier suppression on the initial three-dimensional point cloud coordinates to obtain target three-dimensional point cloud coordinates;

[0116] Determining initial parameters of the calibration board plane equation by using the target three-dimensional point cloud coordinates;

[0117] On the basis of the initial parameters, iteratively solving the calibration board plane equation by re-weighted least squares in order to minimize the weighted residual sum of squares;

[0118] When the iteration termination condition is met, outputting the target parameters of the calibration board plane equation, and determining the plane equation of the fitted calibration board plane based on the target parameters.

[0119] Specifically, the first mark point coordinates, in order to improve the accuracy of plane fitting, the application needs to use the internal parameters (focal length, principal point) and external parameters (rotation, translation) of the binocular camera to reconstruct the first mark point coordinates, so as to convert the first mark point coordinates into initial three-dimensional point cloud coordinates.

[0120] In order to improve the accuracy of plane fitting, it is necessary to suppress outliers of the initial three-dimensional point cloud coordinates, specifically including:

[0121] Randomly sample a small number of points (such as 3 non-collinear points) from the initial point cloud, fit an initial plane; calculate the distance of all points in the initial three-dimensional point cloud to the plane, and according to a predetermined threshold, distinguish between inliers (points with a distance less than the predetermined threshold) and outliers (points with a distance greater than the predetermined threshold); repeat the iteration, select the inlier set corresponding to the plane with the most inliers as the effective inlier set after removing outliers, and the effective inlier set includes a plurality of target three-dimensional point cloud coordinates.

[0122] Suppose the calibration board plane equation is: θ=(a,b,c,d), which needs to satisfy ax+by+cz+d=0, where (a,b,c) is the plane normal vector, and d is the distance from the plane to the origin. In order to avoid parameter scale ambiguity, a 2 +b 2 +c 2 =1.

[0123] The initial parameters θ0=(a0,b0,c0,d0) of the calibration board plane equation are calculated by ordinary least squares (OLS, Ordinary Least Squares) or singular value decomposition (SVD, Singular Value Decomposition), and on the basis of the initial parameters, the calibration board plane equation is iteratively reweighted least squares. When the iteration termination condition is met, the target parameters can be output.

[0124] Where the residual R i of each point (x i ,y i ,z i ) can be determined according to formula (4):

[0125] R i =ax i +by i +cz i +d0 (4)

[0126] On the basis of the initial parameters, the calibration board plane equation is iteratively reweighted least squares, including:

[0127] 1st iteration: calculate the residual of all points using the initial parameters Calculate the weight of all points according to the residual Then solve the new parameter θ1 by weighted least squares;

[0128] The second round of iteration: calculate the residual of all points using θ1 Update the weight of all points according to the residual Then solve the new parameter θ2 by weighted least squares;

[0129] …

[0130] The kth round of iteration: calculate the residual of all points using θ k-1 Calculate the residual of all points Update the weight of all points according to the residual Then solve the new parameter θ by weighted least squares k ;

[0131] If the parameter change of the kth round of iteration is less than the preset threshold, terminate the iteration and output θ k as the target parameter, so the plane equation of the calibration plate plane is determined.

[0132] Similarly, the plane equation of the jig plate and the calibration plate is determined in the same way, and the second mark point is processed according to the same method as described above, and the plane equation of the jig plate plane can be obtained.

[0133] Since the centering calibration of the depth sensor needs to determine the first center coordinates of the calibration plate and the second center coordinates of the jig plate, the first center coordinates of the calibration plate can be determined according to the coordinates of the plurality of first mark points, and the second center coordinates of the jig plate can be determined according to the coordinates of the plurality of second mark points.

[0134] It should be noted that the fitting of the plane equation and the determination of the center coordinates can not be in order, generally speaking, if the plane parallelism needs to be calibrated first, then the plane equation needs to be fitted first; if the center of the depth sensor needs to be calibrated first, then the first center coordinates and the second center coordinates need to be determined first.

[0135] The determination method of the first center coordinates and the second center coordinates is the same, and the first center coordinates are taken as an example to explain:

[0136] First, the plurality of first mark points are converted into a three-dimensional point cloud by using the intrinsic and extrinsic parameters of the binocular camera; a de-distortion function is called to perform de-distortion operation on the three-dimensional point cloud to obtain a de-distorted three-dimensional point cloud.

[0137] A camera projection matrix is constructed, a triangulation function is called, and the de-distorted three-dimensional point cloud is triangulated by using the camera projection matrix to obtain a 4D homogeneous coordinate point set;

[0138] The 4D homogeneous coordinate point set is subjected to homogeneous normalization, and the normalized 4D homogeneous coordinate point set is converted into a 3D point set;

[0139] The x-axis coordinate of each coordinate point in the 3D point set is averaged, the y-axis coordinate of each coordinate point in the 3D point set is averaged, and the z-axis coordinate of each coordinate point in the 3D point set is averaged to obtain the 3D center point coordinate of the calibration plate plane. The 3D center point coordinate of the calibration plate is the first center coordinate.

[0140] The second center coordinate of the jig plate can be determined in the same way as described above. In this way, the first center coordinate and the second center coordinate are real three-dimensional coordinates in the camera coordinate system.

[0141] S213, calibrating the plane parallelism of the depth sensor according to the calibration plate plane and the jig plate plane, and calibrating the center of the depth sensor according to the first center coordinate and the second center coordinate.

[0142] After the calibration plate plane and the jig plate plane are fitted, the plane parallelism of the depth sensor can be calibrated according to the calibration plate plane and the jig plate plane, including:

[0143] obtaining a first plane equation of the calibration plate plane and a second plane equation of the jig plate plane;

[0144] determining an offset angle between the jig plate plane and the calibration plate plane according to the first plane equation of the calibration plate plane and the second plane equation of the jig plate plane;

[0145] calibrating the plane parallelism of the depth sensor using the offset angle.

[0146] For example, assuming that the target parameters of the first plane equation are θ1=(a1, b1, c1, d1), and the target parameters of the second plane equation are θ2=(a2, b2, c2, d2), then the normal vector of the calibration plate plane is n1=(a1, b1, c1), and the normal vector of the jig plate plane is n2=(a2, b2, c2);

[0147] The vector dot product of the two normal vectors is: cosβ=|a1a2+b1b2+c1c2|;

[0148] Then the offset angle between the jig plate plane and the calibration plate plane is β=arccosa1a2+b1b2+c1c2|.

[0149] If the offset angle is greater than a preset angle threshold, it means that the parallelism of the jig plate plane does not meet the requirements. Since the jig plate and the depth sensor are relatively mounted on the same bearing panel of the test jig, the parallelism of the jig plate plane and the parallelism of the depth sensor are consistent. Therefore, it also means that the parallelism of the depth sensor does not meet the requirements.

[0150] At this time, the offset angle can be decomposed into a pitch angle η1 and a yaw angle η2, and the depth sensor can be rotated reversely around the X axis by η1 and reversely around the Y axis by η2 by using a test fixture (a six-axis platform) to offset the difference of the original offset angle, so as to calibrate the parallelism of the depth sensor plane.

[0151] In an embodiment, the center of the depth sensor is calibrated according to the first center coordinate and the second center coordinate, comprising:

[0152] determining a first position offset between the center of the fixture plate and the center of the calibration plate according to the first center coordinate and the second center coordinate;

[0153] determining a second position offset between the center of the fixture plate and the center of the depth sensor;

[0154] determining a target position offset of the center of the depth sensor relative to the center of the calibration plate according to the first position offset and the second position offset;

[0155] calibrating the center of the depth sensor center by using the target position offset.

[0156] In an embodiment, the second position offset between the center of the fixture plate and the center of the depth sensor is determined, comprising:

[0157] determining a horizontal offset x_offset of the center of the depth sensor relative to the center of the fixture plate according to the formula x_offset=AO-D1;

[0158] determining a vertical offset y_offset of the center of the depth sensor relative to the center of the fixture plate according to the formula y_offset=BO-D2; wherein,

[0159] AO is a theoretical distance of one edge of the bearing panel from the center of the depth sensor in the horizontal direction, and BO is a theoretical distance of the other edge of the bearing panel from the center of the depth sensor in the vertical direction, D1 is an actual distance of one edge of the bearing panel from the center of the depth sensor in the horizontal direction, and D2 is an actual distance of the other edge of the bearing panel from the center of the depth sensor in the vertical direction. One edge of the bearing panel and the other edge of the bearing panel are perpendicular to each other.

[0160] Specifically, since the fixture plate and the depth sensor are relatively loaded on the same bearing panel of the test fixture, the center of the fixture plate and the center of the depth sensor should be coaxial in theory, but in the actual installation process, human error will inevitably be introduced, so it is necessary to first determine the second position offset of the center of the depth sensor relative to the center of the fixture plate.

[0161] Similarly, theoretically, the jig plate center and the calibration plate center should also be coaxial, so it is also necessary to determine the first position offset of the jig plate center relative to the calibration table center; and finally the target position offset of the depth sensor center relative to the calibration plate center can be determined according to the first position offset and the second position offset.

[0162] If the target position offset is greater than the preset distance threshold, it means that the depth sensor center and the calibration plate center are not aligned, and then the depth sensor center needs to be calibrated according to the target position offset until it is determined that the position offset between the calibrated depth sensor center (optical axis) and the calibration plate center is less than the preset distance threshold (such as 1mm).

[0163] In the present application, the binocular camera can obtain the three-dimensional point cloud data of the jig plate and the calibration plate through stereo vision, directly solve the complete attitude of the two fitted planes, realize the global three-dimensional attitude comparison, and thus improve the calibration accuracy of the plane parallelism; similarly, when calibrating the center of the depth sensor, the first center coordinates and the second center coordinates determined by the binocular camera through the parallax principle are also three-dimensional, so the offset of the depth sensor relative to the calibration plate center can be accurately calculated, thereby improving the calibration accuracy of the depth sensor center.

[0164] Based on the same inventive concept as in the foregoing embodiments, the present embodiment also provides a kind of calibration electronic equipment of depth sensor, as shown in Figure 3 The electronic device comprises:

[0165] The acquisition unit 31 is used to receive the jig plate image and the calibration plate image collected by the binocular camera; along the shooting direction of the binocular camera, the jig plate and the depth sensor are relatively loaded on the same bearing panel of the test jig; the calibration plate is located behind the jig plate and has a preset distance from the jig plate;

[0166] The first determination unit 32 is used to determine a plurality of first marker point coordinates in the calibration plate image by using a hybrid feature detection algorithm and to determine a plurality of second marker point coordinates in the jig plate image by using a circle detection algorithm;

[0167] The second determination unit 33 is used to fit the calibration plate plane according to the plurality of first marker point coordinates and to determine the first center coordinates of the calibration plate, respectively; and to fit the jig plate plane according to the plurality of second markers and to determine the second center coordinates of the jig plate, respectively.

[0168] The calibration unit 34 is used to calibrate the plane parallelism of the depth sensor according to the calibration plate plane and the calibration plate plane, and to calibrate the center of the depth sensor according to the first center coordinates and the second center coordinates.

[0169] The first determination unit 31 is specifically configured to:

[0170] Determine the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction in the jig plate image;

[0171] According to the gray scale gradient of each pixel in the X-axis direction and the gray scale gradient in the Y direction, a corresponding second-order matrix is determined, and the second-order matrix is smoothed to obtain a second-order smoothed matrix;

[0172] According to the second-order smoothed matrix, a response value is determined, and the response value is normalized to obtain a normalized response value;

[0173] A plurality of target normalized response values are obtained by screening a plurality of normalized response values;

[0174] A plurality of first candidate corner points are obtained by performing non-maximum suppression processing on the plurality of target normalized response values;

[0175] The plurality of candidate corner points are further screened by using a boundary constraint condition and a minimum spacing constraint condition to obtain a plurality of second candidate corner points;

[0176] Sub-pixel level coordinate calculation is performed on the plurality of second candidate corner points by using quadratic function fitting to obtain sub-pixel level coordinates of the plurality of second candidate corner points;

[0177] The plurality of second candidate corner points are sorted in descending order of response value, and the sub-pixel level coordinates of the first N second candidate corner points are determined as the plurality of first marker point coordinates.

[0178] The electronic device introduced in the embodiment of the present application is the device used in the calibration method of the depth sensor of the embodiment of the present application, so based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and modification of the device, so it will not be repeated here. Any device used in the method of the embodiment of the present application belongs to the scope of protection of the present application.

[0179] Through one or more embodiments of the present application, the present application has the following beneficial effects or advantages:

[0180] The application provides a calibration method, device, medium and equipment of a depth sensor, the method comprising: receiving a jig plate image and a calibration plate image collected by a binocular camera; the jig plate and the depth sensor are loaded on the same bearing panel of a test jig in the shooting direction of the binocular camera; the calibration plate is located behind the jig plate and has a preset distance from the jig plate; a plurality of first marker point coordinates in the calibration plate image are determined by using a hybrid feature detection algorithm, and a plurality of second marker point coordinates in the jig plate image are determined by using a circle detection algorithm; a calibration plate plane is fitted according to the plurality of first marker point coordinates respectively, and a first center coordinate of the calibration plate is determined; the jig plate plane is fitted according to the plurality of second marker points respectively, and a second center coordinate of the jig plate is determined; the planar parallelism of the depth sensor is calibrated according to the calibration plate plane and the calibration plate plane, and the center of the depth sensor is calibrated according to the first center coordinate and the second center coordinate; in this way, the binocular camera can obtain three-dimensional point cloud data of the jig plate and the calibration plate through stereovision, directly solve the complete attitude of the two fitted planes, realize global three-dimensional attitude comparison, and then the calibration accuracy of the planar parallelism can be improved; similarly, when the center of the depth sensor is calibrated, the first center coordinate and the second center coordinate determined by the binocular camera through the parallax principle are also three-dimensional, so that the offset of the depth sensor relative to the center of the calibration plate can be accurately calculated, and the calibration accuracy of the center of the depth sensor can be improved.

[0181] Although preferred embodiments of the application have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such changes and modifications that fall within the scope of the application.

[0182] The above description is only the preferred embodiment of the application, and is not intended to limit the protection scope of the application, and any modification, equivalent replacement and improvement within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A calibration method for a depth sensor, characterized in that, The method includes: The device receives images of the fixture plate and calibration plate captured by a binocular camera; along the shooting direction of the binocular camera, the fixture plate and the depth sensor are mounted opposite each other on the same support panel of the test fixture; the calibration plate is located behind the fixture plate and at a preset distance from the fixture plate. The coordinates of multiple first marker points in the calibration plate image are determined using a hybrid feature detection algorithm, and the coordinates of multiple second marker points in the fixture plate image are determined using a circle detection algorithm. The calibration plate plane is fitted and the first center coordinates of the calibration plate are determined based on the coordinates of the plurality of first marker points; the jig plate plane is fitted and the second center coordinates of the jig plate are determined based on the plurality of second marker points. The planar parallelism of the depth sensor is calibrated according to the plane of the calibration plate and the plane of the fixture plate, and the center of the depth sensor is calibrated according to the first center coordinate and the second center coordinate.

2. The method as described in claim 1, characterized in that, The calibration board is a checkerboard background board. The step of determining the coordinates of multiple first marker points in the calibration board image using a hybrid feature detection algorithm includes: Determine the grayscale gradient of each pixel in the calibration plate image in the X-axis direction and the grayscale gradient in the Y-axis direction; The corresponding second-order matrix is ​​determined based on the gray-level gradient of each pixel in the X-axis direction and the gray-level gradient in the Y-axis direction, and the second-order matrix is ​​smoothed to obtain a second-order smoothed matrix. The response value is determined based on the second-order smoothing matrix, and the response value is normalized to obtain a normalized response value. Multiple normalized response values ​​are filtered to obtain multiple target normalized response values; Non-maximum suppression processing is applied to the normalized response values ​​of the multiple targets to obtain multiple first candidate corner points; The multiple candidate corner points are further filtered using boundary constraints and minimum spacing constraints to obtain multiple second candidate corner points; Subpixel-level coordinates of the plurality of second candidate corner points are calculated to obtain the subpixel-level coordinates of the plurality of second candidate corner points; The multiple second candidate corner points are sorted in descending order of response value, and the second candidate corner points corresponding to the first N response values ​​are selected. The sub-pixel level coordinates of the first N second candidate corner points are determined as the coordinates of the multiple first marker points.

3. The method as described in claim 1, characterized in that, The fixture plate has multiple circular holes of the same diameter; the step of determining the coordinates of multiple second marker points in the fixture plate image using a circle detection algorithm includes: The jig plate image is processed to obtain a grayscale image; Edge extraction is performed on the grayscale image to obtain multiple edge contours; Establish a spatial accumulator (x0, y0, r), which is used to record the number of times each three-dimensional array (x0, y0, r) is voted; (x0, y0) is the candidate circle center, and r is any radius within a preset radius range; Traverse all edge pixels on each edge contour, determine whether the edge pixel and the candidate circle center meet the voting conditions, and if they do, accumulate the spatial accumulator. The three-dimensional arrays that are greater than the accumulator threshold are retained, and the retained three-dimensional arrays are deduplicated to obtain multiple circle center coordinates; the multiple circle center coordinates are the coordinates of the multiple second marker points.

4. The method as described in claim 1, characterized in that, The step of fitting the calibration plate plane based on the coordinates of the plurality of first marker points includes: The parameters of the binocular camera are used to convert the multiple first marker points into initial three-dimensional point cloud coordinates, and outlier suppression is performed on the initial three-dimensional point cloud coordinates to obtain the target three-dimensional point cloud coordinates. The initial parameters of the calibration plate plane equation are determined using the target's three-dimensional point cloud coordinates. Based on the initial parameters, the calibration plate plane equation is subjected to iterative reweighted least squares processing; When the iteration termination condition is met, the target parameters of the calibration plate plane equation are output, and the plane equation of the fitted calibration plate plane is determined based on the target parameters.

5. The method as described in claim 1, characterized in that, The calibration of the plane parallelism of the depth sensor based on the plane of the fixture plate and the plane of the calibration plate includes: Obtain the first plane equation of the calibration plate plane and the second plane equation of the fixture plate plane; The offset angle between the fixture plate plane and the calibration plate plane is determined based on the first plane equation of the calibration plate plane and the second plane equation of the fixture plate plane; The plane parallelism of the depth sensor is calibrated using the offset angle.

6. The method as described in claim 1, characterized in that, The center of the depth sensor is calibrated based on the first center coordinate and the second center coordinate, including: The first position offset between the center of the fixture plate and the center of the calibration plate is determined based on the first center coordinate and the second center coordinate. Determine the second position offset between the center of the fixture plate and the center of the depth sensor; The target position offset of the depth sensor center relative to the calibration plate center is determined based on the first position offset and the second position offset; The center of the depth sensor is calibrated using the target position offset.

7. The method as described in claim 6, characterized in that, Determining the second position offset between the center of the fixture plate and the center of the depth sensor includes: The horizontal offset x_offset of the depth sensor center relative to the center of the fixture plate is determined according to the formula x_offset=AO-D1; The vertical offset y_offset of the depth sensor center relative to the jig plate center is determined according to the formula y_offset = BO - D2; where... AO is the theoretical distance in the horizontal direction between one edge of the support panel and the center of the depth sensor, BO is the theoretical distance in the vertical direction between the other edge of the support panel and the center of the depth sensor, D1 is the actual distance in the horizontal direction between one edge of the support panel and the center of the depth sensor, and D2 is the actual distance in the vertical direction between the other edge of the support panel and the center of the depth sensor.

8. A calibration electronic device for a depth sensor, characterized in that, The electronic device includes: The acquisition unit is used to receive images of the fixture plate and the calibration plate acquired by the binocular camera; along the shooting direction of the binocular camera, the fixture plate and the depth sensor are mounted opposite each other on the same support panel of the test fixture; the calibration plate is located behind the fixture plate and at a preset distance from the fixture plate; The first determining unit is used to determine the coordinates of multiple first marker points in the calibration plate image using a hybrid feature detection algorithm and to determine the coordinates of multiple second marker points in the fixture plate image using a circle detection algorithm; The second determining unit is configured to fit the calibration plate plane according to the coordinates of the plurality of first marker points and determine the first center coordinates of the calibration plate; and to fit the jig plate plane according to the plurality of second marker points and determine the second center coordinates of the jig plate. The calibration unit is used to calibrate the planar parallelism of the depth sensor according to the calibration plate plane and the calibration plate plane, and to calibrate the center of the depth sensor according to the first center coordinate and the second center coordinate.

9. The apparatus as claimed in claim 8, characterized in that, The first determining unit is specifically used for: Determine the grayscale gradient of each pixel in the jig plate image in the X-axis direction and the grayscale gradient in the Y-axis direction; The corresponding second-order matrix is ​​determined based on the gray-level gradient of each pixel in the X-axis direction and the gray-level gradient in the Y-axis direction, and the second-order matrix is ​​smoothed to obtain a second-order smoothed matrix. The response value is determined based on the second-order smoothing matrix, and the response value is normalized to obtain a normalized response value. Multiple normalized response values ​​are filtered to obtain multiple target normalized response values; Non-maximum suppression processing is applied to the normalized response values ​​of the multiple targets to obtain multiple first candidate corner points; The multiple candidate corner points are further filtered using boundary constraints and minimum spacing constraints to obtain multiple second candidate corner points; The subpixel-level coordinates of the plurality of second candidate corner points are calculated by fitting a quadratic function. The multiple second candidate corner points are sorted in descending order of response value, and the second candidate corner points corresponding to the first N response values ​​are selected. The sub-pixel level coordinates of the first N second candidate corner points are determined as the coordinates of the multiple first marker points.

10. A calibration system for a depth sensor, characterized in that, The system includes: a binocular camera, a test fixture, a fixture plate, a calibration plate, and the calibration electronic equipment as described in claim 8; Along the shooting direction of the binocular camera, the fixture plate and the depth sensor are mounted opposite each other on the same support panel of the test fixture; the calibration plate is located behind the fixture plate and at a predetermined distance from the fixture plate; The calibration electronic device is specifically used for: acquiring images of a fixture plate and a calibration plate captured by the binocular camera; determining the coordinates of multiple first marker points in the calibration plate image using a hybrid feature detection algorithm and determining the coordinates of multiple second marker points in the fixture plate image using a circle detection algorithm; fitting a calibration plate plane based on the coordinates of the multiple first marker points and determining the first center coordinates of the calibration plate; fitting the fixture plate plane based on the multiple second marker points and determining the second center coordinates of the fixture plate; calibrating the planar parallelism of the depth sensor based on the calibration plate plane and the calibration plate plane; and calibrating the center of the depth sensor based on the first center coordinates and the second center coordinates.

Images