Calibration method of speckle structured light camera

Abstract

Embodiments of the present application are suitable for the field of optical measurement technology, and provide a calibration method of a speckle structured light camera, which comprises: collecting a reference plane speckle image and a plurality of calibration plate images by the speckle structured light camera, the calibration plate image being an image of a circular dot array calibration plate; performing preliminary calibration based on the coordinates of each white dot on the calibration plate image and the reference plane speckle image to obtain first calibration parameters and second calibration parameters; then performing joint calibration based on the first calibration parameters and the second calibration parameters to obtain optimized target calibration parameters; and then correcting the reference plane speckle image based on the target calibration parameters to obtain a virtual reference plane speckle image parameter. When the speckle structured light camera is used for shooting, the virtual reference plane speckle image parameter can be used to calculate the depth information of the pixel points in the image. Through the above method, the error in the calibration process can be reduced, and the accuracy of the depth ranging result can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of optical measurement, and particularly relates to a calibration method of a speckle structured light camera. BACKGROUND

[0002] The speckle structured light is a non-contact optical 3D measurement technology, and has many advantages such as high frame rate, low cost, near distance precision and the like compared with other optical 3D measurement technologies, and is widely applied in many fields such as consumer electronics and industrial measurement.

[0003] The picture captured by the speckle structured light camera can include depth information of pixels in the picture. When the speckle structured light camera is used, calibration of the speckle structured light camera is needed, so that three-dimensional information can be obtained by using two-dimensional information, and the captured picture can have depth information.

[0004] The speckle structured light camera has systematic errors caused by many factors, which seriously affect the accuracy of the depth measurement result. These errors can include errors of speckle structured light camera parameters, errors of projector parameters, and errors caused by the fact that the reference plane is not perpendicular to the z-axis of the camera coordinate system and the reference distance is not accurate. These errors will affect the accuracy of the depth measurement of the speckle structured light camera.

[0005] In addition, in the calibration of the speckle structured light camera, a reference plane for calibration is needed, which increases the difficulty of calibration and reduces the efficiency of calibration. SUMMARY

[0006] Therefore, the embodiments of the present application provide a calibration method of a speckle structured light camera, which can improve the accuracy of calibration, thereby ensuring the accuracy of depth measurement, and at the same time, reduce the difficulty of calibration and improve the efficiency of calibration.

[0007] The first aspect of the embodiments of the present application provides a calibration method of a speckle structured light camera, comprising:

[0008] The speckle structured light camera captures a reference plane speckle image and a plurality of calibration board images, the reference plane speckle image is an image of a white reflective plane as a reference plane, the calibration board image is an image of a circular dot array calibration board, the circular dot array calibration board includes a plurality of white circular dots, the plurality of calibration board images include a plurality of first calibration board images of the circular dot array calibration board in different postures under uniform illumination and a plurality of second calibration board images under speckle illumination, and the speckle structured light camera includes a shooting camera and a speckle projector, wherein the speckle illumination is provided by the speckle projector;

[0009] determine first calibration parameters of the speckle structured light camera based on the coordinates of each white circle dot in the reference plane speckle image and the first calibration board image, the first calibration parameters including camera intrinsic parameters of the shooting camera and attitude parameters of the circle dot array calibration board;

[0010] determine second calibration parameters of the speckle structured light camera based on the first calibration parameters and the coordinates of each white circle dot in the second calibration board image, the second calibration parameters including center point coordinate parameters of the speckle projector and reference plane equation parameters;

[0011] determine target calibration parameters of the speckle structured light camera based on the first calibration parameters and the second calibration parameters, the target calibration parameters including target camera intrinsic parameters, target attitude parameters, target center point coordinate parameters and target reference plane equation parameters;

[0012] correct the reference plane speckle image based on the target calibration parameters to obtain virtual reference plane speckle image parameters, the virtual reference plane speckle image parameters being used to calculate depth information of a pixel point in an image when the speckle structured light camera is working.

[0013] A second aspect of the embodiment of the application provides a speckle structured light camera calibration device, comprising:

[0014] an image acquisition module, configured to acquire a reference plane speckle image and a plurality of calibration board images by a speckle structured light camera, the reference plane speckle image being an image of a white reflective plane as a reference plane, the calibration board image being an image of a circle dot array calibration board, the circle dot array calibration board including a plurality of white circle dots, the plurality of calibration board images including a plurality of first calibration board images of the circle dot array calibration board in different attitudes under uniform illumination and a plurality of second calibration board images under speckle illumination, the speckle structured light camera including a shooting camera and a speckle projector, wherein the speckle illumination is provided by the speckle projector;

[0015] a first calibration module, configured to determine first calibration parameters of the speckle structured light camera based on the coordinates of each white circle dot in the reference plane speckle image and the first calibration board image, the first calibration parameters including camera intrinsic parameters of the shooting camera and attitude parameters of the circle dot array calibration board;

[0016] a second calibration module, configured to determine second calibration parameters of the speckle structured light camera based on the first calibration parameters and the coordinates of each white circle dot in the second calibration board image, the second calibration parameters including center point coordinate parameters of the speckle projector and reference plane equation parameters;

[0017] A joint calibration module is configured to determine target calibration parameters of the speckle structured light camera based on the first calibration parameters and the second calibration parameters, the target calibration parameters including target camera intrinsic parameters, target pose parameters, target center point coordinate parameters, and target reference plane equation parameters.

[0018] A correction module is configured to correct the reference plane speckle image based on the target calibration parameters to obtain virtual reference plane speckle image parameters, the virtual reference plane speckle image parameters being used to calculate depth information of a pixel point in an image when the speckle structured light camera is working.

[0019] A third aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the first aspect.

[0020] A fourth aspect of the embodiments of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0021] A fifth aspect of the embodiments of the present application provides a computer program product, when the computer program product is executed on a terminal device, the terminal device executes the method of the first aspect.

[0022] Compared with the prior art, the embodiments of the present application have the following advantages:

[0023] When the method in the embodiment of the present application is applied to calibrate the speckle structured light camera, the speckle structured light camera can be used to collect a reference plane speckle image and a plurality of calibration board images, wherein the calibration board images are images of a circular dot array calibration board, the circular dot array calibration board includes a plurality of white circular dots, and the plurality of calibration board images include a plurality of first calibration board images of the circular dot array calibration board in different postures under uniform illumination and a plurality of second calibration board images under speckle illumination. Then, based on the first calibration board images, the camera intrinsic parameters of the shooting camera and the posture parameters of the calibration board can be determined; and based on the second calibration board images, the center point coordinate parameters of the speckle projector and the reference plane equation parameters can be determined. After separate calibration, the camera intrinsic parameters of the shooting camera, the posture parameters of the calibration board, the center point coordinate parameters and the reference plane equation parameters obtained can be used to jointly calibrate the shooting camera, the projector and the reference plane of the speckle projector, so as to obtain the optimized camera intrinsic parameters of the shooting camera, the posture parameters of the calibration board, the center point coordinate parameters and the reference plane equation parameters. Based on the optimized camera intrinsic parameters of the shooting camera, the posture parameters of the calibration board, the center point coordinate parameters and the reference plane equation parameters, the reference plane speckle image can be corrected, so that the final virtual reference plane speckle image parameters used for depth ranging can be obtained. Based on the parameters of the preliminary calibration, the joint calibration is performed, so that the accuracy of the parameters can be improved and the error of the parameters can be reduced; at the same time, based on the more accurate parameters obtained by the calibration, the correction is performed again, so that the accuracy of the depth ranging can be further improved. In addition, the calibration method in the present application only uses the circular dot array calibration board, does not need to mark the reference plane, and does not need to be perpendicular to the optical axis of the camera, which is convenient to implement, can improve the calibration efficiency, and is suitable for application in large-scale production lines. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows.

[0025] Figure 1 is a step flow diagram of a speckle structured light camera calibration method provided by the embodiment of the present application;

[0026] Figure 2 is a reference plane speckle diagram provided by the embodiment of the present application;

[0027] Figure 3 is a circular dot array calibration board diagram provided by the embodiment of the present application;

[0028] Figure 4 is a calibration board image diagram under uniform illumination provided by the embodiment of the present application;

[0029] Figure 5is a calibration board image schematic diagram under speckle illumination provided by an embodiment of the present application;

[0030] Figure 6 is a calibration geometric relationship schematic diagram provided by an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a calibration device of a speckle structured light camera provided by an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0034] The speckle structured light camera mainly includes two parts, first, a speckle projector for projecting speckles into the scene, and second, a shooting camera for shooting the scene speckle image. The projector and the shooting camera are spaced apart by a certain distance to form a baseline necessary for triangulation. The speckle structured light camera stores a reference plane speckle image for speckle matching with the shot scene speckle image, calculating the parallax, and then obtaining the depth. In the present application, the calibration of the speckle structured light camera is equivalent to obtaining the reference plane speckle image.

[0035] The speckle structured light camera has systematic errors caused by many factors, which seriously affect the accuracy of the depth measurement result. These errors mainly include: camera lens distortion, principal point, focal length error; projector center skew error relative to the x-axis of the camera coordinate system; error caused by the non-perpendicularity of the reference plane to the z-axis of the camera coordinate system and the inaccuracy of the reference distance. The existing speckle structured light camera calibration method can usually only handle the first two of the three errors. For the third error, it needs to rely on special markers on the reference plane or speckle image, which increases the cost and complexity of calibration, reduces the calibration efficiency, and is not very convenient to apply.

[0036] Based on this, the present application provides a speckle structured light camera calibration method, which can reduce the calibration difficulty, improve the calibration efficiency, and improve the calibration accuracy.

[0037] The technical solutions of the present application are described below through specific embodiments.

[0038] REFERENCE Figure 1, a schematic diagram of a step flow of a calibration method of a speckle structured light camera provided by an embodiment of the present application is shown, and can specifically include the following steps:

[0039] S101, a reference plane speckle image and a plurality of calibration board images are collected by a speckle structured light camera.

[0040] The execution subject of the embodiment of the present application can be a terminal device, which can be a shooting device, and the shooting device can be calibrated using the method in the present application; or it can be other computer devices, which can obtain the image collected by the shooting device, calculate the virtual reference plane speckle image parameters, and then send the virtual reference plane speckle image parameters to the shooting device for use. The specific type of the terminal device is not limited in the embodiment of the present application.

[0041] The reference plane speckle image is the image of a white reflective plane as a reference plane, and the reflective characteristic of the white plane is close to that of a diffuse reflective material. Figure 2 is a reference plane speckle diagram provided by an embodiment of the present application. The speckle structured light camera is placed at a certain distance from the white reflective plane, and the reference plane speckle image can be collected.

[0042] The calibration board image is the image of a circular dot array calibration board, which includes a plurality of white circular dots. Figure 3 is a circular dot array calibration board diagram provided by an embodiment of the present application. As shown in Figure 3 , the circular dot array calibration board has a known circular dot spacing, the circular dots are white material, the background is black material, and the reflective characteristic is close to that of a diffuse reflective material.

[0043] The plurality of calibration board images include a plurality of first calibration board images of the circular dot array calibration board in uniform illumination and a plurality of second calibration board images in speckle illumination, wherein the speckle illumination is provided by a speckle projector. Figure 4 is a calibration board image diagram in uniform illumination provided by an embodiment of the present application; Figure 5 is a calibration board image diagram in speckle illumination provided by an embodiment of the present application. That is Figure 4 is a schematic diagram of a first calibration board image, Figure 5 is a schematic diagram of a second calibration board image.

[0044] S102, based on the coordinates of each white circular dot in the reference plane speckle image and the first calibration board image, the first calibration parameter of the speckle structured light camera is determined.

[0045] The first calibration parameter can include the camera intrinsic parameter of the shooting camera and the attitude parameter of the circular dot array calibration board.

[0046] Figure 6The calibration geometric relationship schematic diagram is provided by the embodiment of the application. Figure 6 The reference plane in the reference plane corresponds to the reference plane speckle pattern, and the corresponding expression is Ax+By+z=D, the calibration board corresponds to the calibration board image, and the center of the projector is P C

[0047] When calibration is performed, the first circle pixel coordinates of each white circle point in the pixel coordinate system can be determined based on the coordinate conversion relationship between the camera coordinate system and the pixel coordinate system of the camera. Each white circle point is a plurality of white circle point images in each first calibration board image.

[0048] The camera model of "pinhole+distortion" can be used to calibrate the camera parameters. The point P=[x,y,z] in the camera coordinate system T After perspective projection, distortion mapping and pixel sampling, it can become a point U=[u,v] in the pixel coordinate system T , and the specific process is represented as:

[0049] x'=x / z

[0050] y'=y / z

[0051] x''=x'(1+k1r 2 +k2r 4 +k3r 6 )+2p1x'y'+p2(r 2 +2x' 2 )

[0052] y''=y'(1+k1r 2 +k2r 4 +k3r 6 )+2p2x'y'+p1(r 2 +2y' 2 )

[0053] u=f x x''+c x

[0054] v=f y y''+c y

[0055] Wherein, [x, y, z] is used to represent the point coordinates in the camera coordinate system, [u, v] is used to represent the point coordinates in the pixel coordinate system, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the principal point in the x direction, c y is the principal point in the y direction, k1, k2 and k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients.

[0056] The present scheme uses U=cam(P) to represent the imaging process.

[0057] As shown in Figure 6 , the coordinates of the circular dots on the calibration board are U x =[u x , v x , 0] T , the point coordinates P x in the camera coordinate system are related to the pose R(φ) of the calibration board and t, and can be expressed as:

[0058] P x =R(φ)U+t

[0059] where φ is the axis angle, t is the translation vector, and R(φ) is the rotation matrix after the transformation by the Rodrigues formula. P x After imaging by the camera, the pixel coordinates are expressed as U b =cam(P x ).

[0060] That is, based on the coordinate conversion relationship between the camera coordinate system and the pixel coordinate system of the shooting camera, the first circular dot pixel coordinates of each white circular dot in the pixel coordinate system can be determined according to the following formula:

[0061] U x =[u x , v x , 0] T

[0062] P x =R(φ)U x +t

[0063] U b =cam(P x )

[0064] where U x is the coordinates of the white circular dot in the first calibration board image, P x is the point coordinates of the white circular dot in the camera coordinate system, φ is the axis angle, t is the translation vector, R(φ) is the rotation matrix, U b is the first circular dot pixel coordinates, and cam is the coordinate conversion relationship.

[0065] In addition, based on the circular dot extraction algorithm, each first calibration board image can be processed to obtain the second circular dot pixel coordinates of each white circular dot of each first calibration board image.

[0066] Then, the first calibration parameters can be determined based on the first circular dot pixel coordinates and the second circular dot pixel coordinates.

[0067] It should be noted that in actual calculations, the coordinates of the white dot can be the coordinates of the center of the dot.

[0068] Calculate the value of the first function based on the first and second pixel coordinates of each white dot.

[0069] The parameter corresponding to the minimum value of the first function is used as the first calibration parameter;

[0070] The first function is:

[0071]

[0072] Among them, Ω b The first calibration parameter can include f. x f y c x c y ,k1,k2,k3,p1,p2,φ i , t i , where f x f y c x c y k1, k2, k3, p1, p2 are camera intrinsic parameters, φ i , t i Here are the attitude parameters of the calibration board, where i is the attitude index of the calibration board, j is the index of the white dot, and U... b The pixel coordinates of the first dot. These are the pixel coordinates of the second dot.

[0073] S103, based on the first calibration parameters and the coordinates of each white dot in the second calibration plate image, determine the second calibration parameters of the speckle structured light camera.

[0074] The second calibration parameters mentioned above include the center point coordinates of the speckle projector and the reference plane equation parameters.

[0075] Based on the center point coordinates of the speckle projector and the first calibration parameters, the pixel coordinates of the third point of each white dot in the camera coordinate system can be determined.

[0076] like Figure 6 As shown, let the coordinates of the projector's center point be P. c =[x c y c , z c ] T The equation of the reference plane is Ax + By + z = D, and the point P is a circle. x The coordinates of the intersection point P of the speckle pattern and the reference plane. r =[x r y r , zr ] T is expressed as:

[0077]

[0078] P r = P c + s(P x -P c )

[0079] After being imaged by the camera, the pixel coordinates of the white dots are U s = cam(P r ).

[0080] That is, based on the center point coordinates of the speckle projector, the third pixel coordinates of the white dots in the camera coordinate system can be determined according to the following formula:

[0081]

[0082] P r = P c + s(P x -P c )

[0083] U s = cam(P r )

[0084] Wherein, A, B, D are parameters of the plane reference equation, P c is the center point coordinates, P x is the point coordinates of the white dots in the camera coordinate system, P r is the intersection coordinates of the white dots and the reference plane, and cam is the coordinate conversion relationship.

[0085] Based on the second pixel coordinates of the white dots, the fourth pixel coordinates of the white dots in the camera coordinate system are determined. In the above step, after the camera intrinsic parameter calibration, Ω b is known, therefore, the second pixel coordinates of the white dots are obtained After that, the speckle patterns at the matching dots on the reference plane speckle pattern are matched using the digital image correlation algorithm, and the coordinates of the same-named speckle dots on the reference image are obtained That is, the fourth pixel coordinates of the white dots are obtained.

[0086] Based on the third pixel coordinates and the fourth pixel coordinates, the second calibration parameter is determined. Specifically, the values of the second function can be calculated using the third pixel coordinates and the fourth pixel coordinates of the white dots. Then the parameter value corresponding to the minimum value of the second function is taken as the value of the second calibration parameter.

[0087] Exemplarily, the reference plane equation and the projector light center are obtained by minimizing a second function as follows:

[0088]

[0089] wherein the second calibration parameter Ω s may include A, B, D, P c , P c is a center point coordinate parameter, A, B, and D are reference plane equation parameters, i is an index of a calibration board pose, j is an index of a white circle point, U s is a third circle point pixel coordinate, is a fourth circle point pixel coordinate.

[0090] S104, determining a target calibration parameter of the speckle structured light camera based on the first calibration parameter and the second calibration parameter.

[0091] Based on the previously calibrated first calibration parameter and the second calibration parameter, joint optimization of the camera, the projector, and the reference plane parameters can be performed, and a target function as follows is minimized:

[0092]

[0093] wherein Ω is a target calibration parameter, i is an index of a calibration board pose, j is an index of a white circle point, U b is a first circle point pixel coordinate, is a second circle point pixel coordinate, U s is a third circle point pixel coordinate, is a fourth circle point pixel coordinate.

[0094] The parameter value corresponding to the minimum value is taken as a value of the target calibration parameter. The target calibration parameter includes a target camera intrinsic parameter, a target pose parameter, a target center point coordinate parameter, and a target reference plane equation parameter;

[0095] S105, correcting the reference plane speckle image based on the target calibration parameter to obtain a virtual reference plane speckle image parameter.

[0096] After the above supplement, the camera intrinsic parameter, the projector light center, and the reference plane equation can be obtained. After the parameters are obtained, the reference reference plane speckle image can be corrected, and the virtual reference plane speckle image can be the corrected reference plane speckle image.

[0097] When the correction is performed, the terminal device can construct a virtual camera coordinate system, and an origin of the virtual camera coordinate system coincides with an origin of the camera coordinate system. A relative pose between the virtual camera coordinate system and the camera coordinate system can be represented using a virtual rotation matrix R′.

[0098] The rotation matrix in the camera coordinate system is R′=[e1,e2,e3], where e3 = e1 × e2.

[0099] Given the image coordinates [u′, v′] of a virtual camera. T Focal length f′ x ,f′ y Principal point c′ x ,c′ y The distance to the virtual reference plane is Z0. The coordinates of the intersection point of the line connecting the origin and the pixel with the reference plane in the original camera coordinate system are P′. x for

[0100] P′ x =Z0R′[(u′-c′) x ) / f′ x ,(v′-c′ y ) / f′ y ,1] T

[0101] Following the speckle imaging process in the same calibration procedure, the coordinates U′ of the speckle image on the virtual reference plane in the original reference image can be calculated. s Then, by interpolation, the grayscale of each pixel in the virtual reference plane speckle image can be obtained. The grayscale of each pixel in the virtual reference plane speckle image is the aforementioned virtual reference plane speckle image parameter. The virtual reference plane speckle image is written into the camera for subsequent speckle matching.

[0102] The virtual reference plane speckle image parameters are used to calculate the depth information of pixels in the image during the operation of the speckle structured light camera. After calibration, during shooting, distortion correction and epipolar correction are performed on the acquired scene speckle image. An image matching method is then used to perform speckle matching with the virtual reference plane speckle image to obtain the disparity d of each pixel. The speckle structured light camera then determines the pixel depth using the following formula:

[0103]

[0104]

[0105] Where z is the depth of a pixel in the captured image, Z0 is the distance to the virtual reference plane, [x c ,y c ,z c [ ] represents the coordinates of the center point of the speckle projector, f′ x Let d be the virtual focal length in the x-direction, and d be the parallax between the pixels in the captured image and the pixels in the virtual reference plane image.

[0106] The embodiment of the application provides a calibration method of a speckle structured light camera, a joint calibration method is performed on a camera, a projector and a reference plane, camera intrinsic parameters, a projector light center and a reference plane equation can be accurately obtained, three systematic errors are effectively eliminated, and the accuracy of a depth ranging result is ensured. Meanwhile, the calibration method in the application only uses a circular dot array calibration board, does not need to mark a reference plane, does not need the perpendicularity between the plane and the optical axis of the camera, is convenient to implement, can improve the calibration efficiency, and is suitable for large-scale production in a production line.

[0107] It should be noted that the size of the serial number of each step in the above embodiment does not mean the order of execution, the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0108] Reference Figure 7 , a schematic diagram of a speckle structured light camera calibration device provided by the embodiment of the application is shown, which can specifically include an image acquisition module 71, a first calibration module 72, a second calibration module 73, a joint calibration module 74 and a correction module 75, wherein:

[0109] The image acquisition module 71 is configured to acquire a reference plane speckle image and a plurality of calibration board images by using a speckle structured light camera, the reference plane speckle image is an image of a white reflective plane as a reference plane, the calibration board image is an image of a circular dot array calibration board, the circular dot array calibration board includes a plurality of white dots, the plurality of calibration board images include a plurality of first calibration board images of the circular dot array calibration board in different postures under uniform illumination and a plurality of second calibration board images under speckle illumination, the speckle structured light camera includes a shooting camera and a speckle projector, and the speckle illumination is provided by the speckle projector;

[0110] The first calibration module 72 is configured to determine first calibration parameters of the speckle structured light camera based on the coordinates of each white dot in the reference plane speckle image and the first calibration board image, wherein the first calibration parameters include camera intrinsic parameters of the shooting camera and posture parameters of the circular dot array calibration board;

[0111] The second calibration module 73 is configured to determine second calibration parameters of the speckle structured light camera based on the first calibration parameters and the coordinates of each white dot in the second calibration board image, wherein the second calibration parameters include center point coordinate parameters of the speckle projector and reference plane equation parameters;

[0112] The joint calibration module 74 is configured to determine target calibration parameters of the speckle structured light camera based on the first calibration parameters and the second calibration parameters, wherein the target calibration parameters include target camera intrinsic parameters, target posture parameters, target center point coordinate parameters and target reference plane equation parameters.

[0113] a correction module 75, configured to correct the reference plane speckle image based on the target calibration parameter, to obtain a virtual reference plane speckle image parameter, the virtual reference plane speckle image parameter being used to calculate depth information of a pixel point in an image when the structured light camera works.

[0114] In a possible implementation, the first calibration module 72 described above comprises:

[0115] a first circle point pixel coordinate determination sub-module, configured to determine first circle point pixel coordinates of each white circle point in a pixel coordinate system based on a coordinate conversion relationship between a camera coordinate system and the pixel coordinate system of the shooting camera;

[0116] a second circle point pixel coordinate determination sub-module, configured to process each of the first calibration board images based on a circle point extraction algorithm to obtain second circle point pixel coordinates of each white circle point;

[0117] a first calibration parameter determination sub-module, configured to determine the first calibration parameter based on the first circle point pixel coordinates and the second circle point pixel coordinates.

[0118] In a possible implementation, the first circle point pixel coordinates of each white circle point in the pixel coordinate system are determined based on the coordinate conversion relationship between the camera coordinate system and the pixel coordinate system of the shooting camera, and are achieved according to the following formula:

[0119] U x =[u x ,v x ,0] T

[0120] P x =R(φ)U x +t

[0121] U b =cam(P x )

[0122] wherein U x is a coordinate of the white circle point in the first calibration board image, P x is a point coordinate of the white circle point in the camera coordinate system, φ is an axis angle, t is a translation vector, R(φ) is a rotation matrix, U b is the first circle point pixel coordinate, and cam is the coordinate conversion relationship;

[0123] wherein the coordinate conversion relationship is:

[0124] x′=x / z

[0125] y′=y / z

[0126] r 2 = x' 2 + y' 2

[0127] x" = x'(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p1x'y' + p2(r 2 + 2x' 2 )

[0128] y" = y'(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p2x'y' + p1(r 2 + 2y' 2 )

[0129] u = f x x" + c x

[0130] v = f y y" + c y

[0131] where [x, y, z] is used to represent the point coordinates in the camera coordinate system, [u, v] is used to represent the point coordinates in the pixel coordinate system, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the principal point in the x direction, c y is the principal point in the y direction, k1, k2 and k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients.

[0132] In a possible implementation, the first calibration parameter determination sub-module described above comprises:

[0133] a calculation unit, configured to calculate a value of a first function based on the first circle point pixel coordinates and the second circle point pixel coordinates of each white circle point;

[0134] a determination unit, configured to determine a parameter corresponding to a minimum value of the first function as the first calibration parameter;

[0135] The first function is:

[0136]

[0137] where Ω b is the first calibration parameter, i is a calibration board posture index, j is a white circle point index, U b is the first circle point pixel coordinate, The second circle pixel coordinates are determined.

[0138] In a possible implementation, the second calibration module 73 includes:

[0139] A third circle pixel coordinate determination submodule is configured to determine third circle pixel coordinates of each white circle in the camera coordinate system based on the center point coordinates of the speckle projector and the first calibration parameters;

[0140] A fourth circle pixel coordinate determination submodule is configured to determine fourth circle pixel coordinates of each white circle in the camera coordinate system based on the second circle pixel coordinates of the white circle;

[0141] A second calibration parameter determination submodule is configured to determine the second calibration parameters based on the third circle pixel coordinates and the fourth circle pixel coordinates.

[0142] In a possible implementation, the third circle pixel coordinates of each white circle in the camera coordinate system are determined based on the center point coordinates of the speckle projector according to the following formula:

[0143]

[0144] P r =P c +s(P x -P c )

[0145] U s =cam(P r )

[0146] wherein A, B, and D are parameters of a plane reference equation, P c is the center point coordinates, P x is the point coordinates of the white circle in the camera coordinate system, P r is the intersection coordinates of the white circle and the reference plane, and cam is a coordinate conversion relationship.

[0147] In a possible implementation, the joint calibration module 74 includes:

[0148] A calculation submodule is configured to calculate a value of a target function based on the first circle pixel coordinates, the second circle pixel coordinates, the third circle pixel coordinates, and the fourth circle pixel coordinates of each white circle;

[0149] A determination submodule is configured to determine parameters corresponding to a minimum value of the target function as the target calibration parameters;

[0150] The target function is as follows:

[0151]

[0152] wherein Ω is a target calibration parameter, i is a calibration board pose index, j is a white circle point index, U b is a first circle point pixel coordinate, is a second circle point pixel coordinate, U s is a third circle point pixel coordinate, is a fourth circle point pixel coordinate.

[0153] In a possible implementation, the above-mentioned correction module 75 comprises:

[0154] a construction sub-module, configured to construct a virtual camera coordinate system, an origin of the virtual camera coordinate system coinciding with an origin of the camera coordinate system;

[0155] a coordinate mapping relationship determination sub-module, configured to determine a coordinate mapping relationship of a virtual corresponding coordinate of an intersection point of a line connecting the origin in the virtual camera coordinate system and the reference plane in the camera coordinate system;

[0156] a virtual reference plane speckle image parameter determination sub-module, configured to calculate a gray value of each pixel point in a virtual reference plane speckle image based on the coordinate mapping relationship, the gray value of each pixel point in the virtual reference plane speckle image being the virtual reference plane speckle image parameter.

[0157] In a possible implementation, the coordinate mapping relationship is:

[0158] P′ x = Z0R′[(u′-c′ x ) / f′ x ,(v′-c′ y ) / f′ y ,1] T

[0159] R′ = [e1, e2, e3],

[0160]

[0161]

[0162] e3 = e1 x e2

[0163] wherein P′ x is a virtual corresponding coordinate, Z0 is a virtual reference plane distance, R′ is the virtual rotation matrix, the virtual rotation matrix being used to represent a pose of the virtual camera coordinate system relative to the camera coordinate system, [u′, v′] is a coordinate of a point in the virtual camera coordinate system, f′ x is a virtual focal length in an x direction, f′y Let c′ be the virtual focal length in the y-direction. x Let c′ be the virtual principal point in the x-direction. y Let be the virtual principal point in the y-direction.

[0164] In one possible implementation, the speckle structured light camera determines the depth of a pixel using the following formula:

[0165]

[0166]

[0167] Where z is the depth of a pixel in the captured image, Z0 is the distance to the virtual reference plane, [x c ,y c ,z c ] represents the coordinates of the center point of the speckle projector, f′ x Let d be the virtual focal length in the x-direction, and d be the parallax between the pixels in the captured image and the pixels in the virtual reference plane image.

[0168] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0169] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 8 As shown, the terminal device 8 of this embodiment includes: at least one processor 80 ( Figure 8 (Only one is shown in the diagram), memory 81, and computer program 82 stored in said memory 81 and executable on said at least one processor 80, which, when executed, implements the steps in any of the above method embodiments.

[0170] The terminal device 8 may be a camera or other terminal device. This terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of terminal device 8 and does not constitute a limitation on terminal device 8. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0171] The processor 80 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0172] The memory 81 can be an internal storage unit of the terminal device 8 in some embodiments, such as a hard disk or a memory of the terminal device 8. The memory 81 can also be an external storage device of the terminal device 8 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 81 can include both an internal storage unit and an external storage device of the terminal device 8. The memory 81 is used to store an operating system, application programs, a boot loader, data, and other programs, such as program codes of the computer program, etc. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0173] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each of the above-mentioned method embodiments.

[0174] The embodiments of the present application provide a computer program product. When the computer program product is run on a terminal device, the terminal device is caused to implement the steps in each of the above-mentioned method embodiments.

[0175] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A calibration method for a speckle structured light camera, characterized in that, include: A speckle structured light camera acquires a speckle image of a reference plane and multiple calibration plate images. The speckle image of the reference plane is an image of the white reflective plane of the reference plane. The calibration plate images are images of a dot array calibration plate, which includes multiple white dots. The multiple calibration plate images include multiple first calibration plate images of the dot array calibration plate in different orientations under uniform illumination and multiple second calibration plate images under speckle illumination. The speckle structured light camera includes a camera and a speckle projector, wherein the speckle illumination is provided by the speckle projector. Based on the coordinates of each white dot in the reference plane speckle image and the first calibration plate image, the first calibration parameters of the speckle structured light camera are determined. The first calibration parameters include the camera intrinsic parameters of the shooting camera and the attitude parameters of the dot array calibration plate. Based on the first calibration parameters and the coordinates of each white dot in the second calibration plate image, the second calibration parameters of the speckle structured light camera are determined. The second calibration parameters include the center point coordinate parameters of the speckle projector and the reference plane equation parameters. Based on the first calibration parameters and the second calibration parameters, the target calibration parameters of the speckle structured light camera are determined. The target calibration parameters include the target camera intrinsic parameters, target attitude parameters, target center point coordinate parameters, and target reference plane equation parameters. Based on the target calibration parameters, the reference plane speckle image is corrected to obtain virtual reference plane speckle image parameters. The virtual reference plane speckle image parameters are used to calculate the depth information of pixels in the image when the speckle structured light camera is working. The step of correcting the speckle image of the reference plane based on the target calibration parameters to obtain virtual reference plane speckle image parameters includes: Construct a virtual camera coordinate system, the origin of which coincides with the origin of the camera coordinate system; Determine the coordinate mapping relationship between the intersection of the line connecting the point in the virtual camera coordinate system and the origin and the reference plane, and the virtual corresponding coordinates in the camera coordinate system. Based on the coordinate mapping relationship, the grayscale value of each pixel in the virtual reference plane speckle image is calculated, and the grayscale value of each pixel in the virtual reference plane speckle image is the parameter of the virtual reference plane speckle image.

2. The method as described in claim 1, characterized in that, The determination of the first calibration parameters of the speckle structured light camera based on the coordinates of each white dot in the reference plane speckle image and the first calibration plate image includes: Based on the coordinate transformation relationship between the camera coordinate system and the pixel coordinate system of the shooting camera, the first pixel coordinate of each white dot in the pixel coordinate system is determined. Based on the dot extraction algorithm, each of the first calibration board images is processed to obtain the second dot pixel coordinates of each white dot; The first calibration parameter is determined based on the first dot pixel coordinates and the second dot pixel coordinates.

3. The method as described in claim 2, characterized in that, The coordinate transformation relationship between the camera coordinate system and the pixel coordinate system of the shooting camera is used to determine the first pixel coordinate of each white dot in the pixel coordinate system, which is achieved according to the following formula: in, Here are the coordinates of the white dot in the image of the first calibration board. Here are the coordinates of the white dot in the camera coordinate system. The axial angle, It is a translation vector. Let be a rotation matrix. The pixel coordinates of the first dot. This refers to coordinate transformation relationships; The coordinate transformation relationship is as follows: in, Used to represent the coordinates of a point in the camera coordinate system. Used to represent the coordinates of a point in the pixel coordinate system Let x be the focal length in the x-direction. Let be the focal length in the y-direction. Let be the principal point in the x-direction. Let it be the principal point in the y-direction. The radial distortion coefficient is... denoted as the tangential distortion coefficient.

4. The method as described in claim 2, characterized in that, The step of determining the first calibration parameter based on the first dot pixel coordinates and the second dot pixel coordinates includes: The value of the first function is calculated based on the pixel coordinates of the first and second white dots. The parameter corresponding to the minimum value of the first function is used as the first calibration parameter; The first function is: in, Here, i represents the first calibration parameter, j represents the calibration board attitude index, and j represents the index of the white dot. The pixel coordinates of the first dot. These are the pixel coordinates of the second dot.

5. The method according to any one of claims 2-4, characterized in that, The determination of the second calibration parameters of the speckle structured light camera based on the first calibration parameters and the coordinates of each white dot in the second calibration plate image includes: Based on the center point coordinates of the speckle projector and the first calibration parameters, the third pixel coordinates of each white dot in the camera coordinate system are determined. Based on the second pixel coordinates of the white dots, determine the fourth pixel coordinates of each white dot in the camera coordinate system; The second calibration parameter is determined based on the pixel coordinates of the third and fourth dots.

6. The method as described in claim 5, characterized in that, The third pixel coordinates of each white dot in the camera coordinate system are determined based on the center point coordinates of the speckle projector and the first calibration parameters, according to the following formula: Where A, B, and D are the parameters of the reference plane equation. The coordinates of the center point are... Here are the coordinates of the white dot in the camera coordinate system. The coordinates of the intersection point of the white dot and the reference plane are given. This is a coordinate transformation relationship. The coordinates of the third circle are in pixels.

7. The method as described in claim 5, characterized in that, The step of determining the target calibration parameters of the speckle structured light camera based on the first calibration parameters and the second calibration parameters includes: The value of the objective function is calculated based on the pixel coordinates of the first, second, third, and fourth white dots. The parameter corresponding to the minimum value of the objective function is used as the target calibration parameter; The objective function is: in, Here are the target calibration parameters, where i is the calibration board attitude index and j is the white dot index. The pixel coordinates of the first dot. The pixel coordinates of the second circle are... The pixel coordinates of the third circle are... The coordinates of the fourth circle are in pixels.

8. The method as described in claim 1, characterized in that, The coordinate mapping relationship is as follows: ， ， ， in, For virtual corresponding coordinates, Distance to a virtual reference plane This is a virtual rotation matrix, used to represent the pose of the virtual camera coordinate system relative to the camera coordinate system. These are the coordinates of a point in the virtual camera coordinate system. The virtual focal length in the x-direction. Let be the virtual focal length in the y-direction. Let be the virtual principal point in the y-direction. The coordinates are the center point coordinates of the speckle projector.

9. The method as described in claim 8, characterized in that, The speckle structured light camera determines the depth of a pixel using the following formula: in, The depth of the pixels in the captured image. The distance to the virtual reference plane. The coordinates of the center point of the speckle projector are: Let d be the virtual focal length in the x-direction, and d be the parallax between the pixels in the captured image and the pixels in the virtual reference plane image.

Images