Dual-target positioning method, device and storage medium for wearable device

Abstract

The present invention provides a binocular calibration method, device and storage medium for a wearable device, wherein the method comprises: moving the wearable device based on a calibration plate, and two cameras of the wearable device taking multiple photos of the calibration plate from different viewing angles and positions; calibrating the internal parameters of each camera through a plane homography relationship, the internal parameters including the horizontal coordinate and vertical coordinate of the optical center of the camera, the focal length of the camera in the horizontal and vertical coordinate directions, and distortion parameters; calibrating the external parameters between the two cameras according to the internal parameters of each camera and the positional relationship of the calibration plate relative to the two cameras of the wearable device, the external parameters being the rotation matrix and translation vector of the right camera relative to the left camera; and performing binocular correction on the two cameras of the wearable device to obtain a stereo correction matrix. The present invention shortens the debugging time, simplifies the calibration process, improves the calibration efficiency, and is conducive to large-scale mass production of wearable devices.

Description

Background Art

[0002] Smart wearable devices are increasingly being used. In related technologies, the hardware characteristics of wearable devices, such as smart glasses, include but are not limited to the focal length, optical center position, and distortion coefficient of the binocular camera of smart glasses. Currently, the calibration method for wearable devices is manual calibration, which has a complex debugging process and a long debugging time, resulting in the inability to mass-produce wearable devices and a low degree of intelligence. The existing calibration method is manual calibration, which takes a long time to debug, is cumbersome, and inefficient.

[0003] Therefore, the present invention provides a dual-target positioning method, device and storage medium for a wearable device. Summary of the invention

[0004] In response to the problems in the prior art, the purpose of the present invention is to provide a binocular calibration method, device and storage medium for wearable devices, overcome the difficulties of the prior art, realize binocular automatic calibration of wearable devices, shorten the debugging time, simplify the calibration process, improve the calibration efficiency, and contribute to the large-scale mass production of wearable devices.

[0005] An embodiment of the present invention provides a dual-target positioning method for a wearable device, comprising the following steps:

[0006] S100, moving the wearable device based on the calibration plate, and using two cameras of the wearable device to take multiple photos of the calibration plate from different viewing angles and positions;

[0007] S200, calibrating internal parameters of each camera according to the homography relationship between the calibration plate and the camera imaging plane, the internal parameters including the horizontal coordinate and the vertical coordinate of the optical center of the camera, the focal length of the camera in the horizontal and vertical coordinate directions, and distortion parameters;

[0008] S300, calibrating external parameters between the cameras according to internal parameters of each camera and the positional relationship of the calibration plate relative to the two cameras of the wearable device, wherein the external parameters are a rotation matrix and a translation vector of the right camera relative to the left camera;

[0009] S400: Perform binocular correction on two cameras of the wearable device to obtain a stereo correction matrix, and equate the two cameras of the wearable device to two cameras with the same internal parameters and aligned in the horizontal direction.

[0010] Preferably, the step S200 includes the following steps:

[0011] S201, assume a spatial point M (X, Y, Z) in a world coordinate system, establish a camera coordinate system in each of the photos obtained by the camera, and obtain pixel coordinates m (u, v) of the spatial point M in the camera coordinate system;

[0012] S202, respectively establish homogeneous coordinates of the spatial point M (X, Y, Z) and the pixel coordinate m (u, v): and get

[0013]

[0014] Among them, A is the camera intrinsic parameter matrix with 3 rows and 3 columns, s is the scaling parameter, (u0, v0) is the coordinate of the projection point of the camera optical center in the pixel coordinate system, and f u and f v is the focal length of the camera in the x and y directions, R is the column vector of the camera's rotation matrix [r1 r2 r3], t is the translation vector of 3 rows and 1 column, let Z = 0 in the spatial point M (X, Y, Z), and we get

[0015]

[0016] S203, let M = (X, Y) T Reduce it to a two-dimensional vector, then we have

[0017] get

[0018]

[0019] Among them, the homography matrix H = A[r1, r2, t],

[0020] S204. Let H = [h1, h2, h3], and r1 and r2 are orthogonal, so we get

[0021]

[0022]

[0023] Among them, A -T A -1 is the projection matrix of the absolute quadratic curve in the image coordinate system;

[0024] S205, obtain the symmetric matrix B:

[0025]

[0026] S206, let a 6-dimensional vector b = [B 11 B 12 B 22 B 13 B 23 B 33 ] -T , take the i-th column vector of H matrix as: h i =[h i1,h i2 ,h i3 ] T , take the jth column vector of H matrix as h j , we can get:

[0027]

[0028] v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ] T ;

[0029] S207, obtain

[0030] S208, Order Then V*b=0;

[0031] Wherein, vector b is a 6-dimensional vector. When V is a full-rank matrix, the only solution for vector b is obtained;

[0032] S209, calculating the ordinate v0 of the optical center of the camera by using the symmetric matrix B;

[0033] The normalization coefficient λ of the camera is calculated by the symmetric matrix B and the ordinate v0 of the optical center of the camera;

[0034] The focal length f of the camera in the horizontal and vertical directions is calculated by the camera normalization coefficient λ and the symmetric matrix B. u and f v ;

[0035] The symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions are u and f v , the camera normalization coefficient λ is calculated to obtain the orthogonality coefficient γ describing the pixel;

[0036] Described by the orthogonality coefficient γ of the pixel, the symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions respectively u and f v, the vertical coordinate v0 of the optical center of the camera, and the camera normalization coefficient λ are used to calculate the horizontal coordinate u0 of the optical center of the camera.

[0037] Preferably, the step S209 further includes the following steps:

[0038] S210, obtaining a second formula group:

[0039]

[0040] S211, set m ij is the coordinate value of the jth detected corner point on the i-th image; M j is the three-dimensional coordinate of the jth corner point in the world coordinate system determined by the calibration plate; M j The coordinate values ​​actually projected onto the camera image can be used to obtain the objective function:

[0041]

[0042] Then obtain the calibration result;

[0043] S212, after obtaining the internal parameters of the camera, the distortion of the lens must be calculated, and the coordinates of the actual distorted point in the pixel coordinate system are obtained by calculating the coordinates of the ideal undistorted point in the pixel coordinate system, the coordinates of the ideal undistorted point in the image coordinate system and the distortion parameters.

[0044] Preferably, in step S211, an optimal iterative algorithm is used to obtain an optimal calibration result.

[0045] Preferably, the step S300 includes the following steps:

[0046] S301, suppose there is a point P in the space, its coordinates in the world coordinate system are Pw, and its coordinates in the left and right camera coordinate systems of the wearable device can be expressed as:

[0047] P l =R l P w +T' l

[0048] P r =R r P w +T' r

[0049] Among them, P r =RP l +T'

[0050] S302: The rotation matrix R of the relative calibration object obtained by the right camera through single-target calibration rThe rotation matrix R of the relative calibration object obtained by single-target calibration with the left camera l The rotation matrix R between the left and right cameras is calculated by the inverse matrix of

[0051] The translation vector T' relative to the calibration object obtained by the right camera through single-target calibration r , the translation vector T' of the left camera relative to the calibration object obtained by single-target calibration l The translation vector T' between the left and right cameras is calculated by using the rotation matrix R between the left and right cameras;

[0052] Among them, the left and right cameras perform single-target positioning respectively.

[0053] Preferably, the step S400 includes the following steps:

[0054] S401, setting the intersection of the line connecting the origins of the two camera coordinate systems in space and the image plane as the pole;

[0055] S402, set a rotation matrix R rect Such that the pole is located at an infinite distance from the wearable device;

[0056]

[0057] S403, obtain T' is the translation vector between the left and right cameras calculated by the rotation matrix R between the left and right cameras, T'=[T' x T' y T' z ] T ;

[0058] S404, let e2 and e1 be orthogonal, select the cross product of the main optical axis direction (0, 0, 1) and e1, and obtain

[0059] S405, let e3 be orthogonal to e1 and e2, then we can get: e3 = e1 × e2;

[0060] S406, R rect Multiply it left to the matrix of the left and right camera coordinate systems after R decomposition to obtain the stereo correction matrix.

[0061] Preferably, the step S406 further includes the following steps:

[0062] S407, obtain the output matrix Q of the stereo correction matrix, and realize the reprojection matrix Q of the conversion between the world coordinate system {world} and the pixel coordinate system {pixel}

[0063] S408, obtain

[0064] Where d represents the parallax, the three-dimensional coordinates are (X / W, Y / W, Z / W), and cx and cx′ represent the optical centers of the left and right images respectively;

[0065] S409, let cx'=cx, then the fourth row and fourth column element of Q is 0, and the three-dimensional coordinates of the spatial object can be expressed as

[0066] (x,y,z)=(X / W,Y / W,Z / W).

[0067] Preferably, a checkerboard pattern is provided on the surface of the calibration plate, and the intersections between the checkerboards are corner points.

[0068] Preferably, a directional checkerboard pattern is provided on the surface of the calibration plate, a unique asymmetric pattern is provided in each grid, the intersections between the checkerboard grids are corner points, and each corner point has a unique code.

[0069] Preferably, the following steps are also included: S500, determining the calibration accuracy to determine whether the first determination condition and the second determination condition are simultaneously met;

[0070] The first judgment condition is that the binocular module is aligned with the chessboard, the left and right images are corrected using the calibrated parameters, and the chessboard corner points are detected in the corrected images. If the difference in the ordinate of the matching point is less than a preset first threshold value;

[0071] The second judgment condition is to use the detected matching points to restore the spatial coordinates of the corner points using the disparity map, if the difference between the distance between the spatial coordinates of the adjacent empty chessboard corner points and the actual distance between the chessboard corner points is less than a second preset threshold.

[0072] An embodiment of the present invention further provides a dual-target positioning device for a wearable device, comprising:

[0073] processor;

[0074] a memory storing executable instructions of the processor;

[0075] Wherein, the processor is configured to execute the steps of the above-mentioned dual-target positioning method for the wearable device by executing the executable instructions.

[0076] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, wherein the program, when executed, implements the steps of the dual-target positioning method for the wearable device.

[0077] The binocular calibration method, device and storage medium of the wearable device of the present invention can realize binocular automatic calibration of the wearable device, shorten the debugging time, simplify the calibration process, improve the calibration efficiency, and contribute to the large-scale mass production of wearable devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0079] Figure 1 is a flow chart of a dual-target positioning method for a wearable device of the present invention;

[0080] Figure 2 is a schematic diagram of implementing the dual-target calibration method for a wearable device of the present invention based on the first calibration board;

[0081] Figure 3 is a schematic diagram of implementing the dual-target calibration method for a wearable device of the present invention based on the second calibration plate;

[0082] Figure 4 is a schematic diagram of a binocular camera before stereo calibration in a binocular positioning method for a wearable device of the present invention;

[0083] Figure 5 is a schematic diagram of a binocular camera after stereo calibration in a binocular positioning method for a wearable device of the present invention;

[0084] Figure 6 is a histogram of the difference between the y coordinates of the corner points of the left and right images after correction in the binocular positioning method of the wearable device of the present invention;

[0085] Figure 7 is a histogram of average spacings of corner points restored from depth information after correction in the binocular localization method for a wearable device of the present invention;

[0086] Figure 8 is a schematic structural diagram of a dual-target positioning device for a wearable device of the present invention; and

[0087] Fig. 9 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0088] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted.

[0089] Figure 1 FIG. 1 is a flow chart of a dual-target positioning method using a wearable device of the present invention. Figure 1 As shown, the dual-target positioning method of the wearable device of the present invention comprises the following steps:

[0090] S100, moving the wearable device based on the calibration plate, and using two cameras of the wearable device to take multiple photos of the calibration plate from different viewing angles and positions;

[0091] S200, calibrating internal parameters of each camera according to the homography relationship between the calibration plate and the camera imaging plane, the internal parameters including the horizontal coordinate and the vertical coordinate of the optical center of the camera, the focal length of the camera in the horizontal and vertical coordinate directions, and distortion parameters;

[0092] S300, calibrating external parameters between the cameras according to internal parameters of each camera and the positional relationship of the calibration plate relative to the two cameras of the wearable device, wherein the external parameters are a rotation matrix and a translation vector of the right camera relative to the left camera;

[0093] S400: Perform binocular correction on two cameras of the wearable device to obtain a stereo correction matrix, and convert the two cameras of the wearable device into two cameras with the same internal parameters and aligned in the horizontal direction.

[0094] The present invention can realize binocular automatic calibration of wearable devices, shorten the debugging time, simplify the calibration process, improve the calibration efficiency, and contribute to the large-scale mass production of wearable devices.

[0095] In a preferred embodiment, step S200 includes the following steps:

[0096] S201, assume a spatial point M (X, Y, Z) in a world coordinate system, establish a camera coordinate system in each of the photos obtained by the camera, and obtain pixel coordinates m (u, v) of the spatial point M in the camera coordinate system;

[0097] S202, respectively establish homogeneous coordinates of the spatial point M (X, Y, Z) and the pixel coordinate m (u, v): and get

[0098]

[0099] Among them, A is the camera intrinsic parameter matrix with 3 rows and 3 columns, s is the scaling parameter, (u0, v0) is the coordinate of the projection point of the camera optical center in the pixel coordinate system, and f u and f v is the focal length of the camera in the x and y directions, R is the column vector of the camera's rotation matrix [r1 r2 r3], t is the translation vector of 3 rows and 1 column, let Z = 0 in the spatial point M (X, Y, Z), and we get

[0100]

[0101] S203, let M = (X, Y)T Reduce it to a two-dimensional vector, then we have

[0102] get

[0103]

[0104] Among them, the homography matrix H = A[r1, r2, t],

[0105] S204. Let H = [h1, h2, h3], and r1 and r2 are orthogonal, so we get

[0106]

[0107]

[0108] Among them, A -T A -1 is the projection matrix of the absolute quadratic curve in the image coordinate system;

[0109] S205, obtain the symmetric matrix B:

[0110]

[0111] S206, let a 6-dimensional vector b = [B 11 B 12 B 22 B 13 B 23 B 33 ] -T , take the i-th column vector of H matrix as: h i =[h i1 ,h i2 ,h i3 ] T , take the jth column vector of H matrix as h j , we can get:

[0112]

[0113] v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 hj3 h i3 h j3 ] T ;

[0114] S207, obtain

[0115] S208, Order Then V*b=0;

[0116] Wherein, vector b is a 6-dimensional vector. When V is a full-rank matrix, the only solution for vector b is obtained;

[0117] S209, calculating the ordinate v0 of the optical center of the camera by using the symmetric matrix B;

[0118] The normalization coefficient λ of the camera is calculated by the symmetric matrix B and the ordinate v0 of the optical center of the camera;

[0119] The focal length f of the camera in the horizontal and vertical directions is calculated by the camera normalization coefficient λ and the symmetric matrix B. u and f v ;

[0120] The symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions are u and f v , the camera normalization coefficient λ is calculated to obtain the orthogonality coefficient γ describing the pixel;

[0121] Described by the orthogonality coefficient γ of the pixel, the symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions respectively u and f v , the vertical coordinate v0 of the optical center of the camera, and the camera normalization coefficient λ are used to calculate the horizontal coordinate u0 of the optical center of the camera.

[0122] In a preferred embodiment, step S209 further includes the following steps:

[0123] S210, obtaining a second formula group:

[0124]

[0125] S211, set m ij is the coordinate value of the jth detected corner point on the i-th image; M j is the three-dimensional coordinate of the jth corner point in the world coordinate system determined by the calibration plate; M j The coordinate values ​​actually projected onto the camera image can be used to obtain the objective function:

[0126]

[0127] Then obtain the calibration result;

[0128] S212, after obtaining the internal parameters of the camera, the distortion of the lens must be calculated, and the coordinates of the actual distorted point in the pixel coordinate system are obtained by calculating the coordinates of the ideal undistorted point in the pixel coordinate system, the coordinates of the ideal undistorted point in the image coordinate system and the distortion parameters.

[0129] In a preferred solution, an optimal iterative algorithm is used in step S211 to obtain an optimal calibration result.

[0130] In a preferred embodiment, step S300 includes the following steps:

[0131] S301, suppose there is a point P in the space, its coordinates in the world coordinate system are Pw, and its coordinates in the left and right camera coordinate systems of the wearable device can be expressed as:

[0132] P l =R l P w +T' l

[0133] P r =R r P w +T' r

[0134] Among them, P r =RP l +T'

[0135] S302: The rotation matrix R of the relative calibration object obtained by the right camera through single-target calibration r The rotation matrix R of the relative calibration object obtained by single-target calibration with the left camera l The rotation matrix R between the left and right cameras is calculated by the inverse matrix of

[0136] The translation vector T' relative to the calibration object obtained by the right camera through single-target calibration r , the translation vector T' of the left camera relative to the calibration object obtained by single-target calibration l The translation vector T' between the left and right cameras is calculated by using the rotation matrix R between the left and right cameras;

[0137] Among them, the left and right cameras perform single-target positioning respectively.

[0138] In a preferred embodiment, step S400 includes the following steps:

[0139] S401, setting the intersection of the line connecting the origins of the two camera coordinate systems in space and the image plane as the pole;

[0140] S402, set a rotation matrix R rect Such that the pole is located at an infinite distance from the wearable device;

[0141]

[0142] S403, obtain T' is the translation vector between the left and right cameras calculated by the rotation matrix R between the left and right cameras, T'=[T' x T' y T' z ] T ;

[0143] S404, let e2 and e1 be orthogonal, select the cross product of the main optical axis direction (0, 0, 1) and e1, and obtain

[0144] S405, let e3 be orthogonal to e1 and e2, then we can get: e3 = e1 × e2;

[0145] S406, R rect Multiply it left to the matrix of the left and right camera coordinate systems after R decomposition to obtain the stereo correction matrix.

[0146] In a preferred embodiment, the step S406 further includes the following steps:

[0147] S407 , obtaining an output matrix Q of the stereo correction matrix, and realizing the reprojection matrix Q for the conversion between the world coordinate system {world} and the pixel coordinate system {pixel}.

[0148] S408, obtain

[0149] Where d represents the parallax, the three-dimensional coordinates are (X / W, Y / W, Z / W), and cx and cx′ represent the optical centers of the left and right images respectively;

[0150] S409, let cx'=cx, then the fourth row and fourth column element of Q is 0, and the three-dimensional coordinates of the spatial object can be expressed as

[0151] (x,y,z)=(X / W,Y / W,Z / W).

[0152] In a preferred embodiment, a checkerboard pattern is provided on the surface of the calibration plate, and the intersection points between the checkerboards are corner points.

[0153] In a preferred embodiment, a directional checkerboard pattern is provided on the surface of the calibration plate, a unique asymmetric pattern is provided in each grid, the intersections between the checkerboard grids are corner points, and each corner point has a unique code.

[0154] In a preferred embodiment, the following steps are also included: S500, determining the calibration accuracy to determine whether the first determination condition and the second determination condition are simultaneously met;

[0155] The first judgment condition is that the binocular module is aligned with the chessboard, the left and right images are corrected using the calibrated parameters, and the chessboard corner points are detected in the corrected images. If the difference in the ordinate of the matching point is less than the first preset threshold;

[0156] The second judgment condition is to use the detected matching points to restore the spatial coordinates of the corner points using the disparity map, if the difference between the distance between the spatial coordinates of the adjacent empty chessboard corner points and the actual distance between the chessboard corner points is less than a second preset threshold.

[0157] refer to Figure 2 As shown, the present invention uses a robotic arm 4 to fix a calibration board 2 with a black and white chessboard 21 of required specifications, and a computer 3 is connected to at least one pair of glasses, which is placed flat at a certain distance from the chessboard, and connected to the computer 3 to ensure that the chessboard is within the picture of the glasses seen from the computer 3. The robotic arm will move to several preset positions, and each time it moves to a position, the program will control the left camera 12 and the right camera 11 on both sides of the smart glasses 1 (wearable device) to take a picture respectively. At the same time, the background will analyze the captured pictures. After all preset positions are moved, the calibration algorithm is used to complete the calibration calculation. If it is within the pre-configured threshold range, the calibration is passed, otherwise the calibration fails and needs to be retried. The specific method of calibration is as follows:

[0158] The calibration board in the present invention has a checkerboard on its surface. The checkerboard can be a calibration object with known geometric dimensions required for camera calibration. We use a checkerboard, which is usually composed of known side lengths, number of rows, number of columns, and alternating black and white.

[0159] The calibration plate in the present invention can be as follows Figure 2 The most common checkerboard pattern in the image processing is easy to make and has fast detection. However, the disadvantage is that the change of the checkerboard position cannot be too large, otherwise the order of the detected corner points will be wrong, and if the checkerboard is blocked, the detection will fail.

[0160] Alternatively, the calibration page in the present invention can be as follows Figure 3 Another checkerboard pattern in the image, each corner point has a unique code, so no matter how the checkerboard rotates, the corner points can always be detected in the correct order, and the corner point position can be correctly detected even if the checkerboard is blocked. However, it is more difficult to make a checkerboard and the detection algorithm is more complicated.

[0161] In the present invention, the camera parameters refer to the focal length, optical center position, distortion coefficient, etc. of the camera itself. Aim the camera at the chessboard, move the chessboard as far as possible while ensuring that the corners can be detected, take 15-20 sets of images and run the calibration program, at which time the parameters of the left and right cameras will be calculated.

[0162] During calibration, a calibration plate with dot matrix distribution is used as a reference, and the camera is used to capture images of the calibration plate at different viewing angles and postures. The positions of the corner points of the dots are extracted from the images, and then the maximum likelihood estimation method is used to restore the internal and external parameters of the camera through the obtained corner points.

[0163] Let the space point M in the world coordinate system = (X, Y, Z) T , the projection point on the imaging plane is m = (u, v) T . Their homogeneous coordinates are: and get:

[0164]

[0165] Where A is the intrinsic parameter matrix of the camera, s is the scaling parameter, u0, v0 are the horizontal and vertical coordinates of the camera optical center, f u , f v is the focal length of the camera in the horizontal and vertical coordinate directions. For ease of calculation, let the corner point of the calibration plate M = [X, Y, Z] T If Z=0, then:

[0166]

[0167] Where r1, r2, r3 are 3×1 column vectors, [r1 r2 r3] is the rotation matrix of the camera, and M = (X, Y) T Reduce it to a two-dimensional vector. Then we have At this time, formula (2-15) is simplified to:

[0168]

[0169] In the formula, H = A[r1, r2, t], which is called the homography matrix. It is used to characterize the mapping relationship between the corner points extracted on the calibration plate and the corresponding pixel points in the image coordinate system. Let H = [h1, h2, h3], and r1 and r2 are orthogonal, we can get:

[0170]

[0171] In the formula, A -T A -1 is the projection matrix of the absolute quadratic curve in the image coordinate system.

[0172] According to the properties of the absolute quadratic curve, let

[0173]

[0174] This matrix is ​​a symmetric matrix. Define a 6-dimensional vector b = [B 11 B 12 B 22 B 13 B 23 B 33 ] -T . Take the i-th column vector of H matrix as: h i =[h i1 ,h i2 ,h i3 ] T , take the jth column vector of H matrix as h j , we can get:

[0175]

[0176] In the formula,

[0177] v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ] T ;

[0178] , and then formula (2-19) can be obtained:

[0179]

[0180] make The above formula can be rewritten as: V b =0

[0181] In the formula, vector b is a 6-dimensional vector. When the number of acquired images is large enough, V in the above formula is a 6×6 matrix. If V is a full-rank matrix (usually n is large enough to make V columns full-rank), the unique solution of b can be found. The internal parameters of the camera can be further deduced through formula (2-21).

[0182]

[0183] Among them, u0, v0 are the horizontal and vertical coordinates of the optical center of the camera, and the normalization coefficient λ of the camera is calculated by the symmetric matrix B and the vertical coordinate v0 of the optical center of the camera;

[0184] The focal length f of the camera in the horizontal and vertical directions is calculated by the camera normalization coefficient λ and the symmetric matrix B. u and f v ;

[0185] The symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions are u and f v , the camera normalization coefficient λ is calculated to obtain the orthogonality coefficient γ describing the pixel;

[0186] Described by the orthogonality coefficient γ of the pixel, the symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions respectively u and f v , the vertical coordinate v0 of the optical center of the camera, and the camera normalization coefficient λ are used to calculate the horizontal coordinate u0 of the optical center of the camera.

[0187] Through formula (2-22), we can solve the internal parameter A of the camera. Then we can easily get the external parameter as follows:

[0188]

[0189] In order to obtain more accurate parameters, the maximum likelihood estimation method is often used. ij is the coordinate value of the jth detected corner point on the i-th image; M j is the three-dimensional coordinate of the jth corner point in the world coordinate system determined by the calibration plate; M j The coordinate value actually projected onto the camera image. Then the objective function can be obtained:

[0190]

[0191] Then the optimal iterative algorithm (Levenberg-Marquqrat algorithm) is used to obtain the optimal calibration result.

[0192] After calculating the internal parameters of the camera, the distortion of the lens must be calculated, and its expression is

[0193]

[0194]

[0195] Where (u, v) is the ideal undistorted pixel coordinate, is the actual pixel coordinate after distortion; (u0, v0) represents the optical center, and k1 and k2 are the distortion parameters of the first two orders.

[0196] Calibrate external parameters between cameras

[0197] After calibrating the internal parameters of the left and right binocular cameras, the positional relationship between the left and right cameras can be calculated based on the calculated position of the chessboard relative to the left and right cameras. The internal parameters calculated in the previous step are optimized using the reprojection relationship. The optimization goal is to minimize the reprojection error.

[0198] The main difference between binocular camera calibration and monocular camera calibration is that binocular camera needs to calibrate the relative relationship between the left and right camera coordinate systems. The rotation matrix R and translation matrix T' are usually used to describe the relative relationship between the left and right camera coordinate systems. Specifically, the coordinates of the left camera are converted to the coordinates of the right camera. Assume that there is a point P in space, whose coordinates in the world coordinate system are Pw, and its coordinates in the left and right camera coordinate systems can be expressed as:

[0199]

[0200] Among them, P r =RP l +T'

[0201] Combining the above formula, we can deduce

[0202]

[0203] T'=T' r -RT' l

[0204] Among them, R l , T' l is the rotation matrix and translation vector of the left camera relative to the calibration object obtained by single-target calibration, R r , T' r The rotation matrix and translation vector of the right camera relative to the calibration object obtained by single-target calibration

[0205] The left and right cameras are respectively positioned on a single target, and R l , T' l , R r , T' r , by substituting into the above formula, we can find the rotation matrix R and translation vector T' between the left and right cameras.

[0206] The obtained R and T' are the results of stereo calibration.

[0207] Both monocular and binocular cameras need to be calibrated. Stereo cameras have more parameters to calibrate than monocular cameras: R and T'. These parameters mainly describe the relative position relationship between the two cameras. These parameters are very useful in stereo correction and alignment geometry.

[0208] Binocular correction, the ultimate goal of binocular positioning is to make the left and right cameras equivalent to two cameras with completely consistent internal parameters and completely aligned in the horizontal direction. The process of achieving this goal is called binocular correction.

[0209] The main task of the binocular camera system is to measure distance, and the formula for calculating distance by parallax is derived when the binocular system is in an ideal condition. However, in the actual binocular stereo vision system, there are no two camera image planes that are completely coplanar and aligned. So we need to perform stereo correction. The purpose of stereo correction is to correct the two images that are not actually coplanar and aligned to coplanar alignment. (Coplanar alignment: the image planes of the two cameras are on the same plane, and when the same point is projected onto the image planes of the two cameras, it should be on the same row of the two pixel coordinate systems), correcting the actual binocular system to an ideal binocular system.

[0210] An ideal binocular system means that the image planes 11 and 12 of the two cameras are parallel, the optical axis and the image plane are perpendicular, and the pole is at an infinite distance. At this time, the level line corresponding to the point (x0, y0) is y=y0.

[0211] refer to Figure 4 and 5 During the correction process, both image planes 11 and 12 are rotated by half R, which can minimize the reprojection distortion. At this time, the two camera image planes are coplanar (the optical axes are also parallel after distortion correction), but not aligned.

[0212] The pole is the intersection of the line connecting the origins of the two camera coordinate systems and the image plane. To make the pole at infinity (i.e., line alignment), the image planes 13 and 14 of the two cameras must be adjusted to be parallel to the line connecting the origins of the two camera coordinate systems.

[0213] A rotation matrix Rrect can be calculated so that the poles are at infinity:

[0214]

[0215] Since the image plane is ultimately parallel to the line connecting the origin of the camera coordinate system, we have

[0216]

[0217] where T' = [T' x T' y T' z ] T, e2 and e1 are orthogonal, select the cross product of the principal optical axis direction (0, 0, 1) and e1;

[0218]

[0219] Let e3 be orthogonal to e1 and e2, we can get:

[0220] e3=e1×e2

[0221] The final stereo correction matrix can be obtained by multiplying Rrect by the matrix of R decomposition acting on the left and right camera coordinate systems.

[0222] Obtain the coordinate information of spatial objects from the calibrated binocular camera

[0223] The reprojection matrix Q realizes the conversion between the world coordinate system {world} and the pixel coordinate system {pixel}. The details are as follows:

[0224]

[0225] but

[0226]

[0227] Where d represents the parallax, the three-dimensional coordinates are (X / W, Y / W, Z / W), cx and cx' represent the optical centers of the left and right images respectively, and can usually be set to cx'=cx. Then the fourth row and fourth column element of Q is 0, and the three-dimensional coordinates of the spatial object can be expressed as

[0228]

[0229] The calibration accuracy determination process is as follows:

[0230] Place a black and white chessboard of known size in front of the binocular camera, and use the left and right cameras to shoot the chessboard at the same time, and detect the coordinates of the corner points of the chessboard on the left and right cameras respectively. First, restore the spatial coordinates of the corner points of the chessboard in the left camera coordinate system, and then use (Formula 2-23) to transfer the spatial coordinates in the left camera to the right camera coordinate system, and reproject them into the right camera image. Compare the reprojected corner point coordinates with the original corner point coordinates detected in the right image, and calculate their reprojection error. The reprojection error is used as a criterion for the quality of the calibration result. If the error is less than the preset threshold, the calibration is considered to be passed, otherwise it is not passed.

[0231] refer to Figure 6 , Figure 1 The difference histogram of the y coordinates of the left and right image corner points is mainly distributed around 0 pixels, and the calibration result is good.

[0232] refer to Figure 7The average spacing of the corner points recovered from the depth information has a peak value of about 18.3mm, which is very close to the actual length of 18.4mm. The calibration result is good. If the calibration passes, the calibration parameters will be generated, and the intermediate process data of the calibration will be backed up. At the same time, the calibration parameters will be burned into the glasses and backed up to the server to obtain a unique glasses serial number. If the burning or backup fails, you need to try again. You can recalibrate, or click the "Burn" button to burn and back up again. After the burning and backup are completed, the glasses serial number will be printed out in the preset label format through the label printer program so that it can be pasted on the outer packaging and product maintenance card, etc. If the printing fails, you can click "ReadSN" to read the serial number burned in the glasses, and then click the "Print" button to print the label paper. If any of the intermediate processes fails, you can continue from an intermediate process.

[0233] The dual-target positioning method of the wearable device of the present invention solves the dual-target positioning problem and improves productivity. It reduces the human involvement of the middleman and reduces the occurrence of human operation errors.

[0234] The embodiment of the present invention further provides a dual-target positioning device for a wearable device, comprising a processor and a memory in which executable instructions of the processor are stored, wherein the processor is configured to execute the steps of the dual-target positioning method for the wearable device by executing the executable instructions.

[0235] As shown above, this embodiment can accurately reflect the actual displacement of the robot movement within a period of time, reduce measurement deviation, improve measurement accuracy, and has no cumulative error, and the positioning range can be infinitely expanded.

[0236] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0237] Figure 8 Schematic diagram of the structure of the dual-target positioning device of the wearable device of the present invention. Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0238] like Figure 8As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), etc.

[0239] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0240] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0241] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0242] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0243] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0244] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the dual-target positioning method of the wearable device are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned electronic prescription circulation processing method section of this specification.

[0245] As shown above, this embodiment can accurately reflect the actual displacement of the robot movement within a period of time, reduce measurement deviation, improve measurement accuracy, and has no cumulative error, and the positioning range can be infinitely expanded.

[0246] Fig. 9 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0247] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0248] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0249] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0250] In summary, the present invention provides a binocular calibration method, device and storage medium for wearable devices, which can realize binocular automatic calibration of wearable devices, shorten the debugging time, simplify the calibration process, improve the calibration efficiency, and contribute to the large-scale mass production of wearable devices.

[0251] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A dual-target positioning method for a wearable device, characterized in that: The following steps are involved: S100, moving the wearable device based on the calibration plate, and using two cameras of the wearable device to take multiple photos of the calibration plate from different viewing angles and positions; S200, calibrating internal parameters of each camera according to the homography relationship between the calibration plate and the camera imaging plane, the internal parameters including the horizontal coordinate and the vertical coordinate of the optical center of the camera, the focal length of the camera in the horizontal and vertical coordinate directions, and distortion parameters; S300, calibrating external parameters between the cameras according to internal parameters of each camera and the positional relationship of the calibration plate relative to the two cameras of the wearable device, wherein the external parameters are a rotation matrix and a translation vector of the right camera relative to the left camera; S400, performing binocular correction on two cameras of the wearable device to obtain a stereo correction matrix, and equating the two cameras of the wearable device to two cameras with the same internal parameters and aligned in the horizontal direction, wherein step S400 includes the following steps: S401, setting the intersection of the line connecting the origins of the two camera coordinate systems in space and the image plane as the pole; S402, set a rotation matrix R rect Such that the pole is located at an infinite distance from the wearable device; S403, obtain T' is the translation vector between the left and right cameras calculated by the rotation matrix R between the left and right cameras, T'=[T' x T' y T' z ] T ; S404, let e2 and e1 be orthogonal, select the cross product of the main optical axis direction (0, 0, 1) and e1, and obtain S405, let e3 be orthogonal to e1 and e2, then we can get: e3 = e1 × e2; S406, R rect Multiply it left to the matrix of the left and right camera coordinate systems after R decomposition to obtain the stereo correction matrix.

2. The dual-target positioning method for a wearable device as claimed in claim 1, characterized in that: The step S200 includes the following steps: S201, assume a spatial point M (X, Y, Z) in a world coordinate system, establish a camera coordinate system in each of the photos obtained by the camera, and obtain pixel coordinates m (u, v) of the spatial point M in the camera coordinate system; S202, respectively establish homogeneous coordinates of the spatial point M (X, Y, Z) and the pixel coordinate m (u, v): and get Among them, A is the camera intrinsic parameter matrix with 3 rows and 3 columns, s is the scaling parameter, (u0, v0) is the coordinate of the projection point of the camera optical center in the pixel coordinate system, and f u and f v is the focal length of the camera in the x and y directions, R is the column vector of the camera's rotation matrix [r1r2 r3], t is the translation vector of 3 rows and 1 column, let Z = 0 in the spatial point M (X, Y, Z), and we get S203, let M = (X, Y) T Reduce it to a two-dimensional vector, then we have get Among them, the homography matrix H = A[r1, r2, t], S204. Let H = [h1, h2, h3], and r1 and r2 are orthogonal, so we get Among them, A -T A -1 is the projection matrix of the absolute quadratic curve in the image coordinate system; S205, obtain the symmetric matrix B: S206, suppose a 6-dimensional vector b=[B 11 B 12 B 22 B 13 B 23 B 33 ] -T , take the i-th column vector of H matrix as: h i =[h i1 ,h i2 ,h i3 ] T , take the jth column vector of H matrix as h j , we can get: v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ] T ; S207, obtain S208, Order Then V*b=0; Wherein, vector b is a 6-dimensional vector. When V is a full-rank matrix, the only solution for vector b is obtained; S209, calculating the ordinate v0 of the optical center of the camera by using the symmetric matrix B; The normalization coefficient λ of the camera is calculated by the symmetric matrix B and the ordinate v0 of the optical center of the camera; The focal length f of the camera in the horizontal and vertical directions is calculated by the camera normalization coefficient λ and the symmetric matrix B. u and f v ; The symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions are u and f v , the camera normalization coefficient λ is calculated to obtain the orthogonality coefficient γ describing the pixel; Described by the orthogonality coefficient γ of the pixel, the symmetric matrix B and the focal length f of the camera in the horizontal and vertical directions respectively u and f v , the ordinate v0 of the optical center of the camera, and the normalization coefficient λ of the camera are used to calculate the abscissa u0 of the optical center of the camera, and the internal parameters of the camera are obtained based on the following formula:

3. The dual-target positioning method for a wearable device as claimed in claim 2, characterized in that: The step S209 further includes the following steps: S210, obtaining a second formula group: S211, set m ij is the coordinate value of the jth detected corner point on the i-th image; M j is the three-dimensional coordinate of the jth corner point in the world coordinate system determined by the calibration plate; M j The coordinate values ​​actually projected onto the camera image can be used to obtain the objective function: Then obtain the calibration result; S212, after obtaining the internal parameters of the camera, the distortion of the lens must be calculated, and the coordinates of the actual distorted point in the pixel coordinate system are obtained by calculating the coordinates of the ideal undistorted point in the pixel coordinate system, the coordinates of the ideal undistorted point in the image coordinate system and the distortion parameters.

4. The dual-target positioning method for a wearable device as claimed in claim 3, characterized in that: In step S211, an optimal iterative algorithm is used to obtain an optimal calibration result.

5. The dual-target positioning method for a wearable device as claimed in claim 1, characterized in that: The step S300 includes the following steps: S301, suppose there is a point P in the space, its coordinates in the world coordinate system are Pw, and its coordinates in the left and right camera coordinate systems of the wearable device can be expressed as: P l =R l P w +T' l P=RP+T' r r w r Among them, P r =RP l +T' S302: The rotation matrix R of the relative calibration object obtained by the right camera through single-target calibration r The rotation matrix R of the relative calibration object obtained by single-target calibration with the left camera l The rotation matrix R between the left and right cameras is calculated by the inverse matrix of The translation vector T' relative to the calibration object obtained by the right camera through single-target calibration r , the translation vector T' of the left camera relative to the calibration object obtained by single-target calibration l The translation vector T' between the left and right cameras is calculated by using the rotation matrix R between the left and right cameras; Among them, the left and right cameras perform single-target positioning respectively.

6. The dual-target positioning method for a wearable device as claimed in claim 1, characterized in that: The surface of the calibration plate is provided with a checkerboard pattern, and the intersections between the checkerboards are corner points.

7. The dual-target positioning method for a wearable device as claimed in claim 1, characterized in that: The surface of the calibration plate is provided with a directional checkerboard pattern, each grid is provided with a unique asymmetric pattern, the intersections between the checkerboard grids are corner points, and each corner point has a unique code.

8. The dual-target positioning method for a wearable device according to claim 6 or 7, characterized in that: The following steps are also included: S500, determining the calibration accuracy to determine whether the first determination condition and the second determination condition are simultaneously met; The first judgment condition is that the binocular module is aligned with the chessboard, the left and right images are corrected using the calibrated parameters, and the chessboard corner points are detected in the corrected images. If the difference in the ordinate of the matching point is less than a preset first threshold value; The second judgment condition is to use the detected matching points to restore the spatial coordinates of the corner points using the disparity map, if the difference between the distance between the spatial coordinates of the adjacent empty chessboard corner points and the actual distance between the chessboard corner points is less than a second preset threshold.

9. A dual-target positioning device for a wearable device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the dual-target positioning method for a wearable device according to any one of claims 1 to 8 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the dual-target positioning method for a wearable device described in any one of claims 1 to 8 are implemented.

Images