Camera extrinsic parameter calibration method for mixed display

Abstract

The invention discloses a camera external parameter calibration method for mixed display. The method comprises the following steps: S11, obtaining an internal parameter matrix of a camera through an internal parameter calibration method; s12, the external positioner is started; s13, screen led lamp beads are controlled, bright spots are regarded as feature mark points, M lamp beads are lightened, images are shot N times at different positions, and meanwhile output of the positioner at the moment is recorded; s14, listing a re-projection residual equation participating in coordinate system deviation according to the shot feature mark points by combining with the spatial pose of the locator; s15, the optimal external participation coordinate system deviation can be obtained through iteration by minimizing the re-projection residual error; compared with a traditional calibration mode, the method does not need to use a distance measuring instrument for measurement, meanwhile, the offset between the original point of the positioner and the screen coordinate system is considered, and the external parameters of the camera and the positioner and the offset between the original point of the positioner and the screen can be solved at the same time by sampling the synchronous positioning data of the multiple sets of cameras and the positioner.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic calibration, and in particular to a camera extrinsic calibration method for hybrid display. Background Art

[0002] In the fields of mixed display and virtual shooting, a key to achieving seamless interaction is to seamlessly stitch real captured images with virtual images. It is necessary to obtain the relative position and posture of the camera and the screen in 3D space to reversely calculate the captured images. Existing solutions include directly using the image information captured by the camera to calculate the camera posture, which is referred to as the direct method below, or installing an external high-speed locator, pre-calibrating the external parameters of the camera and locator, obtaining the locator data in real time, and solving the camera position and posture, which is referred to as the indirect method below.

[0003] Although the theoretical accuracy of the direct method meets the application requirements, in practical applications, calibration patterns or QR codes still need to be arranged during the operation phase, and the method is easily affected by the environment, with poor real-time and robustness. The indirect method meets the requirements of real-time and robustness, but the positioning accuracy is affected by the accuracy of the external parameter calibration. Traditional calibration methods generally use manual measurement or do not consider the offset between the locator origin and the screen coordinate system, resulting in large errors and the inability to achieve seamless stitching. To this end, we propose a camera external parameter calibration method for hybrid display. Summary of the invention

[0004] The object of the present invention is to provide a camera extrinsic calibration method for hybrid display to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: a camera extrinsic calibration method for hybrid display, which introduces a locator origin and a screen offset to remodel the camera, and uses nonlinear optimization to simultaneously solve the camera and locator extrinsic parameters, the locator origin and the screen offset;

[0006] The method comprises the following steps:

[0007] S11, obtaining the intrinsic parameter matrix of the camera by an intrinsic parameter calibration method;

[0008] S12, external positioner starts;

[0009] S13, control the screen LED lamp beads, treat the bright spots as feature marking points, light up M of them, and take images N times at different positions, and record the output of the locator at this moment;

[0010] S14, according to the photographed feature marker points, by combining with the spatial pose of the locator, the reprojection residual equation of the external participant coordinate system deviation is listed;

[0011] S15. By minimizing the reprojection residual, the optimal external participating coordinate system deviation can be iteratively obtained.

[0012] Preferably, the internal parameter matrix K in S11 is a 3-dimensional square matrix, defined as:

[0013]

[0014] f is the focal length of the camera, and u0 and v0 correspond to the offset of the camera center point.

[0015] Preferably, in S12, an external locator is started to keep the locator system synchronized with the camera system, and the spatial position and posture are recorded in real time during the shooting process.

[0016] Preferably, in S13, the characteristic marking points are evenly distributed in the camera field of view, the camera posture information is captured from multiple angles, the lower left corner of the screen is set as the origin of the screen coordinate system, and the coordinates of the light bead in row i and column j are obtained as follows:

[0017] P=(j*w led ,i*h led ,0);

[0018] w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are:

[0019] Points = {P0,P1,…P M};

[0020] At the same time, record the positioner output The corresponding picture I i .

[0021] Preferably, the marking method of the characteristic marking points photographed in S14 is:

[0022] Known feature points in screen coordinate system O sereen The coordinates of Points={P0,P1,…P M}, the pixel coordinates observed by the camera at time i are:

[0023] Preferably, the combination of the feature marker points and the spatial pose of the locator in S14 is:

[0024] The camera pose is set to Then for each point we have Since there is noise in the pixel observation, that is:

[0025]

[0026] Preferably, the way to list the reprojection residual equation of the external parameter and coordinate system deviation in S14 is as follows: Optimized residual equations are listed by observing multiple groups of feature points multiple times:

[0027]

[0028] where T loc2camera is the external parameter. Expand to obtain Then the residual equation is modified to:

[0029]

[0030] Preferably, in S15, the gradient descent method is used to find the optimal external parameter and coordinate system deviation. First, the Jacobian matrix J of the constraint equation is obtained, and then the step function can be obtained:

[0031] step = (J T J + uI) -1 J T f;

[0032] where T represents taking the transpose of the matrix, -1 represents taking the inverse of the matrix, u is the damping coefficient, and I is the identity matrix;

[0033] Set the threshold threshold to control the accuracy and the number of iterations;

[0034] Set the evaluation function rou = mean[f(X + ΔX) - f(X) / (J(X)ΔX)], where mean represents taking the average, and X represents the variable to be optimized, that is, the external parameter and coordinate system deviation.

[0035] Preferably, the subsequent iterative optimization steps are as follows:

[0036] S21. Give the initial external parameter and coordinate system deviation. Substitute into JTJ to obtain a square matrix, and take the maximum value on its diagonal element as the initial value of the confidence domain u = u0;

[0037] S22. Substitute into the calculation of rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2;

[0038] S23. Determine whether the step function needs to be updated based on whether the value of u is updated as above. The step function obtains step(X), and the updated external parameter and coordinate system deviation is X' = X + step(X);

[0039] S24. Substitute X' into the constraint equation and calculate the average to obtain the error of this iteration. If error > threshold, substitute X' into step 2 for recalculation. If error < threshold, break out of the loop, and the final X' is the deviation of the external reference from the coordinate system.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] Compared with the traditional calibration method, the method of the present invention does not require the use of a ranging instrument for measurement. At the same time, it takes into account the offset between the origin of the locator and the screen coordinate system. By sampling multiple groups of positioning data synchronized by the camera and the locator, the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen, can be solved simultaneously. Description of the Drawings

[0042] Figure 1 It is the image captured by the camera in S13 of the present invention. Detailed Embodiment

[0043] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] The present invention provides a technical solution: a method for calibrating the external parameters of a camera for hybrid display. The method re - models by introducing the offset between the origin of the locator and the screen, and uses non - linear optimization to simultaneously solve the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen.

[0045] The method includes the following steps:

[0046] S11. Obtain the internal parameter matrix of the camera through the method of internal parameter calibration.

[0047] S12. Start the external locator.

[0048] S13. Control the screen LED beads, consider the bright points as feature marker points, light up M of them, and take images N times at different positions, while recording the output of the locator at this moment.

[0049] S14. According to the captured feature marker points, list the reprojection residual equation of the deviation of the external reference from the coordinate system by combining with the spatial pose of the locator.

[0050] S15. By minimizing the reprojection residual, the optimal deviation of the external reference from the coordinate system can be iteratively obtained.

[0051] As a preferred implementation, the internal parameter matrix K in S11 is a 3-dimensional square matrix, defined as:

[0052]

[0053] f is the focal length of the camera (in this case, f is the focal length in pixels), and u0 and v0 correspond to the offset of the camera center point.

[0054] As a preferred implementation, in step S12, the external locator is started, and it is necessary to ensure that the locator system is synchronized with the camera system so that the spatial position and posture can be recorded in real time during the shooting process.

[0055] Among them, the data of the external locator is obtained through the network protocol and can eventually be converted into position information rotation and translation.

[0056] As a preferred implementation, in S13, the characteristic marking points are evenly distributed in the camera field of view so as to capture the camera's posture information from multiple angles. The lower left corner of the screen can be set as the origin of the screen coordinate system, and the coordinates of the light beads in row i and column j are obtained as follows:

[0057] P=(j*w led ,i*h led ,0);

[0058] w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are:

[0059] Points = {P0,P1,…P M};

[0060] At the same time, record the positioner output The corresponding picture I i .

[0061] As a preferred implementation, the marking method of the characteristic marking points photographed in S14 is:

[0062] Known feature points in screen coordinate system O screen The coordinates of Points={P0,P1,…P M}, the pixel coordinates observed by the camera at time i are:

[0063] As a preferred implementation, the combination of the feature marker points and the spatial pose of the locator in S14 is as follows:

[0064] The camera pose is set to Then for each point we have Since there is noise in the pixel observation, that is:

[0065]

[0066] As a preferred embodiment, the method of listing the reprojection residual equation of the external parameter and coordinate system deviation in S14 is as follows: List the optimization residual equation by observing multiple groups of feature points multiple times:

[0067]

[0068] where T loc2camera is the external parameter. Expand to obtain Then the residual equation is modified to:

[0069]

[0070] As a preferred embodiment, in S15, the gradient descent method is used to find the optimal external parameter and coordinate system deviation. First, find the Jacobian matrix J of the constraint equation, and then the step function can be obtained:

[0071] step = (J T J + uI) -1 J T f;

[0072] where T represents taking the transpose of the matrix, -1 represents taking the inverse of the matrix, u is the damping coefficient, and I is the identity matrix;

[0073] Set the threshold threshold to control the accuracy and the number of iterations;

[0074] Set the evaluation function rou = mean[f(X + ΔX) - f(X) / (J(X)ΔX)], where mean represents taking the average, and X represents the variable to be optimized, that is, the external parameter and coordinate system deviation.

[0075] As a preferred embodiment, the subsequent iterative optimization steps are as follows:

[0076] S21. Give the initial external parameter and coordinate system deviation, substitute it into JTJ to obtain a square matrix, and take the maximum value on its diagonal element as the initial value of the confidence domain u = u0;

[0077] S22. Substitute and calculate rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2;

[0078] S23. Determine whether to update the step function based on whether the value of u is updated as above. The step function obtains step(X), and the updated external parameter and coordinate system deviation is X' = X + step(X);

[0079] S24. Substitute X' into the constraint equation and calculate the average to obtain the error of this iteration. If error > threshold, substitute X' into step 2 for recalculation. If error < threshold, break out of the loop. The final X' is the deviation of the external participation coordinate system.

[0080] Please refer to Figure 1 , some QR codes can be seen. The size unit of these QR codes is the width of the lamp beads. Using these QR codes, feature points at the sub-millimeter level (lamp beads are generally less than 1 mm) can be extracted.

[0081] Compared with the traditional calibration method, this method does not require the use of a ranging instrument for measurement. At the same time, it takes into account the offset between the origin of the locator and the screen coordinate system. By sampling multiple groups of positioning data of the camera and the locator synchronized, the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen, can be solved simultaneously.

[0082] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0083] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for calibrating the external parameters of a camera for hybrid display, characterized in that: This method remodels by introducing the origin of the locator and the screen offset, and simultaneously solves the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen through non-linear optimization; This method includes the following steps: S11. Obtain the internal parameter matrix of the camera through the method of internal parameter calibration; S12. Start the external locator; S13. Control the screen LED beads, regard the bright points as feature marker points, light up M of them, and take images N times at different positions, and record the output of the locator at this moment; S14. According to the captured feature marker points, list the reprojection residual equation of the deviation between the external parameters and the coordinate system by combining with the spatial pose of the locator; S15. By minimizing the reprojection residual, the optimal deviation between the external parameters and the coordinate system can be iteratively obtained.

2. The camera extrinsic calibration method for hybrid display according to claim 1, characterized in that: In the S11, the internal parameter matrix K is a 3D square matrix, defined as: f is the camera focal length, and u0 and v0 correspond to the offsets of the camera center point.

3. The camera extrinsic calibration method for hybrid display according to claim 2, characterized in that: In the S12, start the external locator, keep the locator system synchronized with the camera system, and record the spatial position and attitude in real time during the shooting process.

4. The camera extrinsic calibration method for hybrid display according to claim 3, characterized in that: In the S13, the feature marker points are evenly distributed in the camera's field of view, capture the attitude information of the camera from multiple angles, set the lower left of the screen as the origin of the screen coordinate system, and obtain the coordinates of the illuminated LED beads in the i-th row and j-th column as: P=(j*w led ,i*h led ,0); w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are: Points={P0,P1,…P M }; At the same time, record the positioner output The corresponding picture I i .

5. The camera extrinsic calibration method for hybrid display according to claim 4, characterized in that: The marking method of the feature marker points captured in the S14 is: Known feature points in screen coordinate system O screen The coordinates of Points={P0,P1,…P M }, the pixel coordinates observed by the camera at time i are:

6. The camera extrinsic calibration method for hybrid display according to claim 5, characterized in that: The combination method of the feature marker points and the spatial pose of the locator in the S14 is: The camera pose is set to T i screen2camera , then for every point p i,j =KT i screen2camera P j , since there is noise in the pixel observation, that is:

7. The camera extrinsic calibration method for hybrid display according to claim 6, characterized in that: The method of listing the reprojection residual equation of the deviation between the external parameters and the coordinate system in the S14 is: list the optimization residual equation by observing multiple groups of feature points multiple times: Where T i screen2camera =T loc2camera T i screen2loc ,T loc2camera is an external reference, for T i screen2loc Expand and get T i screen2loc =T i world2loc T screen2worid , then the residual equation is modified as follows:

8. The camera extrinsic calibration method for hybrid display according to claim 7, characterized in that: In the S15, use the gradient descent method to find the optimal deviation between the external parameters and the coordinate system. First, find the Jacobian matrix J of the constraint equation, and then the step size function can be obtained: step=(J T J+uI) -1 J T f; Where T represents taking the transpose of the matrix, -1 represents taking the inverse of the matrix, u is the damping coefficient, and I is the identity matrix; Set the threshold threshold to control the accuracy and the number of iterations; Set the evaluation function rou = mean[f(X + ΔX) - f(X) / (J(X)ΔX)], where mean represents taking the average, X represents the variable to be optimized, that is, the deviation between the external parameters and the coordinate system.

9. The camera extrinsic calibration method for hybrid display according to claim 8, characterized in that: The subsequent iterative optimization steps are: S21. Give the initial deviation between the external parameters and the coordinate system, substitute it into JTJ to obtain a square matrix, and take the largest value on its diagonal element as the initial value of the confidence domain u = u0; S22. Substitute and calculate rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2; S23. Judge whether the step size function needs to be updated according to whether the value of u is updated as above. The step size function gets step(X), and the updated deviation between the external parameters and the coordinate system is X' = X + step(X); S24. Substitute X' into the constraint equation and calculate the average to obtain the error of this iteration error. If error > threshold, then substitute X' into step 2 to recalculate. If error < threshold, then jump out of the loop, and the final X' is the deviation between the external parameters and the coordinate system.

Images