A camera extrinsic calibration method for hybrid display

Abstract

The present invention discloses a camera extrinsic parameter calibration method for hybrid display, which comprises the following steps: S11, obtaining an intrinsic parameter matrix of a camera by an intrinsic parameter calibration method; S12, starting an external locator; S13, controlling LED lamp beads on a screen, treating bright spots as feature markers, lighting up M of them, capturing images N times at different positions, and recording the output of the locator at that moment; S14, listing a reprojection residual equation of the extrinsic parameter coordinate system deviation based on the captured feature markers and combining them with the spatial posture of the locator; S15, iteratively obtaining the optimal extrinsic parameter coordinate system deviation by minimizing the reprojection residual; compared with the traditional calibration method, the present invention does not require the use of a rangefinder for measurement, while taking into account the offset between the locator origin and the screen coordinate system, and sampling multiple sets of positioning data synchronized with the camera and the locator to simultaneously solve the camera and locator extrinsic parameters and the locator origin and screen offset.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic parameter calibration, and in particular to a camera extrinsic parameter calibration method for hybrid display. Background Art

[0002] In areas such as mixed display and virtual photography, a key to achieving seamless interaction is the seamless splicing of real-world images with virtual ones. This requires obtaining the relative position and posture of the camera and screen in 3D space to inversely calculate the captured images. Existing solutions include directly calculating the camera posture using the image information captured by the camera, hereinafter referred to as the direct method, or installing an external high-speed locator, pre-calibrating the external parameters of the camera and locator, obtaining the locator data in real time, and calculating the camera position and posture, hereinafter referred to as the indirect method.

[0003] While direct methods offer sufficient theoretical accuracy, they still require the placement of calibration patterns or QR codes during real-time operation. These methods are susceptible to environmental influences, resulting in poor real-time and robustness. While indirect methods offer both real-time and robustness, their positioning accuracy is affected by the accuracy of extrinsic calibration. Traditional calibration methods typically rely on manual measurement or fail to account for the offset between the locator's origin and the screen coordinate system, resulting in significant errors and the inability to achieve seamless stitching. To address this issue, we propose a camera extrinsic calibration method for hybrid displays. Summary of the Invention

[0004] The object of the present invention is to provide a camera extrinsic parameter calibration method for hybrid display to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: a camera extrinsic parameter calibration method for hybrid display, which remodels the camera by introducing the locator origin and screen offset, and uses nonlinear optimization to simultaneously solve the camera and locator extrinsic parameters, and the locator origin and screen offset;

[0006] The method comprises the following steps:

[0007] S11. Obtaining the camera's intrinsic parameter matrix through an intrinsic parameter calibration method;

[0008] S12, external positioner starts;

[0009] S13: Control the LEDs on the screen, treat the bright spots as feature markers, light up M of them, and take N images at different positions, while recording the output of the locator at that moment;

[0010] S14. Based on the captured feature markers, by combining them with the spatial pose of the locator, a reprojection residual equation of the external participant coordinate system deviation is listed;

[0011] S15. By minimizing the reprojection residual, the optimal external participating coordinate system deviation can be iteratively obtained.

[0012] Preferably, the internal parameter matrix K in S11 is a 3-dimensional square matrix, defined as:

[0013]

[0014] f is the focal length of the camera, and u0 and v0 correspond to the offset of the camera center point.

[0015] Preferably, in S12, an external locator is started, the locator system is kept synchronized with the camera system, and the spatial position and posture are recorded in real time during the shooting process.

[0016] Preferably, in S13, the characteristic marker points are evenly distributed in the camera field of view, the camera posture information is captured from multiple angles, and the lower left corner of the screen is set as the origin of the screen coordinate system. The coordinates of the light bead in row i and column j are obtained as follows:

[0017] P=(j*w led ,i*h led ,0);

[0018] w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are:

[0019] Points={P0,P1,…P M};

[0020] At the same time, record the locator output And the corresponding picture I i .

[0021] Preferably, the marking method of the characteristic marking points photographed in S14 is:

[0022] Known feature points in screen coordinate system O sereen The coordinates of Points={P0,P1,…P M}, the pixel coordinates observed by the camera at time i are:

[0023] Preferably, the combination of the feature markers and the spatial pose of the locator in S14 is:

[0024] The camera pose is set to Then for each point there is Since there is noise in the pixel observation, that is:

[0025]

[0026] Preferably, the method of listing the reprojection residual equation of the external participation coordinate system deviation in S14 is as follows: optimize the residual equation by observing multiple groups of feature points multiple times:

[0027]

[0028] where T loc2camera is the external parameter. Expand to obtain Then the residual equation is modified to:

[0029]

[0030] Preferably, in S15, the gradient descent method is used to find the optimal external participation coordinate system deviation. First, find the Jacobian matrix J of the constraint equation, and then the step function can be obtained:

[0031] step = (J T J + uI) -1 J T f;

[0032] where T represents the transpose of the matrix, -1 represents the inverse of the matrix, u is the damping coefficient, and I is the identity matrix;

[0033] Set the threshold threshold to control the accuracy and the number of iterations;

[0034] Set the evaluation function rou = mean[f(X + ΔX)-f(X) / (J(X)ΔX)], where mean represents taking the average, and X represents the variable to be optimized, that is, the external participation coordinate system deviation.

[0035] Preferably, the subsequent iterative optimization steps are as follows:

[0036] S21. Give the initial external participation coordinate system deviation, substitute it into JTJ to obtain a square matrix, and take the maximum value on its diagonal element as the initial value of the confidence domain u = u0;

[0037] S22. Substitute and calculate rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2;

[0038] S23. Determine whether to update the step function based on whether the value of u is updated above. The step function obtains step(X), and the updated external participation coordinate system deviation is X' = X + step(X);

[0039] S24. Substitute X' into the constraint equation and calculate the average to obtain the error of this iteration. If error > threshold, substitute X' into step 2 for recalculation. If error < threshold, break out of the loop, and the final X' is the deviation of the external reference from the coordinate system.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] Compared with the traditional calibration method, the method of the present invention does not require the use of a ranging instrument for measurement. At the same time, considering the offset between the origin of the locator and the screen coordinate system, by sampling multiple groups of positioning data synchronized by the camera and the locator, the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen, can be solved simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the image captured by the camera in S13 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] The present invention provides a technical solution: a method for calibrating the external parameters of a camera for mixed display. The method rebuilds the model by introducing the offset between the origin of the locator and the screen, and simultaneously solves the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen by using nonlinear optimization;

[0045] The method includes the following steps:

[0046] S11. Obtain the internal parameter matrix of the camera through the method of internal parameter calibration;

[0047] S12. Start the external locator;

[0048] S13. Control the screen LED beads, regard the bright points as feature marker points, light up M of them, and take images N times at different positions, while recording the output of the locator at this moment;

[0049] S14. According to the captured feature marker points, list the reprojection residual equation of the deviation of the external reference from the coordinate system by combining with the spatial pose of the locator;

[0050] [[ID=3,8]]S15. By minimizing the reprojection residual, the optimal deviation of the external reference from the coordinate system can be iteratively obtained.

[0051] As a preferred embodiment, the internal parameter matrix K in S11 is a 3-dimensional square matrix, defined as:

[0052]

[0053] f is the focal length of the camera (in pixels at this time), and u0 and v0 correspond to the offset of the camera center point.

[0054] As a preferred embodiment, in step S12, an external locator is started, and it is necessary to ensure that the locator system is synchronized with the camera system so that the spatial position and posture can be recorded in real time during the shooting process.

[0055] Among them, the data of the external locator is obtained through the network protocol and can eventually be converted into position information rotation and translation.

[0056] As a preferred embodiment, in S13, the characteristic marker points are kept evenly distributed in the camera field of view so as to capture the camera's posture information from multiple angles. The lower left corner of the screen can be set as the origin of the screen coordinate system, and the coordinates of the light bead in row i and column j are obtained as follows:

[0057] P=(j*w led ,i*h led ,0);

[0058] w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are:

[0059] Points={P0,P1,…P M};

[0060] At the same time, record the locator output And the corresponding picture I i .

[0061] As a preferred embodiment, the marking method of the characteristic marking points photographed in S14 is:

[0062] Known feature points in screen coordinate system O screen The coordinates of Points={P0,P1,…P M}, the pixel coordinates observed by the camera at time i are:

[0063] As a preferred embodiment, the combination of the feature markers and the locator spatial pose in S14 is as follows:

[0064] The camera pose is set to Then for each point there is Since there is noise in the pixel observation, that is:

[0065]

[0066] As a preferred implementation manner, the way to list the reprojection residual equation of the external parameter and coordinate system deviation in S14 is as follows: List the optimization residual equation by observing multiple groups of feature points multiple times:

[0067]

[0068] where T loc2camera is the external parameter. Expand to obtain Then the residual equation is modified to:

[0069]

[0070] As a preferred implementation manner, in S15, the gradient descent method is used to find the optimal external parameter and coordinate system deviation. First, find the Jacobian matrix J of the constraint equation, and then the step function can be obtained:

[0071] step=(J T J + uI) -1 J T f;

[0072] where T represents taking the transpose of the matrix, -1 represents taking the inverse of the matrix, u is the damping coefficient, and I is the identity matrix;

[0073] Set the threshold threshold to control the accuracy and the number of iterations;

[0074] Set the evaluation function rou = mean[f(X + ΔX)-f(X) / (J(X)ΔX)], where mean represents taking the average, and X represents the variable to be optimized, that is, the external parameter and coordinate system deviation.

[0075] As a preferred implementation manner, the subsequent iterative optimization steps are as follows:

[0076] S21. Give the initial external parameter and coordinate system deviation, substitute it into JTJ to obtain a square matrix, and take the maximum value on its diagonal element as the initial value of the confidence domain u = u0;

[0077] S22. Substitute and calculate rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2;

[0078] S23. Determine whether to update the step function according to whether the value of u is updated as above. The step function obtains step(X), and the updated external parameter and coordinate system deviation is X' = X + step(X);

[0079] S24. Substitute X' into the constraint equation and calculate the average to obtain the error of this iteration. If error > threshold, substitute X' into Step 2 for recalculation. If error < threshold, jump out of the loop. The final X' is the deviation of the external participation coordinate system.

[0080] Please refer to Figure 1 , some QR codes can be seen. The size unit of these QR codes is the width of the light beads. Using these QR codes, feature points at the sub - millimeter level (the light beads are generally less than 1 mm) can be extracted.

[0081] Compared with the traditional calibration method, this method does not need to use a ranging instrument for measurement. At the same time, it takes into account the offset between the origin of the locator and the screen coordinate system. By sampling multiple groups of synchronous positioning data of the camera and the locator, the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen, can be solved simultaneously.

[0082] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non - exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0083] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for calibrating the external parameters of a camera for hybrid display, characterized in that: This method re - models by introducing the origin of the locator and the screen offset, and simultaneously solves the external parameters of the camera and the locator, as well as the offset between the origin of the locator and the screen, using non - linear optimization; This method includes the following steps: S11. Obtain the internal parameter matrix of the camera through the method of internal parameter calibration; S12. Start the external locator; S13. Control the screen LED beads, consider the bright points as feature marker points, light up M of them, and take images N times at different positions, while recording the output of the locator at this moment; S14. According to the captured feature marker points, list the reprojection residual equations of the external parameters and the coordinate system deviation by combining with the spatial pose of the locator; S15. By minimizing the reprojection residual, the optimal external parameters and coordinate system deviation can be iteratively obtained. In S11, the internal parameter matrix K is a 3 - dimensional square matrix, defined as: f is the camera focal length, u0 and v0 correspond to the offset of the camera center point. In S12, start the external locator, keep the locator system and the camera system synchronized, and record the spatial position and attitude in real - time during the shooting process. In S13, the feature marker points are evenly distributed in the camera's field of view, and the attitude information of the camera is captured from multiple angles. Set the lower - left corner of the screen as the origin of the screen coordinate system, and the coordinates of the illuminated LED bead in the i - th row and j - th column are: P=(j*w led ,i*h led ,0); w led and h led is the width and height of a single lamp bead, then the three-dimensional coordinates of the M feature points are: Points={P0,P1,…P M }; At the same time, record the locator output And the corresponding picture I i , the marking method of the characteristic marking points photographed in S14 is: Known feature points in screen coordinate system o screen The coordinates of points={p0,p1,…p M }, the pixel coordinates observed by the camera at time i are:

2. The camera extrinsic parameter calibration method for hybrid display according to claim 1, wherein: The combination method of the feature marker points and the spatial pose of the locator in S14 is: The camera pose is set to Then for each point there is Since there is noise in the pixel observation, that is:

3. The camera extrinsic parameter calibration method for hybrid display according to claim 2, wherein: The method of listing the reprojection residual equations of the external parameters and the coordinate system deviation in S14 is: list the optimization residual equations by observing multiple groups of feature points multiple times: in For external reference, Expand and get Then the residual equation is modified as follows:

4. The camera extrinsic parameter calibration method for hybrid display according to claim 3, wherein: In S15, use the gradient - descent method to find the optimal external parameters and coordinate system deviation. First, find the Jacobian matrix J of the constraint equation, and then the step - size function can be obtained: step=(J T J+uI) -1 J T f; where T represents taking the transpose of the matrix, - 1 represents taking the inverse of the matrix, u is the damping coefficient, and I is the identity matrix; Set the threshold threshold to control the accuracy and the number of iterations; Set the evaluation function rou = mean[f(X + ΔX)-f(X) / (J(X)ΔX)], where mean represents taking the average, X represents the variable to be optimized, that is, the external parameters and the coordinate system deviation.

5. The camera extrinsic parameter calibration method for hybrid display according to claim 4, characterized in that: The subsequent iterative optimization steps are: S21. Give the initial external parameters and coordinate system deviation, substitute them into JTJ to obtain a square matrix, and take the maximum value on its diagonal element as the initial value of the confidence domain u = u0; S22. Substitute and calculate rou. If rou < 0.25, let u = u * 2. If 0.25 < rou < 0.75, u remains unchanged. If rou > 0.75, then u = u / 2; S23. Determine whether to update the step - size function according to whether the value of u is updated as above. The step - size function gets step(X), and the updated external parameters and coordinate system deviation are X’ = X + step(X); S24. Substitute X’ into the constraint equation and calculate the average to obtain the error of this iteration error. If error > threshold, substitute X’ into step 2 to recalculate. If error < threshold, jump out of the loop. The final X’ is the external parameters and the coordinate system deviation.

Images