Camera calibration method, system and application in augmented reality visual display

Abstract

The present invention belongs to the field of augmented reality technology, and discloses a camera calibration method, system and application in an augmented reality visual display, which refines and purifies feature points detected by a Harris corner detection algorithm; re-derives the standard camera imaging model formula and introduces lens distortion coefficients; improves the original beetle whisker search algorithm BAS, introduces two variables μ and eta, and solves the optimization of the initial values ​​of camera parameters. The present invention improves the original beetle whisker search algorithm, adopts variable step coefficient attenuation to replace the attenuation step coefficient of each iteration, and simultaneously utilizes the different orders of magnitude of the intrinsic distortion coefficients, and adjusts the step component size according to the order of magnitude ratio, thereby solving the shortcomings of the original optimization algorithm that is easy to fall into the local optimal value and the poor optimization effect of high-dimensional variables. The present invention improves the camera imaging model, and uses the functional relationship of the intrinsic distortion coefficients for fitting, so that the calibration accuracy of the camera is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality technology, and in particular relates to a camera calibration method, system and application in an augmented reality visual display. Background Art

[0002] At present, in the visual display of augmented reality, cameras are usually used as physical tracking devices, and the images taken by the camera are analyzed by mature computer vision technology to meet the high quality requirements of the augmented reality system for three-dimensional registration. Currently, commonly used visual displays generally include smartphones, tablets, head mounted displays (Head Mounted Display, HMD), etc. In augmented reality, virtual electronic information needs to be aligned with the real physical world, and the camera is the medium that connects the two, so it is necessary to establish an accurate camera geometry model. Therefore, camera calibration is crucial.

[0003] Camera calibration is to measure the camera's internal parameters and the lens' distortion coefficients. These parameters are usually considered constants and do not require online real-time measurement. They only need to be calibrated and the parameters solved before the device is started or when it leaves the factory, and then used in subsequent calculations. Therefore, in augmented reality devices, offline calibration is more considered, and the requirements on the time complexity of the algorithm can be relaxed, with more consideration given to accuracy and robustness.

[0004] Camera calibration methods are generally divided into traditional camera calibration methods and camera self-calibration methods. Since the calibration accuracy of camera self-calibration methods is low, the robustness is poor, and a high-precision camera model cannot be established, the traditional camera calibration method is generally used in augmented reality devices. The traditional camera calibration method uses some landmarks with known sizes and specific rules. Through the relationship between the three-dimensional position of a point in space and the corresponding point on the image plane, the relative geometric posture relationship of the camera and the camera's intrinsic distortion coefficient are solved. At present, the most widely used methods are Zhang Zhengyou calibration method, Tsai two-step calibration method and direct linear transformation method. Zhang Zhengyou calibration method uses a black and white plane chessboard, takes multiple groups of pictures from different positions and angles, solves the camera parameters through these pictures, and then performs nonlinear optimization, but the calibration accuracy is low. Tsai two-step calibration method uses the solution of overdetermined linear equations to obtain camera parameters, and then considers the distortion coefficient, but it is impossible to calibrate all intrinsic parameters through a plane, and the nonlinear operation involved may make the result unstable.

[0005] In recent years, many researchers have been studying how to optimize the initial value of calibration. They first use Zhang Zhengyou's calibration method to obtain the initial value of the camera's internal parameters, and then use intelligent optimization algorithms for iterative optimization. Xu Chengyi et al. improved the particle swarm algorithm to achieve nonlinear adaptive adjustment of inertial parameters at different iteration stages to balance local and global search capabilities, and used a dispersal mechanism to expand the spatial range of the particle swarm to avoid premature convergence of the algorithm. You Jiang et al. improved the genetic simulated annealing algorithm, combining a genetic algorithm with good convergence and a simulated annealing algorithm with strong local search capabilities, and proposed a method of adaptive crossover and mutation probability based on competitive weights. In the late stage of evolution, based on the population stabilization criterion, a simulated annealing mechanism was added for optimization. Wang Daolei et al. proposed an optimization method based on the longhorn beetle whisker search algorithm to simulate the foraging process of longhorn beetles, but it is not suitable for the optimization of high-dimensional vectors and is prone to fall into local optimal solutions.

[0006] Through the above analysis, the problems and defects of the prior art are as follows:

[0007] (1) The camera self-calibration method has low calibration accuracy and poor robustness, and cannot establish a high-precision camera model.

[0008] (2) Among the traditional camera calibration methods, the calibration accuracy of Zhang Zhengyou's calibration method is relatively low; Tsai's two-step calibration method cannot calibrate all the intrinsic parameters through one plane, and the nonlinear calculation involved may make the result unstable.

[0009] (3) The existing optimization method based on the beetle whisker search algorithm is not suitable for the optimization of high-dimensional vectors and is prone to fall into local optimal solutions. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a camera calibration method, system and application in an augmented reality (AR) visual display.

[0011] The present invention is implemented in such a way that a camera calibration method in an augmented reality visual display comprises the following steps:

[0012] Step 1: refine and purify the feature points detected by the Harris corner detection algorithm, and select an optimal number of refinement iterations;

[0013] Step 2: re-derive the standard camera imaging model formula and introduce the lens distortion coefficient to make the imaging model more accurate;

[0014] Step three, the original beetle whisker search algorithm BAS is improved by introducing two variables μ and eta, and used to optimize the initial values ​​of the camera parameters solved in steps one and two. By comparing the algorithm with the traditional beetle whisker search algorithm and the MATLAB calibration toolbox calibration algorithm, ablation experiments are performed on the two newly introduced parameter variables μ and eta to fully verify the superiority of this method.

[0015] Furthermore, the sub-pixel refinement of the corner point in step 1 includes: obtaining a more accurate sub-pixel coordinate value by sub-pixel refinement. When q0 is the initial pixel point obtained by the corner point detection algorithm, the final sub-pixel point is q t Sub-pixel refinement is an iterative optimization process. To solve the new sub-pixel point q1, multiple pixel points p in a window are selected with the integer coordinate closest to point q0 as the center. i , through the pixel point p i The coordinate information of the center pixel of the window is continuously iterated and optimized to finally obtain the refined sub-pixel coordinate value.

[0016] The specific steps are as follows: Select pixel point p i Then, calculate the pixel point p i The gradient information G i .

[0017] If you choose p i Located on the edge of the chessboard, the gradient G at this time i With vector (p i -q1) orthogonal; if the selected p i If it is not on the edge, then p i The pixel values ​​near the point are the same, and there is no gradient size. For the selected single pixel point p i , the following always holds true:

[0018] G i *(p i -q1)=0;

[0019] If all selected pixels satisfy the above formula, the new sub-pixel point can be obtained by calculating the pseudo-inverse in the least squares form:

[0020]

[0021] After obtaining the sub-pixel coordinates, multiple iterations are performed. During the iteration process, more errors are introduced when selecting surrounding pixels and calculating gradient information. Therefore, the accuracy threshold ε is specified as the iteration termination condition. At this time, it is considered that accurate sub-pixel coordinates are obtained:

[0022] ||q t -q t-1 ||2≤ε.

[0023] Furthermore, the camera imaging model in step 2 includes:

[0024] When an object is imaged, it is converted in the three-dimensional coordinate system from the world coordinate system OX w Y w Z w Transform to camera coordinate system OX c Y c Z c , and then transformed into the two-dimensional image coordinate system O-XY through projection transformation, and finally transformed into the pixel coordinate system O-UV; the transformation in three-dimensional space is called rigid body transformation, which only changes the orientation and spatial position of the object without changing the shape. It is described by two variables, the rotation matrix R and the translation vector T, and the expression is:

[0025]

[0026] Converted into homogeneous coordinates:

[0027]

[0028] The transformation from the camera coordinate system to the image coordinate system is called perspective projection transformation, which converts three-dimensional information into two-dimensional information and loses depth information. According to the proportional relationship, the following relationship is obtained:

[0029]

[0030]

[0031]

[0032] Radial distortion δ r It can be described by the following mathematical expression:

[0033]

[0034] Tangential distortion δ d It can be described by the following mathematical expression:

[0035]

[0036]

[0037] After introducing the lens distortion, the transformation relationship of the imaging point from the ideal image coordinate point to the real image coordinate point is expressed as:

[0038]

[0039] Written in matrix form:

[0040]

[0041] Convert the image coordinate system to the pixel coordinate system, which is the coordinate system of the final display window:

[0042]

[0043] Among them, dx and dy are the physical sizes of pixels in the X and Y axis directions respectively; u0 and v0 are the coordinates of the imaging center.

[0044] Combining the conversion formulas, we can get the conversion relationship between the world coordinate system and the pixel coordinate system:

[0045]

[0046]

[0047]

[0048]

[0049] Among them, f u 、f v It is called the scale factor of the u and v axes, K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3) is called the camera's internal parameter matrix, [RT] is called the camera's external parameter matrix; the distortion parameters are introduced into the internal parameter matrix K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3).

[0050] Further, the improvement of the original longicorn beetle whisker search algorithm in step 3 includes:

[0051] The BAS algorithm is improved by randomly generating a disturbance vector d of the parameter vector and adding a noise coefficient μ to each component of the initial vector. i :

[0052]

[0053] Use the average reprojection error function as the concentration function of the BAS algorithm:

[0054]

[0055] Calculate the position and concentration value f(X l )、f(X r ), if the concentration value is less than the current local optimal value, update the local optimal value and parameter vector X1; determine the forward direction according to the size of the left and right concentration values, update the parameter vector, and decay the jump step size. The next jump step size is:

[0056] step t+1 =step t *eta;

[0057] Among them, eta represents the iterative variable step size coefficient.

[0058] Compared with the original beetle whisker search algorithm, two additional parameters μ and eta are introduced, where μ controls the convergence value and eta controls the convergence speed.

[0059] Another object of the present invention is to provide a camera calibration system in an augmented reality visual display using the camera calibration method in an augmented reality visual display, the camera calibration system in the augmented reality visual display comprising:

[0060] Feature point refinement and purification module, used to refine and purify the feature points detected by Harris corner point detection algorithm;

[0061] The lens distortion coefficient introduction module is used to re-derive the standard camera imaging model formula and introduce the lens distortion coefficient;

[0062] The BAS algorithm improvement module is used to improve the original beetle whisker search algorithm BAS, introduce two variables μ and eta, and use them in the refinement and purification module and the lens distortion coefficient introduction module to solve the optimization of the initial values ​​of the camera parameters.

[0063] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0064] The feature points detected by the Harris corner detection algorithm are refined and purified; the standard camera imaging model formula is re-derived and the lens distortion coefficient is introduced; the original beetle whisker search algorithm BAS is improved and two variables μ and eta are introduced to solve the optimization of the initial values ​​of the camera parameters.

[0065] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0066] The feature points detected by the Harris corner detection algorithm are refined and purified; the standard camera imaging model formula is re-derived and the lens distortion coefficient is introduced; the original beetle whisker search algorithm BAS is improved and two variables μ and eta are introduced to solve the optimization of the initial values ​​of the camera parameters.

[0067] Another object of the present invention is to provide a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to apply the camera calibration method in an augmented reality visual display.

[0068] Another object of the present invention is to provide an information data processing terminal, which is used to implement the camera calibration method in the augmented reality visual display.

[0069] Another object of the present invention is to provide an application of the camera calibration method in an augmented reality visual display to image calibration of a smart phone, a tablet computer, and a head-mounted display.

[0070] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0071] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0072] The accuracy of camera calibration in traditional camera calibration methods mainly depends on the resolution of the camera, the accuracy of the corner detection algorithm, the accuracy of the nonlinear iterative optimization, etc.

[0073] The extraction of corner points involves image processing technology, and the sub-pixel refinement method is used to further purify the detected corner points to obtain more accurate corner point coordinate values.

[0074] The accuracy of the camera parameters obtained using the traditional camera calibration method can be further optimized. The attenuation coefficient used in the original beetle whisker search algorithm for iteration is a fixed value. In the initial stage of the search, the step length decays quickly and the search accuracy is low. As the number of search layers deepens, the step length decays slowly, the local search efficiency is slow, and the optimization effect for high-dimensional data is poor. The present invention improves the original beetle whisker search algorithm, and uses a variable step length coefficient decay to replace the attenuation step length coefficient of each iteration. At the same time, the internal distortion coefficient has different orders of magnitude, and the step length component size is adjusted according to the order of magnitude ratio, which solves the problem that the original optimization algorithm is prone to fall into the local optimum and the high-dimensional variable optimization effect is poor. In addition, there are too many quantities to be optimized, and if the initial value is not set well, it is easy to fall into the local optimum. The present invention improves the camera imaging model, and fits the functional relationship of the internal distortion coefficient, so that the calibration accuracy of the camera is further improved.

[0075] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0076] The present invention improves the camera imaging model and uses the functional relationship of the internal distortion coefficient for fitting, so that the calibration accuracy of the camera is further improved.

[0077] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0078] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0079] The purpose of studying camera calibration technology is to provide accurate camera pose parameters and distortion parameters, which are the basis for a series of visual tasks. It can be used to initialize a series of tasks such as image distortion correction, intelligent robots, and augmented reality space registration. However, since high-precision cameras are generally used in augmented reality applications, compared with traditional methods, this technical solution has a higher degree of matching with high-precision cameras and is more suitable for augmented reality tasks. Since the current hot technology - Metaverse is closely related to augmented reality technology, this method can be used in advance for calibration in augmented reality space registration, so that the effect of space registration is further improved, promoting the application and development of Metaverse technology.

[0080] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0081] In many camera calibration methods, the sub-pixel refinement method is not used, resulting in errors in the detected initial corner points; this technical solution introduces in detail the re-derivation process of the camera imaging model to make the model more reasonable; at the same time, it improves the traditional beetle whisker search algorithm, which can make high-dimensional vectors converge quickly; and combines the above methods to form the final high-precision camera calibration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 It is a flow chart of a camera calibration method in an augmented reality visual display provided by an embodiment of the present invention.

[0084] Figure 2 It is a schematic diagram of a camera calibration method in an augmented reality visual display provided by an embodiment of the present invention.

[0085] Figure 3 is a structural block diagram of a camera calibration system in an augmented reality visual display provided by an embodiment of the present invention;

[0086] In the figure: 1. Feature point refinement and purification module; 2. Lens distortion coefficient introduction module; 3. BAS algorithm improvement module.

[0087] Figure 4(a) to Figure 4(d) It is a schematic diagram of nonlinear optimization of initial parameters of a camera with μ value changed and eta unchanged provided by an embodiment of the present invention.

[0088] Figure 5(a) to Figure 5(d) It is a schematic diagram of nonlinear optimization of initial parameters of a camera with changing eta value and keeping μ unchanged provided by an embodiment of the present invention.

[0089] Figure 6 This is a 3D diagram that takes into account the impact of the number of iterations and the noise coefficient on the average reprojection error, provided by an embodiment of the present invention.

[0090] Figure 7 It is a schematic diagram of a black and white plane checkerboard calibration plate photographed by Hololens2 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0092] In view of the problems existing in the prior art, the present invention provides a camera calibration method and system in an augmented reality visual display. The present invention is described in detail below in conjunction with the accompanying drawings.

[0093] 1. In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an explanatory embodiment that expands the technical solution of the claims.

[0094] like Figure 1 As shown, the camera calibration method in the augmented reality visual display provided by the embodiment of the present invention includes the following steps:

[0095] S101, refining and purifying the feature points detected by using the Harris corner detection algorithm;

[0096] S102, re-deriving the standard camera imaging model formula and introducing the lens distortion coefficient;

[0097] S103, the original beetle whisker search algorithm BAS is improved by introducing two variables μ and eta, and used to optimize the initial values ​​of the camera parameters solved by S101 and S102.

[0098] The principle diagram of the camera calibration method in the augmented reality visual display provided by the embodiment of the present invention is shown in Figure 2 .

[0099] like Figure 3 As shown, the camera calibration system in the augmented reality visual display provided by the embodiment of the present invention includes:

[0100] Feature point refinement and purification module 1, used to refine and purify feature points detected using Harris corner point detection algorithm;

[0101] The lens distortion coefficient introduction module 2 is used to re-derive the standard camera imaging model formula and introduce the lens distortion coefficient;

[0102] BAS algorithm improvement module 3 is used to improve the original beetle whisker search algorithm BAS, introduce two variables μ and eta, and use them in the refinement and purification module and the lens distortion coefficient introduction module to solve the optimization of the initial values ​​of the camera parameters.

[0103] The technical solution of the present invention is further described below by taking a high-precision camera calibration algorithm suitable for augmented reality visual displays as an example.

[0104] The present invention proposes a high-precision camera calibration algorithm suitable for augmented reality visual displays, the main steps of which are:

[0105] Step 1. Further refine and purify the feature points detected by Harris corner detection algorithm;

[0106] Step 2. Re-derive the standard camera imaging model formula and introduce the lens distortion coefficient in the process;

[0107] Step 3. Improve the original beetle whisker search algorithm by introducing two variables μ and eta, and use them to optimize the initial values ​​of the camera parameters solved in steps 1 and 2.

[0108] The processor used in this algorithm experiment is Intel(R) Core(TM) i7-7700HQ CPU@2.80GHz2.81GHz, RAM 8.00GB, based on x64 processor. The steps of the algorithm are detailed as follows:

[0109] 1. Corner sub-pixel refinement

[0110] In a black and white chessboard, a corner point refers to the intersection of the black and white edges of the chessboard. Harris detectors are generally used to extract corner points. However, in digital images, the grayscale changes on both sides of the edge are slow transitions, not ideal step changes. Therefore, the corner point coordinates obtained by the corner point detection algorithm are not accurate enough, and more accurate sub-pixel coordinate values ​​can be obtained through sub-pixel refinement.

[0111] Assume that q0 is the initial pixel point obtained by the corner detection algorithm, and the final sub-pixel point is q t Sub-pixel refinement is an iterative optimization process. In order to solve the new sub-pixel point q1, we first select multiple pixel points p in a window with the integer coordinate closest to point q0 as the center. i , through the continuous iteration and optimization of the information of these pixels and the initial pixels, the refined sub-pixel coordinate values ​​are finally obtained. After selecting the pixels, the gradient information G of these pixels needs to be calculated i If the selected p i Located on the edge of the chessboard, the gradient G at this time i With vector (p i -q1) orthogonal; if the selected p i Not on the edge, then p i The pixel values ​​near the point are the same, and there is no gradient size. For the selected single pixel point p i , the following formula always holds true.

[0112] G i *(p i -q1)=0

[0113] Considering that all selected pixels satisfy the above formula, the new sub-pixel point can be obtained by calculating the pseudo-inverse in the least squares form:

[0114]

[0115] This is the obtained sub-pixel coordinate. Multiple iterations are performed according to this method. Since more errors may be introduced when selecting surrounding pixels and calculating gradient information during the iteration process, the accuracy threshold ε can be specified as the iteration termination condition. At this time, it is considered that the accurate sub-pixel coordinates are obtained.

[0116] ||q t -q t-1 ||2≤ε

[0117] 2. Camera imaging model

[0118] When an object is imaged, it is first converted in the three-dimensional coordinate system, from the world coordinate system OX w Y w Z w Transform to camera coordinate system OXc Y c Z c , and then transformed to the two-dimensional image coordinate system O-XY through projection transformation, and finally transformed to the pixel coordinate system O-UV. Among them, the transformation in three-dimensional space is called rigid body transformation, which only changes the orientation and spatial position of the object without changing its shape. It can be described by two variables, the rotation matrix R and the translation vector T, and its expression is:

[0119]

[0120] Converted into homogeneous coordinates:

[0121]

[0122] The transformation from the camera coordinate system to the image coordinate system is called perspective projection transformation, which converts three-dimensional information into two-dimensional information and loses depth information. According to the proportional relationship, the following relationship can be obtained:

[0123]

[0124]

[0125]

[0126] Camera imaging uses the pinhole imaging principle and uses multiple lenses to focus to compensate for the lack of exposure. However, the use of lenses and deviations in the lens assembly process will introduce optical distortion, resulting in distortion of the original image. Lens distortion is divided into two categories: radial distortion and tangential distortion. Radial distortion comes from the lens and is a property of the lens itself, while tangential distortion comes from the lens assembly process. In addition, there are other types of distortion, but they are not as significant as radial distortion and tangential distortion. The present invention only considers these two forms of distortion.

[0127] Radial distortion δ r It can be described by the following mathematical expression:

[0128]

[0129] Tangential distortion δ d It can be described by the following mathematical expression:

[0130]

[0131]

[0132] After introducing the lens distortion, the transformation relationship of the imaging point from the ideal image coordinate point to the real image coordinate point can be expressed as:

[0133]

[0134] Written in matrix form:

[0135]

[0136] Finally, the image coordinate system needs to be converted to the pixel coordinate system, which is the coordinate system of the final display window.

[0137]

[0138] Where dx and dy are the physical sizes of the pixel in the X and Y axis directions respectively. u0 and v0 are the coordinates of the imaging center.

[0139] Combining the above conversion formulas, we can get the conversion relationship between the world coordinate system and the pixel coordinate system.

[0140]

[0141]

[0142]

[0143]

[0144] Among them, f u 、f v It is called the scale factor of the u and v axes, K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3) is called the camera's internal parameter matrix, and [RT] is called the camera's external parameter matrix. The distortion parameters are introduced into the internal parameter matrix K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3).

[0145] 3. Improvement of BAS algorithm

[0146] Considering that the original BAS algorithm is not good enough for optimizing multidimensional vectors with different orders of magnitude, and it is easy to fall into the local minimum when using a fixed step size, the BAS algorithm is improved by randomly generating a perturbation vector d of the parameter vector and adding a noise coefficient μ to each component of the initial vector. i :

[0147]

[0148] Use the average reprojection error function as the concentration function of the BAS algorithm:

[0149]

[0150] Calculate the position and concentration value f(X l )、f(X r ), if the concentration value is less than the current local optimal value, then update the local optimal value and parameter vector X1, determine the forward direction according to the size of the left and right concentration values, update the parameter vector, and decay the jump step size. The next jump step size is:

[0151] step t+1 =step t *eta;

[0152] Among them, eta represents the iterative variable step size coefficient.

[0153] Compared with the original beetle beard search algorithm, two additional parameters μ and eta are introduced, where μ controls the convergence value and eta controls the convergence speed.

[0154] 2. In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0155] A black and white flat chessboard with 9 rows and 12 columns was used, with each grid measuring 25mm×25mm. The augmented reality visual display selected was the Hololens2 head-mounted display developed by Microsoft. A total of 20 photos of the chessboard were taken from different positions and angles.

[0156] 1. Sub-pixel refinement

[0157] The average reprojection error is used as the evaluation metric, and the average reprojection error before and after refinement is calculated.

[0158]

[0159] Table 1 Calculation results of average reprojection error before and after refinement

[0160]

[0161]

[0162] It can be seen that the average reprojection error is reduced by 2.5% after using the sub-pixel refinement method.

[0163] 2. Nonlinear optimization of initial camera parameters

[0164] The original beetle whisker search algorithm, the improved beetle whisker search algorithm and the MATLAB calibration toolbox were used to perform nonlinear optimization of the initial camera parameters.

[0165] like Figure 4(a) to Figure 4(d)As shown in the figure, the μ value is changed, eta remains unchanged, and is set to 1, which is the same as the step size of the original beetle whisker search algorithm. The μ values ​​are taken as 0.01, 0.02, 0.05, and 0.1, and the final converged values ​​are 0.50019, 0.50066, 0.48190, and 0.48833, respectively. It can be seen that the μ value can affect the final convergence error, and their final effects are better than those of the original beetle whisker search algorithm.

[0166] like Figure 5(a) to Figure 5(d) As shown in the figure, the eta value is changed, and μ remains unchanged. The eta values ​​are 1, 0.95, 0.90, and 0.85 respectively, and the final converged value remains basically unchanged, but the number of iterations required is 260, 73, 32, and 23. It can be seen that the eta value can affect the convergence speed, but in order to pursue a faster convergence speed, it is inevitable to lose some accuracy.

[0167] Considering the influence of the number of iterations and the noise coefficient on the average reprojection error, a 3D graph is made as follows Figure 6 shown.

[0168] It can be seen that when the μ value is less than 0.05, the calculated error is about 0.50. As the μ value increases, the calculated error is about 0.48, indicating that in this algorithm, the value of μ cannot be too small. The final convergence value of μ between 0.01 and 1.00 is statistically analyzed, and the average error is 0.4854, which is 4.8% higher than the calibration toolbox and 4.9% higher than the original beetle whisker search algorithm.

[0169] The selected chessboard is a black and white plane chessboard with 9 rows and 12 columns. The size of each grid is 25mm×25mm. The chessboard picture taken is as follows: Figure 7 shown.

[0170] 3. Conduct comparative experiments using different devices

[0171] Next, to verify that this method has a good calibration effect on high-precision cameras in augmented reality products, we selected smartphones and tablets that can also achieve augmented reality effects and compared them with high-precision cameras in head-mounted displays. The model of the smartphone is Red Note 10Pro and the model of the tablet is HUAWEI M5. We took 20 pictures from different angles of the same calibration plate and used the same method to calibrate the cameras. The results are as follows.

[0172] Table 2 Back-projection errors solved using different devices

[0173]

[0174] It can be seen that the calibration effect of this method on the augmented reality head-mounted display with Hololens2 high-precision camera is better than that of the calibration toolbox. Although this method can also be used to calibrate the camera of products such as smartphones and tablets, the wide-angle lenses they carry are obviously more suitable for calibration using the traditional method in the MATLAB calibration toolbox. This shows that this method has a good effect on the calibration of high-precision cameras carried in augmented reality head-mounted displays, and further illustrates that the initial value has an extremely important influence on the final effect and performance of this method.

[0175] 3. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with data, charts, etc. of the test process.

[0176] The method of the present invention is compared with an uncalibrated camera, a traditional beetle whisker search algorithm optimization method, and a MATLAB calibration toolbox calibration method, and the following data are obtained, which fully illustrates the superiority of the method of the present invention.

[0177] Table 3 Camera parameters solved using different optimization methods

[0178]

[0179]

[0180] The initial value + refinement experiment refers to the sub-pixel refinement of the initial image. The original BAS experiment refers to the sub-pixel refinement of the image, and then the non-linear optimization is performed after the initial camera parameters are solved using the unimproved camera imaging model. The subsequent improved imaging model + BAS experiment refers to the sub-pixel refinement of the image, and then the initial camera parameters are solved using the improved camera imaging model, and then the original BAS algorithm is used for non-linear optimization.

[0181] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0182] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A camera calibration method in an augmented reality visual display, characterized in that: The camera calibration method in the augmented reality visual display comprises the following steps: Step 1: Refine and purify the feature points detected by Harris corner detection algorithm, and select an optimal number of refinement iterations; Step 2: Re-derive the standard camera imaging model formula and introduce the lens distortion coefficient to make the imaging model more accurate; Step 3: Improve the original beetle whisker search algorithm BAS, introduce two variables μ and eta, and use them to optimize the initial values ​​of the camera parameters solved in step 1 and step 2. By comparing the traditional beetle whisker search algorithm and the MATLAB calibration toolbox calibration algorithm, ablation experiments on the two newly introduced parameter variables μ and eta fully verify the superiority of the method; The sub-pixel refinement of the corner point in step 1 includes: obtaining a more accurate sub-pixel coordinate value through sub-pixel refinement; when q0 is the initial pixel point obtained by the corner point detection algorithm, the final sub-pixel point is q t ; Sub-pixel refinement is a process of continuous iterative optimization. To solve the new sub-pixel point q1, multiple pixel points p in a window are selected with the integer coordinates closest to point q0 as the center. i , through the pixel point p i The coordinate information of the pixel point at the center of the window is continuously iterated and optimized, and finally the refined sub-pixel coordinate value is obtained; the pixel point p is selected i Then, calculate the pixel point p i The gradient information G i ; Selected p i When it is on the edge of the chessboard, the gradient G i With vector (p i -q1) orthogonal; selected p i If it is not on the edge, then p i The pixel values ​​near the point are the same, there is no gradient size, for the selected single pixel point p i , the following always holds true: G i *(p i -q1)=0; If all selected pixels satisfy the above formula, the pseudo-inverse is calculated in the least squares form to obtain the new sub-pixel point: Get the sub-pixel coordinates, perform multiple iterations, and use the precision threshold ε as the iteration termination condition to get the accurate sub-pixel coordinates: ||q t -q t-1 ||2≤ε; The camera imaging model in step 2 includes: when an object is imaged, it is converted in a three-dimensional coordinate system from the world coordinate system OX w Y w Z w Transform to camera coordinate system OX c Y c Z c , and then transformed into the two-dimensional image coordinate system O-XY through projection transformation, and finally transformed into the pixel coordinate system O-UV; the transformation in three-dimensional space is a rigid body transformation, which only changes the orientation and spatial position of the object without changing the shape. The rotation matrix R and the translation vector T are expressed as: Converted into homogeneous coordinates: The transformation from the camera coordinate system to the image coordinate system is a perspective projection transformation, which converts three-dimensional information into two-dimensional information and loses depth information. According to the proportional relationship, the following relationship is obtained: Radial distortion δ r Described by the following mathematical expression: Tangential distortion δ d Described by the following mathematical expression: After introducing the lens distortion, the transformation relationship of the imaging point from the ideal image coordinate point to the real image coordinate point is expressed as: Written in matrix form: Convert the image coordinate system to the pixel coordinate system and finally display the window's coordinate system: Where dx and dy are the physical sizes of pixels in the X and Y axis directions respectively; u0 and v0 are the coordinates of the imaging center; Combining the conversion formulas, we can get the conversion relationship between the world coordinate system and the pixel coordinate system: Among them, f u 、f v It is called the scale factor of the u and v axes, K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3) is called the camera's internal parameter matrix, [RT] is called the camera's external parameter matrix; the distortion parameters are introduced into the internal parameter matrix K(f x ,f y ,u0,v0,k1,k2,p1,p2,k3); The improvement of the original beetle whisker search algorithm in step 3 includes: randomly generating a disturbance vector d of a parameter vector, adding a noise coefficient μ to each component of the initial vector i : Use the average reprojection error function as the concentration function of the BAS algorithm: Calculate the position and concentration value f(X l )、f(X r ), if the concentration value is less than the current local optimal value, the local optimal value and parameter vector X1 are updated; the forward direction is determined according to the size of the left and right concentration values, the parameter vector is updated, and the jump step is attenuated. The next jump step is: step t+1 =step t *eta; Among them, eta represents the iterative variable step size coefficient; Compared with the original beetle whisker search algorithm, two additional parameters μ and eta are introduced, where μ controls the convergence value and eta controls the convergence speed.

2. A camera calibration system in an augmented reality visual display using the camera calibration method in an augmented reality visual display as claimed in claim 1, characterized in that: The camera calibration system in the augmented reality visual display comprises: Feature point refinement and purification module, used to refine and purify the feature points detected by Harris corner point detection algorithm; The lens distortion coefficient introduction module is used to re-derive the standard camera imaging model formula and introduce the lens distortion coefficient; The BAS algorithm improvement module is used to improve the original beetle whisker search algorithm BAS, introduce two variables μ and eta, and use them in the refinement and purification module and the lens distortion coefficient introduction module to solve the optimization of the initial values ​​of the camera parameters.

3. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to perform the camera calibration method in an augmented reality visual display as claimed in claim 1.

4. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the camera calibration method in the augmented reality visual display according to claim 1.

5. A computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to apply the camera calibration method in an augmented reality visual display as claimed in claim 1.

6. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the camera calibration method in the augmented reality visual display as described in claim 1.

7. An application of the camera calibration method in an augmented reality visual display as claimed in claim 1 in image calibration of a smart phone, a tablet computer and a head-mounted display.

Images