A method for measuring optical axis deviation, a smart terminal, and a computer-readable storage medium.

Abstract

This invention discloses a method for measuring optical axis deviation, a smart terminal, and a computer-readable storage medium. The method includes: acquiring a first image and a second image captured for the same scene; calculating a target eigenvalue matrix corresponding to the first image and the second image based on the first image and the second image; and calculating the corresponding optical axis deviation value based on the target eigenvalue matrix. This invention improves the speed and convenience of calculating the optical axis deviation value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a light axis deviation measurement method, a smart terminal and a computer readable storage medium. BACKGROUND

[0002] In order to better present the image, the smart terminal represented by mobile phone adopts multiple cameras. But the prerequisite for better presenting the image is that the optical axes of multiple cameras are parallel, and the optical center spacing is equal to the physical distance between cameras.

[0003] But due to the lens and lighting characteristics of the camera itself, there is a certain distortion, and some errors are prone to occur in the installation process. Therefore, before leaving the factory, the optical axis deviation of these cameras needs to be measured to obtain the optical axis deviation angle between them for camera correction. The conventional optical axis deviation measurement method is mainly realized based on a calibration chart. First, a specific calibration chart is printed, and then the camera is controlled to take a picture of the calibration chart, and then the deviation angle between them is calculated by precision instruments according to the photographed picture. However, this operation is relatively cumbersome and complex. Therefore, the optical axis deviation measurement between multiple cameras needs to be improved. SUMMARY

[0004] The main purpose of the present application is to provide a light axis deviation measurement method, a smart terminal and a computer readable storage medium, which aims to solve the problem of cumbersome optical axis deviation measurement in the prior art.

[0005] To achieve the above purpose, the present application provides a light axis deviation measurement method, which comprises the following steps:

[0006] Obtaining a first image and a second image taken for the same scene;

[0007] According to the first image and the second image, a corresponding target intrinsic matrix is calculated;

[0008] According to the target intrinsic matrix, the optical axis deviation value corresponding to the first image and the second image is calculated.

[0009] Optionally, the light axis deviation measurement method, wherein the target intrinsic matrix is calculated according to the first image and the second image, specifically comprising:

[0010] Extracting feature points in the first image to generate a first feature point set, and extracting feature points in the second image to generate a second feature point set;

[0011] Matching the first feature point set and the second feature point set to generate a plurality of feature point pairs;

[0012] According to the feature point pairs, a target essential matrix between the first image and the second image is calculated.

[0013] Optionally, the optical axis deviation measurement method, wherein the first image is an image captured by a first camera, and the second image is an image captured by a second camera; and the feature points in the first image and the second image are extracted to generate a first feature point set and a second feature point set, specifically including:

[0014] The first image and the second image are subjected to grayscale processing to generate a first grayscale image and a second grayscale image.

[0015] According to a preset first distortion coefficient, the first grayscale image is corrected to generate a first corrected image, and according to a preset second distortion coefficient, the second grayscale image is corrected to generate a second corrected image, wherein the first distortion coefficient is a distortion coefficient of the first camera, and the second distortion coefficient is a distortion coefficient of the second camera.

[0016] The first corrected image is subjected to feature point extraction to generate a corresponding first feature point set, and the second corrected image is subjected to feature point extraction to generate a corresponding second feature point set.

[0017] Optionally, the optical axis deviation measurement method, wherein the target essential matrix between the first image and the second image is calculated according to the feature point pairs, specifically including:

[0018] According to a preset fitting rule, the feature point pairs are subjected to fitting calculation to generate an Nth essential matrix and an Nth inlier value, wherein N is a positive integer, and when the fitting calculation is performed for the first time, N is equal to 1.

[0019] When the Nth inlier value is greater than a preset inlier value threshold, the Nth inlier value is taken as the inlier value threshold, and a preset iteration number is incremented by one, wherein when the fitting calculation is performed for the first time, the iteration number is zero.

[0020] The Nth essential matrix is iteratively calculated, and the iteration number is updated according to the Nth inlier value, until the iteration number is greater than a preset iteration number threshold.

[0021] The essential matrix corresponding to the Nth inlier value with the largest value is determined as the target essential matrix.

[0022] Optionally, the optical axis deviation measurement method, wherein the Nth essential matrix and the Nth inlier value are generated by fitting calculation on the feature point pairs according to a preset fitting rule, specifically including:

[0023] selecting a number of feature point pairs equal to a preset test number from the feature point pairs in the point set as test feature point pairs;

[0024] According to the test feature point pair, the corresponding N eigenmatrix and N inner point value are calculated.

[0025] Optionally, the optical axis deviation measurement method, wherein the N eigenmatrix is calculated according to the test matrix, specifically including:

[0026] The test feature points are written into a preset matrix template to generate a test matrix;

[0027] According to the test matrix, the zero space corresponding to the test matrix is calculated to obtain the N eigenmatrix;

[0028] According to the N eigenmatrix, the number of matched feature points in the feature point pair is calculated to obtain the N inner point value.

[0029] Optionally, the optical axis deviation measurement method, wherein the N eigenmatrix is calculated according to the test matrix, specifically including:

[0030] According to the test matrix, the zero space corresponding to the test matrix is calculated to obtain the N eigenmatrix;

[0031] According to the preset additional equation, the system equation is eliminated to generate an intermediate matrix;

[0032] The intermediate matrix is solved to generate a corresponding intermediate value;

[0033] According to the intermediate value, the zero space is solved to generate the N eigenmatrix.

[0034] Optionally, the optical axis deviation measurement method, wherein the optical axis deviation value corresponding to the first image and the second image is calculated according to the target eigenmatrix, specifically including:

[0035] The target eigenmatrix is decomposed to obtain a candidate translation matrix and a candidate rotation matrix;

[0036] According to the depth value corresponding to the candidate rotation matrix, the target rotation matrix in the candidate rotation matrix is determined;

[0037] The target rotation matrix is subjected to Euler angle conversion to generate an Euler angle value, and the Euler angle value is taken as the optical axis deviation value corresponding to the first image and the second image.

[0038] In addition, to achieve the above object, the present application also provides an intelligent terminal, wherein the intelligent terminal comprises a memory, a processor, and an optical axis deviation measurement program stored in the memory and capable of running on the processor, and the optical axis deviation measurement program implements the steps of the optical axis deviation measurement method when executed by the processor.

[0039] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores an optical axis deviation measurement program, and the optical axis deviation measurement program implements the steps of the optical axis deviation measurement method when executed by a processor.

[0040] The present application first acquires a first image and a second image shot for the same scene, and then calculates a corresponding target essential matrix according to the first image and the second image. After obtaining the target essential matrix, the corresponding optical axis deviation value is calculated according to the target essential matrix. In the conventional optical axis deviation angle calculation process, the rotation matrix is calculated after the parameters of the camera are calibrated through a specific calibration image, and then the optical axis deviation angle is obtained. The present embodiment does not need to calibrate the parameters of the camera in advance, and the multiple cameras are used to shoot the same scene to calculate the essential matrix, so that the specific calibration image does not need to be printed in advance, the calculation complexity is reduced, and the rate and convenience of the optical axis deviation value are improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flow chart of the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0042] Figure 2 is a geometric diagram of the epipolar constraint of the first camera and the second camera in the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0043] Figure 3 is a diagram of the epipolar line in the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0044] Figure 4 is a diagram of the feature point matching in the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0045] Figure 5 is a simplified row gradient matrix after Gauss Jordan elimination in the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0046] Figure 6 is a diagram of four cases of target essential matrix decomposition in the preferred embodiment of the optical axis deviation measurement method provided by the present application;

[0047] Figure 7is the optical axis rotation coordinate system provided in the preferred embodiment of the optical axis deviation measurement method of the present application.

[0048] Figure 8 is a schematic diagram of the operating environment of the preferred embodiment of the intelligent terminal of the present application. Embodiment

[0049] To make the purpose, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the examples described herein are only used to explain the present application and do not limit the present application.

[0050] The optical axis deviation measurement method described in the preferred embodiment of the present application, as shown in Figure 1 The optical axis deviation measurement method comprises the following steps:

[0051] Step S100, acquiring a first image and a second image taken for the same scene.

[0052] Specifically, for ease of understanding, the present embodiment describes the example with two cameras.

[0053] The two cameras are named camera 1 and camera 2 respectively. The two cameras are controlled to take the same scene to obtain a first image and a second image. In the present embodiment, camera 1 is the first camera and camera 2 is the second camera.

[0054] Since feature point detection is needed for the first image and the second image in the following steps, the scene cannot be a pure color image. The scene should be a scene with texture, i.e. a scene with feature points that can be matched.

[0055] Then the first image and the second image are sent to the optical axis deviation measurement program for the optical axis deviation measurement program to acquire the first image and the second image.

[0056] The positional relationship between the two cameras during shooting can refer to Figure 2 , wherein Q is a point in the scene, O1 is the optical center of camera 1, O2 is the optical center of camera 2, I1 is the first image, I2 is the second image, q is the feature point corresponding to the spatial point Q on I1, q' is the feature point corresponding to the spatial point Q on I2, the baseline is the line connecting O1 and O2, the same plane where point Q, point O1 and point O2 are located is the epipolar plane, the intersection line of the epipolar plane and I1 is the epipolar line L1, and the intersection line of the epipolar plane and I2 is the epipolar line L2. The schematic diagram of the epipolar line can refer to Figure 3 .

[0057] Step S200, calculating a corresponding target intrinsic matrix according to the first image and the second image.

[0058] Specifically, when the scene is shot by different camera pairs, there is certainly some overlap, so there is a certain correspondence between the two, namely the epipolar geometry, as shown in Figure 2 It contains the two camera rotation matrix R and translation matrix T in the physical space. T and R describe the relative position of a camera relative to another camera in the global coordinate system. The relative relationship between the cameras can be described by the rotation information and the translation information between the two, and the rotation information is expressed in the form of a rotation matrix, and the translation information is expressed in the form of a translation matrix, and the intrinsic matrix is equal to the cross product of the rotation matrix R and the translation matrix [t] x Therefore, if the coordinates of point q in the coordinate system of the camera corresponding to the first image are represented as (x1, y1, z1), and the coordinates of q' in the coordinate system of the camera corresponding to the second image are represented as (x2, y2, z2), then according to the definition of the intrinsic matrix, it can be known that the intrinsic matrix satisfies the formula q' T Eq = 0, where E is the intrinsic matrix, and T is the inverse matrix. Where [t] x is represented as

[0059] Therefore, the intrinsic matrix corresponding to the first image and the second image, i.e., the target intrinsic matrix, can be calculated according to the different coordinates of the same point in the corresponding space between the first image and the second image.

[0060] Further, the method for calculating the intrinsic matrix in the embodiment is:

[0061] Step S210, extracting feature points in the first image and the second image to generate a first feature point set and a second feature point set.

[0062] Specifically, referring to Figure 4 , the two points connected by the line in the figure are a pair of feature point pairs. The feature points can be extracted from the first image and the second image according to the numerical values of each pixel point in the first image and the second image to generate a feature point pair set. There are many feature point matching extraction methods at present, such as Scale Invariant Feature Transform (SIFT), SpeededUp Robust Features (SURF), etc. The method for extracting feature points from the first image and the second image in the embodiment is not limited.

[0063] Further, in order to ensure the accuracy and effectiveness of the extracted feature points, in the embodiment, the process of extracting the feature points is:

[0064] Step S211, performing grayscale processing on the first image and the second image to generate a first grayscale image and a second grayscale image.

[0065] Specifically, since images captured by a typical camera are color images, such as RGB format images, and the RGB components do not reflect the characteristics of the image, but only represent color adjustments based on optical principles, and color images have larger file sizes than grayscale images, the first and second images are first converted to grayscale to facilitate subsequent calculations. Grayscale conversion methods include component analysis, maximum value analysis, and balanced mean analysis, etc. The method used for grayscale conversion does not affect the real-time performance of this embodiment, so the various methods will not be described here.

[0066] Step S212: Correct the first grayscale image according to the preset first distortion coefficient to generate a first corrected image, and correct the second grayscale image according to the preset second distortion coefficient to generate a second corrected image, wherein the first distortion coefficient is the distortion coefficient of the first camera, and the second distortion coefficient is the distortion coefficient of the second camera.

[0067] Specifically, camera distortion is mainly caused by the distortion produced by its optical lenses, which is related to the lens used in the camera itself and the lack of light at the installation location. After a camera captures an image, due to its inherent characteristics, the image will exhibit a certain degree of distortion, such as linear distortion. Therefore, even at the same location, images taken by different cameras will show feature points in different positions. Therefore, after obtaining the first grayscale image and the second grayscale image, the distortion coefficients of the corresponding first and second cameras (i.e., the first distortion coefficient and the second distortion coefficient) are used to correct both images to eliminate the effects of distortion, thereby generating the first corrected image and the second corrected image.

[0068] Step S213: Extract feature points from the first corrected image to generate a corresponding first feature point set; and extract feature points from the second corrected image to generate a corresponding second feature point set.

[0069] Step S220: Match the first feature point set and the second feature point set to generate several feature point pairs.

[0070] Specifically, there are many matching methods, such as the fast nearest neighbor search algorithm and brute-force matching. Taking brute-force matching as an example, we first match the first and second feature point sets. Specifically, based on the descriptors of each feature point in the first feature point set, we sequentially search for the feature points in the second feature point set whose descriptors are closest to these descriptors, thus obtaining candidate feature point pairs.

[0071] Then, mismatch elimination is performed to obtain the desired feature point pairs from the candidate feature point pairs. Mismatch elimination may include elimination based on homography tests, cross-match elimination, and ratio elimination. To ensure the accuracy of subsequent optical axis deviation measurement based on feature points, this embodiment pre-sets a standard value for the number of feature point pairs obtained, denoted as K, where K is a positive integer greater than 5.

[0072] Step S230: Calculate the target eigenvalue matrix between the first image and the second image based on the feature point pairs.

[0073] Specifically, from formula q′ T Since Eq = 0, once the feature point pairs are determined, the formula can be solved based on the feature point pairs to obtain the target eigenvalue matrix between the first image and the second image.

[0074] Furthermore, due to the large number of feature points, there may be a certain number of outliers. To reduce the impact of these outliers on the calculation results, the calculation process for the intrinsic matrix is ​​based on the Random Sample Consensus (RANSAC) algorithm, as follows:

[0075] Step S231: According to the preset fitting rules, the feature point pair is fitted and calculated to generate the corresponding Nth eigenma matrix and Nth interior point value.

[0076] Specifically, a fitting rule is pre-defined, which includes the number of feature point pairs calculated each time, the criteria for determining the feature point pairs used in the calculation, and a fitting threshold, i.e., the number of fitting operations to be performed in the entire calculation process. Here, N is a positive integer. Using this fitting rule, feature point pairs can be fitted to generate the corresponding Nth eigenvalue matrix and Nth interior point value. N is the iteration number, and when the fitting calculation is performed for the first time, N equals 1. Therefore, the first fitting calculation yields the first eigenvalue matrix and the first interior point value. Fitting calculation refers to calculating the corresponding eigenvalue matrix and interior point value based on the fitting between feature point pairs. The Nth interior point value refers to the number of feature point pairs that can be matched by feature points in the first and second feature point sets based on the Nth eigenvalue matrix. For example, if the number of matched feature point pairs is 3 based on the Nth eigenvalue matrix, then the Nth interior point value corresponding to the Nth eigenvalue matrix is ​​3.

[0077] Furthermore, the matching method can be determined based on the error between feature points. According to the preset error formula, the feature point error value between the feature point pairs to be matched is calculated. If the feature point error value is within the preset feature point error value threshold range, then the two feature points are considered to be matched.

[0078] The error formula used in this embodiment for the error value between feature points is as follows:

[0079] in, and Representing vectors Eq respectively i The square of the j-th element, and the sum vector Eq′ i The square of the j-th element, where j equals 1 and 2.

[0080] Furthermore, this error formula can also be used to evaluate the quality of the calculated Nth eigenma matrix, q i and q′ i For each Nth eigenma matrix, the eigenma matrix error value is calculated using the error formula. If the eigenma matrix error value is greater than a preset eigenma matrix error threshold, the Nth eigenma matrix is ​​deleted.

[0081] Further, step S231 includes:

[0082] Step S2311: Select feature point pairs in the feature point set that are equal to the preset number of test points as test feature point pairs.

[0083] Specifically, in epipolar geometry, the correspondence between poles, polar planes, and epipolar lines can be described using epipolar geometry. To describe epipolar geometry using matrices for ease of calculation, a fundamental matrix can be used. The fundamental matrix is ​​obtained by adding the intrinsic parameters of the two cameras to the eigenvalue matrix. The formula for the fundamental matrix is: Where F represents the fundamental matrix, K1 and K2 represent the intrinsic parameters of the first and second cameras, respectively, R represents the rotation matrix, and [t] x Let E represent the translation matrix, and according to the definition of eigenma matrices, E = [t]. x R. Because of [t] x The eigenma has degrees of freedom in the x, y, and z axes of the coordinate system, and the rotation matrix R has 3 degrees of freedom. Therefore, the highest degree of freedom of the eigenma is 6. At the same time, it satisfies the scale equivalence constraint, so its degree of freedom is 5.

[0084] Therefore, in this embodiment, the preset number of tests is 5, and 5 feature point pairs are randomly selected as test feature point pairs. In addition, before this step, feature point pairs can be eliminated based on methods such as the coordinate distance between them to improve the effectiveness of the test feature point pairs.

[0085] Step S2312: Calculate the corresponding Nth eigenma matrix and Nth interior point value based on the test feature point pair.

[0086] Specifically, based on the test feature point pair, the formula q′ T Solving for Eq=0 yields the eigenvalue matrix between the first and second images, which is taken as the Nth eigenvalue matrix. Five test point pairs are grouped together, with the coordinates of feature points from the same feature point set as the dependent variable and the coordinates of feature points from another feature point set as the independent variable. Solving this formula yields the corresponding eigenvalue matrix, which is taken as the first eigenvalue matrix. Then, based on the first eigenvalue matrix, the number of feature points in each feature point pair that match the first eigenvalue matrix is ​​determined, and this number is taken as the corresponding first interior point value.

[0087] Furthermore, the specific calculation rules are as follows:

[0088] A1. Write the test feature points into a preset matrix template to generate a test matrix.

[0089] Specifically, from formula q′ T Given Eq = 0 and the degrees of freedom of the eigenma matrix being 5, it can be concluded that the target eigenma matrix and the corresponding feature points satisfy... Specifically, when the number of input feature point pairs is 5,

[0090] The five selected test feature points are written into a preset matrix template. In this matrix, a 5x9 matrix can be obtained, which is the test matrix.

[0091] A2. Based on the test matrix, calculate the null space corresponding to the test matrix to obtain the Nth eigenvalue matrix. Specifically, N is a positive integer, which can be determined from the size of the test matrix, and according to the formula... It can be seen that the eigenvalue matrix corresponding to the test matrix is ​​the null space of the test matrix. Therefore, the null space of the test matrix is ​​calculated to obtain the corresponding Nth eigenvalue matrix. Since five dependent variables and five independent variables are used in the formula, this embodiment provides a fast solution method, which is as follows:

[0092] Based on the test matrix, the null space corresponding to the test matrix is ​​eliminated to obtain the corresponding system equations;

[0093] Based on the preset additional equations, the system equations are eliminated to generate an intermediate matrix;

[0094] Solve the intermediate matrix to generate the corresponding intermediate values;

[0095] Based on the intermediate value, the null space is solved to generate the Nth eigenvalue matrix.

[0096] Specifically, the null space of the test matrix can be represented by four vectors, denoted as follows in this embodiment: and Based on the size of the test matrix, each vector can be represented by a 3x3 matrix X, Y, Z, and W. Therefore, the eigenvalue matrix corresponding to the test matrix can be represented by these four matrices, i.e., E = xX + yY + zZ + wW. Since the eigenvalue matrix is ​​scale-invariant, one of its vectors can be represented by 1. In this embodiment, we set the variable w = 1.

[0097] Since the eigenma matrix has the property that its two non-zero singular values ​​are equal, the eigenma matrix satisfies the formula...

[0098] Substituting this formula into the equation E=xX+yY+zZ+wW, we obtain 9 equations in terms of x, y, and z, each of which is a cubic equation in three variables. To simplify the solution of the cubic equations in three variables, we eliminate variables to generate a system of equations.

[0099] Equation elimination refers to transforming multiple equations containing multiple unknowns into a single equation containing only one unknown. The process involves finding one unknown first, and then finding the others. In this embodiment, the unknown z is chosen as the object to be solved first.

[0100] In this embodiment, Gaussian-Jordan elimination is preferably used to eliminate variables in the equation. Gaussian-Jordan elimination ultimately yields a simplified row echelon matrix form, which has a fast elimination rate. Figure 5 This refers to the simplified row echelon matrix form after elimination in this embodiment, i.e., the system equations. Here, "﹒" represents the variable value, and [N] represents an Nth-degree polynomial with respect to the variable z. Each row represents a sub-term of the polynomial corresponding to the ternary cubic equation. For the first column, it corresponds to equations 1 to 9. 〈a〉 represents the first equation, and so on, with 〈j〉 representing the 9th equation.

[0101] To expedite the solution process, further elimination can be performed on the system equations. In this embodiment, this elimination is achieved using additional equations. The additional equations are predefined as follows:

[0102] <k>≡<e>-z<f>;

[0103] <l>≡<g>-z<h>;

[0104] <m>≡-z<j>.

[0105] Based on this additional equation, the system equations can be eliminated to obtain a polynomial in terms of the variable z. Representing this polynomial using an intermediate matrix, we get:

[0106] B x y 1 〈k〉 [3] [3] [4] 〈l〉 [3] [3] [4] 〈m〉 [3] [3] [4]

[0107] Since the intermediate matrix is ​​obtained by eliminating variables from the system equations, and the system equations contain variables x, y, and z, with variable w = 1, the intermediate matrix can be solved based on the aforementioned test feature points to generate intermediate values. There are many calculation methods; this embodiment provides a relatively fast solution. From the formula q′... T Since Eq = 0, the null space of this intermediate matrix is ​​the vector [xy 1]. T Therefore, the matrix is ​​transformed into a tenth-degree polynomial, where the variable in the polynomial is n, and n satisfies 10 =1 and <n> ≡ det(B). Therefore, the adjoint matrix corresponding to the intermediate matrix can be expressed as: The eigenvalues ​​of the adjoint matrix are the solutions to the intermediate matrix. By testing the matrix, the eigenvalues ​​of the adjoint matrix can be solved. These eigenvalues ​​are the values ​​of the unique variable z listed as the intermediate matrix in this embodiment, and variables x and y can be represented by variable z. Since for any eigenvalue, substituting it into the formula E = xX + yY + zZ + wW yields the corresponding Nth eigenvalue matrix.

[0108] Step S232: When the value of the Nth inner point is greater than the preset inner point number threshold, the value of the Nth inner point is used as the inner point number threshold, and the preset iteration number is incremented by one, wherein the iteration number is zero when the fitting calculation is performed for the first time.

[0109] Specifically, a threshold value for interior points is pre-set, for example, 500. Then, each first interior point value is compared with the threshold value. If there is a first interior point value that is larger than the threshold value, it means that the calculation is effective. If the original threshold value is still used as the benchmark to evaluate the interior point values ​​obtained in subsequent calculations, the best result obtained after iterative calculations may be the first interior point value. The interior point values ​​obtained in subsequent calculations should be compared with the first interior point value as the benchmark to ensure that each calculation in the iterative calculation is a valid calculation.

[0110] Furthermore, if the value of the first interior point exceeds the interior point count threshold, the iteration count is updated from 0 to 1, indicating that this calculation is valid. In subsequent iterations, whenever the value of the Nth interior point exceeds the interior point count threshold, the iteration count is increased to account for the valid calculations during the computation process, preventing the calculation from going on indefinitely.

[0111] Step S233: Iteratively calculate the Nth eigenma matrix and update the iteration number according to the Nth interior point value until the iteration number is greater than a preset iteration number threshold.

[0112] Specifically, the Nth eigenvalue matrix and its corresponding Nth interior point value are calculated iteratively, and the number of iterations is adjusted based on the relationship between the Nth interior point value and the interior point number threshold. This continues until the number of iterations exceeds a preset threshold. Since the number of iterations in this embodiment is not the actual number of iterations, but rather the number of valid calculations among all calculations, the calculated result is closer to the optimal value, and the calculation process is faster.

[0113] It is worth noting that, based on the RANSAC algorithm, the threshold for the number of iterations is not constant. Since this is existing technology, it will not be elaborated upon here.

[0114] Step S234: Determine the eigenma corresponding to the largest value of the Nth interior point as the target eigenma.

[0115] Specifically, when the iterative calculation ends, that is, when the updated iteration number is greater than the preset iteration number threshold, the currently calculated Mth eigenma matrix and the Mth interior point value can be obtained. The eigenma matrix corresponding to the Nth interior point value with the largest value is selected as the target eigenma matrix. For example, if the Mth interior point value is the largest, then the Mth interior point value is the largest interior point value in the entire calculation process. In other words, based on the Mth eigenma matrix, the number of matching feature point pairs is the largest. Therefore, the Mth eigenma matrix is ​​used as the target eigenma matrix.

[0116] Step S300: Calculate the optical axis deviation value corresponding to the first image and the second image based on the target intrinsic matrix.

[0117] Specifically, after obtaining the target eigenvalue matrix, since the target eigenvalue matrix can represent the transformation relationship between the first image and the second image, satisfying the formula E = [t] x R, where the translation matrix represents the translation relationship between the first and second images, including the translation direction and translation distance, and the rotation matrix describes the rotation relationship between the first and second images, including the selection angle and rotation direction. Therefore, the rotation angle between the first and second images can be calculated based on the target intrinsic matrix. This rotation angle is the optical axis deviation value. When calculating the optical axis deviation value corresponding to the target intrinsic matrix, the target intrinsic matrix is ​​first split according to this formula. In this embodiment, the splitting scheme is as follows:

[0118] Step S310: Decompose the target eigenma matrix to obtain candidate translation matrix and candidate rotation matrix.

[0119] Specifically, the intrinsic matrix is ​​first decomposed to obtain two pairs of translation and rotation matrices. To reduce the amount of computational data, a dimensionality reduction algorithm is used to decompose the target intrinsic matrix. Algorithms that can be used include Principal Component Analysis (PCA) and Singular Value Decomposition (SVD). Taking SVD as an example, suppose the target intrinsic matrix, after decomposition using the SVD algorithm, yields E = U∑V T Where U and V are orthogonal matrices, and ∑ is the singular value matrix. Based on the intrinsic properties of the target eigenvalue matrix, we can derive... in It is a diagonal matrix.

[0120] According to the SVD decomposition results, for each target eigenma, there exist two pairs of possible translation and rotation matrices corresponding to it, which can be expressed as follows:

[0121] as well as

[0122]

[0123] in This represents the rotation matrix obtained by rotating 90° along the Z-axis of the camera coordinate system of a certain camera.

[0124] Step S320: Determine the target rotation matrix in the candidate rotation matrix based on the depth values ​​corresponding to the candidate rotation matrix and the candidate translation matrix.

[0125] Specifically, since -E and E are equivalent, taking a negative sign for any t will yield the same result. Therefore, when decomposing E to t, R, there are a total of 4 possible solutions. (See also...) Figure 6 Given that the projections of spatial point Q onto cameras O1 and O2 are O1' and O2', and keeping the projected point, Q, unchanged, the four subplots in the diagram correspond to different scenarios: all depth values ​​are positive, all depth values ​​are negative, one positive and one negative depth value, and one negative and one positive depth value. Since depth values ​​are not typically assigned in real-world scenarios, we choose... Figure 6 The first case, where all depth values ​​are positive, corresponds to the target rotation matrix.

[0126] Step S330: Perform Euler angle transformation on the target rotation matrix to generate Euler angle values, and use the Euler angle values ​​as the optical axis deviation values ​​corresponding to the first image and the second image.

[0127] Specifically, once the target rotation angle is determined, the rotation angle can be determined. Conventional rotation angles only have magnitude, not direction. Euler angles define a set of rotation sequences for an object; rotating by a certain number of degrees around an axis will yield different results under different rotation sequences. This embodiment uses a rotation sequence from x to y, then from y to z. The plane containing the two cameras, the plane containing the baseline, and the plane containing the optical axis constitute three coordinate axes.

[0128] Generally, rotation of any dimension can be described as the product of a coordinate vector and a square matrix of appropriate size. Ultimately, rotation is equivalent to introducing a new description of the position of a point in a new coordinate system. Rotating the coordinate system by an angle θ is equivalent to rotating the target point in the opposite direction about the origin of this coordinate system by the same angle θ. Two-dimensional rotation can be represented using matrix multiplication as follows: Figure 7 As shown. A three-dimensional rotation can be decomposed into a two-dimensional rotation about each axis, where the metric representation of the rotation axis remains unchanged. Rotating α, β, γ about the x-axis, y-axis, and z-axis respectively, therefore the rotation matrix R is matrix R. x (α), R y (β), R z The product of (γ), the three are as follows:

[0129]

[0130]

[0131]

[0132] And R = R z (γ)R y (β)R x (α), therefore R can be expressed as:

[0133]

[0134] This matrix can be represented by a simplified matrix as follows:

[0135]

[0136] Therefore, after performing Euler angle transformation on the rotation matrix, the formula for the Euler angle values ​​is:

[0137]

[0138] According to this formula, after obtaining the rotation matrix, Euler angle transformation can be performed on it to generate the corresponding Euler angle values, which are then used as the optical axis deviation angles corresponding to the first and second images. Since the first and second images are captured by the first and second cameras respectively, this also means that the optical axis deviation angle between the first and second cameras is obtained.

[0139] Furthermore, such as Figure 8 As shown, based on the above-mentioned optical axis deviation measurement method, the present invention also provides a smart terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some components of the smart terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0140] In some embodiments, the memory 20 may be an internal storage unit of the smart terminal, such as a hard drive or memory. In other embodiments, the memory 20 may be an external storage device of the smart terminal, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the smart terminal. Further, the memory 20 may include both internal and external storage units of the smart terminal. The memory 20 is used to store application software and various types of data installed on the smart terminal, such as the program code installed on the smart terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an optical axis deviation measurement program 40, which can be executed by the processor 10 to implement the optical axis deviation measurement method of this application.

[0141] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the optical axis deviation measurement method.

[0142] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the smart terminal and to display a visual user interface. The components 10-30 of the smart terminal communicate with each other via a system bus.

[0143] In one embodiment, the optical axis deviation measurement program is implemented when the processor 10 executes the optical axis deviation measurement program 40 in the memory 20.

[0144] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an optical axis deviation measurement program, which, when executed by a processor, implements the steps of the optical axis deviation measurement method as described above.

[0145] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0146] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method of measuring optical axis deviation, characterized by, The optical axis deviation measurement method specifically comprises: Obtaining a first image and a second image photographed for the same scene; According to the first image and the second image, a corresponding target essential matrix is calculated; According to the target essential matrix, an optical axis deviation value corresponding to the first image and the second image is calculated; According to the first image and the second image, the target essential matrix is calculated, specifically comprising: Extracting feature points in the first image to generate a first feature point set, and extracting feature points in the second image to generate a second feature point set; Matching the first feature point set and the second feature point set to generate a plurality of pairs of feature points; According to the feature point pairs, the target essential matrix between the first image and the second image is calculated; According to the feature point pairs, the target essential matrix between the first image and the second image is calculated, specifically comprising: According to a preset fitting rule, the feature point pairs are fitted and calculated to generate a corresponding Nth essential matrix and an Nth inlier value, wherein N is a positive integer and when the fitting calculation is performed for the first time, N is equal to 1; When the Nth inlier value is greater than a preset inlier threshold value, the Nth inlier value is taken as the inlier threshold value, and the preset number of iterations is increased by one, wherein when the fitting calculation is performed for the first time, the number of iterations is zero; Iteratively calculating the Nth essential matrix and updating the number of iterations according to the Nth inlier value until the number of iterations is greater than a preset iteration threshold value; The essential matrix corresponding to the Nth inlier value with the largest value is determined as the target essential matrix.

2. The optical axis deviation measuring method according to claim 1, characterized by, The first image is an image photographed by a first camera, and the second image is an image photographed by a second camera; the feature points in the first image and the second image are extracted to generate a first feature point set and a second feature point set, specifically comprising: The first image and the second image are subjected to grayscale processing to generate a first grayscale image and a second grayscale image; According to a preset first distortion coefficient, the first grayscale image is corrected to generate a first corrected image, and according to a preset second distortion coefficient, the second grayscale image is corrected to generate a second corrected image, wherein the first distortion coefficient is the distortion coefficient of the first camera, and the second distortion coefficient is the distortion coefficient of the second camera; The first corrected image is subjected to feature point extraction to generate a corresponding first feature point set, and the second corrected image is subjected to feature point extraction to generate a corresponding second feature point set.

3. The optical axis deviation measuring method according to claim 1, characterized by, According to the preset fitting rule, the feature point pairs are fitted and calculated to generate a corresponding Nth essential matrix and an Nth inlier value, specifically comprising: Selecting a number of feature point pairs equal to a preset test number in the feature point set as test feature point pairs; According to the test feature point pairs, a corresponding Nth essential matrix and an Nth inlier value are calculated.

4. The optical axis deviation measuring method according to claim 3, characterized by, According to the test feature point pairs, a corresponding Nth essential matrix and an Nth inlier value are calculated, specifically comprising: The test feature points are written into a preset matrix template to generate a test matrix; According to the test matrix, a null space corresponding to the test matrix is calculated to obtain an Nth eigenmatrix; According to the Nth eigenmatrix, the number of matched feature points in the feature point pair is calculated to obtain the Nth in-point value.

5. The optical axis deviation measuring method according to claim 4, characterized by, The step of calculating the Nth eigenmatrix according to the test matrix specifically includes: According to the test matrix, a system equation corresponding to the null space of the test matrix is obtained by elimination; According to a preset additional equation, the system equation is eliminated to generate an intermediate matrix; The intermediate matrix is solved to generate a corresponding intermediate value; According to the intermediate value, the null space is solved to generate the Nth eigenmatrix.

6. A smart terminal, characterized by The intelligent terminal comprises a memory, a processor, and an optical axis deviation measurement program stored on the memory and executable on the processor, and the optical axis deviation measurement program, when executed by the processor, implements the steps of the optical axis deviation measurement method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an optical axis deviation measurement program, and the optical axis deviation measurement program, when executed by the processor, implements the steps of the optical axis deviation measurement method according to any one of claims 1-5.

Images