Camera calibration method and device, computer device and storage medium

Abstract

The application relates to a camera calibration method and device, computer equipment and a storage medium. The method comprises the following steps: for each camera in a dual-camera, acquiring a first matching point pair of a mark point on an epipolar rectification image and a mark point on a calibration object; mapping the mark point on the epipolar rectification image to an original image based on an epipolar rectification reflection mapping relationship to obtain an original image mapping point; the original image is an image captured by the camera; matching the original image mapping point with the mark point on the original image to obtain a second matching point pair; the second matching point pair comprises the mark point on the epipolar rectification image and the matching mark point on the original image; determining a target matching point pair corresponding to each camera based on the first matching point pair and the second matching point pair; the target matching point pair comprises the mark point on the original image and the matching mark point on the calibration object; and performing camera calibration based on the target matching point pair. The method can detect and correct device parameters in real time.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a camera calibration method, apparatus, computer device, and storage medium. Background Technology

[0002] For anyone who uses electronic devices, the factory settings are a crucial part of the setup. These settings include the device's parameter values, displayed variables, and operational logic. For handheld 3D binocular scanning devices, the calibration parameters of the binocular system at the factory are especially important.

[0003] Over the past few decades, camera calibration algorithms have been continuously updated and developed. Zhang Zhengyou improved upon traditional calibration methods such as image burning and recording, enabling simplified calibration of digital cameras through calculations simply by taking pictures with marked points. This method has been used and improved by many enthusiasts and researchers in related fields.

[0004] Based on feedback and research from many practitioners regarding Zhang Zhengyou's calibration method, it's clear that while it's simple to use and applicable to many everyday situations, such as camera image correction and 3D reconstruction where high precision isn't required, its accuracy cannot be further improved without modifying each step of the algorithm.

[0005] To achieve more complex and higher-precision binocular camera calibration, the traditional approach combines the iterative values ​​and iterative residuals of each calibration equation step to propose a higher-precision calibration algorithm. In terms of residual optimization, it can be used to improve the iterative accuracy of the overall equation to achieve more accurate calibration.

[0006] However, complex environments and improper operation can cause slight structural movements in binocular scanning equipment. In other words, the equipment becomes unstable, and there will often be slight movements in the relative positional relationship between the binocular camera and the scanning light source, which will cause the factory-calibrated parameters to not be well adapted to the equipment at this time. Summary of the Invention

[0007] Therefore, it is necessary to provide a camera calibration method, apparatus, computer equipment, and storage medium that can instantly detect and correct device parameters to address the aforementioned technical problems.

[0008] A marker matching method, the method comprising:

[0009] Acquire a scanned point cloud of a calibration object taken from the current scanning viewpoint; the calibration object contains marker points.

[0010] The calibration point cloud is matched with the scan point cloud to obtain a first matching point pair; the first matching point pair includes calibration matching points and matching scan matching points.

[0011] The scanned matching points are projected onto the camera's image coordinate system to obtain image mapping points;

[0012] Acquire an image of the calibration object obtained by the camera taking a picture of the calibration object;

[0013] The image mapping points are matched with the marker points on the calibration object image to obtain a second matching point pair; the second matching point pair includes the scanned matching point and the matching marker point on the calibration object image.

[0014] For the camera, based on the first matching point pair and the second matching point pair, a target matching point pair is determined that matches the marker point on the calibration object image with the marker point on the calibration object.

[0015] A marker matching device, the device comprising:

[0016] The scanning point cloud acquisition module is used to acquire the scanning point cloud obtained by photographing a calibration object from the current scanning viewpoint; the calibration object contains marker points;

[0017] The first matching module is used to match the calibration point cloud with the scan point cloud to obtain a first matching point pair; the first matching point pair includes calibration matching points and matching scan matching points.

[0018] The projection module is used to project the scanned matching points onto the camera's image coordinate system to obtain image mapping points;

[0019] The calibration object image acquisition module is used to acquire the calibration object image obtained by the camera taking pictures of the calibration object;

[0020] The second matching module is used to match the image mapping points with the marker points on the calibration object image to obtain a second matching point pair; the second matching point pair includes a scanned matching point and a matching marker point on the calibration object image.

[0021] The target matching module is used, for the camera, to determine, based on the first matching point pair and the second matching point pair, a target matching point pair that matches the marker point on the calibration object image with the marker point on the calibration object.

[0022] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of embodiments of camera calibration methods.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of embodiments of camera calibration methods.

[0024] The aforementioned camera calibration method, apparatus, computer equipment, and storage medium, for each camera, acquire a first matching point pair between the marker points on the epipolar correction image and the marker points on the calibration object. Then, based on the epipolar correction inverse mapping relationship, map the marker points on the epipolar correction image to the original image to determine a second matching point pair. Based on the first and second matching point pairs, determine the target matching point pair corresponding to each camera. By performing matching separately on the calibration object, the epipolar correction image, and the original image, as many point pairs as possible can be obtained. Furthermore, through multiple matchings, more accurate target marker point pairs can be obtained, allowing for real-time detection and correction of equipment parameters to achieve better scanning results. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating the application environment of a camera calibration method in one embodiment;

[0026] Figure 2 This is a flowchart illustrating a camera calibration method in one embodiment;

[0027] Figure 3 This is a schematic diagram of the camera calibration process based on target matching point pairs in one embodiment;

[0028] Figure 4 This is a schematic diagram of the process of matching the calibration point cloud with the scanned point cloud in one embodiment;

[0029] Figure 5 This is a schematic diagram of the features of the marker points in one embodiment;

[0030] Figure 6 This is a schematic diagram illustrating the features between three point pairs in one embodiment;

[0031] Figure 7 This is a schematic diagram of projecting scan matching points onto the camera's image coordinate system in one embodiment;

[0032] Figure 8 This is a structural block diagram of a camera calibration device in one embodiment;

[0033] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0034] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly. The connection can be a direct connection or an indirect connection.

[0037] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0038] The terms "first," "second," etc., used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first matching point pair may be referred to as a second matching point pair, and similarly, a second matching point pair may be referred to as a first matching point pair. Both the first matching point pair and the second matching point pair are matching point pairs, but they are not the same matching point pair.

[0039] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0040] The marker matching method provided in this application can be applied to, for example... Figure 1 In the application environment. Figure 1This is an application environment diagram of the marker matching method in one embodiment. The terminal device 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The number of cameras 120 can be two. The following embodiment uses a binocular camera as an example. The captured images are described here. A calibration object 130 is used for calibration, and its surface has preset marker points. The calibration object can be any object, such as a calibration board, a cube, a 3D head, etc. The preset marker points can be repositioned as needed.

[0041] In one embodiment, such as Figure 2 The diagram shown is a flowchart of a camera calibration method in one embodiment, including the following steps:

[0042] Step 202: For each camera in the dual-camera setup, acquire the first matching point pair that matches the marker point on the epipolar correction image with the marker point on the calibration object.

[0043] Epipolar correction refers to rotating, translating, and scaling the respective camera coordinate systems and focal planes to the target coordinate system, using a target coordinate system as the primary coordinate system, to form a new focal plane with a common focal length, ensuring that the epipolar lines are collinear and parallel to a coordinate axis of the focal plane. An epipolar-corrected image is an image captured by the camera and subjected to epipolar correction. The first matching point pair includes a marker point on the epipolar-corrected image and a marker point on a calibration object that matches the marker point on the epipolar-corrected image.

[0044] Specifically, each camera has a corresponding first matching point pair. The terminal device can use a marker point matching algorithm to match the first matching point pairs between the marker points on the epipolar corrected image and the marker points on the calibration object.

[0045] Step 204: Based on the epipolar correction inverse mapping relationship, map the marker points on the epipolar corrected image to the original image to obtain the original image mapping points; the original image is the image captured by the camera.

[0046] The original image is a camera-captured image that has not undergone epipolar correction. The epipolar correction mapping relationship refers to the relationship that transforms the original image into an epipolar-corrected image. The epipolar correction inverse mapping relationship refers to the relationship that transforms the epipolar-corrected image back into the original image.

[0047] Specifically, epipolar correction generates four tables, representing the sub-pixel coordinates of each pixel in the corrected image corresponding to the original captured image. Therefore, based on the logic of epipolar correction, a mapping table can be built in reverse, where pixels in the original image correspond to sub-pixels after epipolar correction. Two-dimensional bilinear interpolation can map the coordinates of the sub-pixel markers in the original image to the coordinate system of the epipolar-corrected image, thus finding the correspondence between each pixel in the epipolar-corrected image and the pixel in the original image. The terminal device, based on the inverse mapping relationship of epipolar correction, maps the markers on the epipolar-corrected image to the original image, obtaining the mapped points of the original image. In epipolar correction, matrix C is used to measure the transformation relationship between the original image coordinate system and the corrected image coordinate system. Matrix C' is used to measure the transformation relationship from the corrected image coordinate system to the original image coordinate system.

[0048] P ori =C'P rect

[0049]

[0050] Among them, P l P represents the sub-pixel coordinates of a point after epipolar correction. o P represents the coordinates of the image center after epipolar correction (the camera principal point coordinates in the new coordinate system), and f is the focal length. rect It refers to the original image, P ori It is an epipolar corrected image.

[0051] Step 206: Match the mapping points of the original image with the marker points on the original image to obtain a second matching point pair; the second matching point pair includes the marker points on the epipolar corrected image and the matching marker points on the original image.

[0052] The second matching point pair includes a marker point on the epipolar-corrected image and a marker point on the original image that matches the marker point on the epipolar-corrected image.

[0053] Specifically, the terminal device matches the mapping points of the original image with the marker points on the original image to obtain the marker points on the epipolar corrected image and the matching marker points on the original image.

[0054] Step 208: Based on the first matching point pair and the second matching point pair, determine the target matching point pair corresponding to each camera; the target matching point pair includes the marker points on the original image and the marker points on the matching calibration object.

[0055] Specifically, the first matching point pair includes a marker point on the epipolar-corrected image and a matching marker point on the calibration object. The second matching point pair includes a marker point on the epipolar-corrected image and a matching marker point on the original image. Therefore, the terminal device can determine the matching marker point on the original image from the second matching point pair based on the epipolar-corrected image marker point in the first matching point pair to obtain the target marker point pair. Alternatively, the terminal device can determine the matching marker point on the calibration object from the first matching point pair based on the epipolar-corrected image marker point in the second matching point pair to obtain the target marker point pair.

[0056] Step 210: Perform camera calibration based on target matching point pairs.

[0057] Specifically, the terminal device substitutes the target matching point pairs into the reference collinearity equation to obtain the target collinearity equation, and determines the camera parameters of each camera, such as camera extrinsic parameters and relative pose parameters, based on the target collinearity equation.

[0058] In this embodiment, for each camera, a first matching point pair is obtained where the marker points on the epipolar correction image match the marker points on the calibration object. Then, based on the epipolar correction inverse mapping relationship, the marker points on the epipolar correction image are mapped to the original image to determine a second matching point pair. Based on the first and second matching point pairs, the target matching point pair corresponding to each camera is determined. By matching the calibration object, the epipolar correction image, and the original image separately, as many point pairs as possible can be obtained. Furthermore, through multiple matching operations, more accurate target marker point pairs can be obtained. This allows for real-time detection and correction of device parameters, optimization of unstable factors in each scan, and the acquisition of better scanning results.

[0059] In one embodiment, matching the original image mapping points with the marker points on the original image to obtain a second matching point pair includes: adding a distortion coefficient to the original image mapping points, adjusting the distortion coefficient until the original image mapping points match the adjacent marker points on the original image, and obtaining a second matching point pair.

[0060] Specifically, for each original image mapping point, the terminal device adds a distortion coefficient to the original image mapping point and adjusts the distortion coefficient until the original image mapping point matches the adjacent marker point on the original image. Then, a second matching point pair is obtained where the marker point on the epipolar corrected image matches the marker point on the original image.

[0061] For P ori Adding distortion is necessary because the epipolar-corrected image undergoes distortion removal. To find the relationship between points in the original and corrected images, distortion needs to be added.

[0062]

[0063] x0, y0 are the pixel coordinates of the principal point of the camera's optical center projected onto the image plane.

[0064] Repeat the above equation to iteratively calculate P ori When it almost stops changing, it reaches a critical value, and then the correspondence between the pixel coordinates of the marker points of the original image and the epipolar corrected image can be obtained based on the coordinate deviation.

[0065] In this embodiment, since epipolar correction not only makes the images collinear but also corrects the image distortion, when mapping the epipolar corrected image to the original image, it is necessary to add distortion coefficients to the mapping points of the original image until they match the adjacent points of that point. This allows us to find the correct matching point pair and obtain the distortion coefficients, which can be used for other subsequent processing.

[0066] In one embodiment, camera calibration based on target matching point pairs includes: for each camera, performing single-camera calibration on the camera based on the target matching point pairs to obtain calibrated camera intrinsic parameters; comparing the calibrated camera intrinsic parameters with reference camera intrinsic parameters, determining the target camera intrinsic parameters based on the comparison result, and performing camera calibration based on the target camera intrinsic parameters and the target matching point pairs.

[0067] Among them, the calibration camera intrinsic parameters refer to those calculated based on the aforementioned target matching point pairs. The reference camera intrinsic parameters refer to the reference camera intrinsic parameters at the time the camera was manufactured.

[0068] Specifically, for each camera, the terminal device substitutes the target matching point pair into the reference collinearity equation to obtain the target collinearity equation, then determines the calibration camera intrinsic parameters based on the target collinearity equation, compares the calibration camera intrinsic parameters with the reference camera intrinsic parameters, and determines the target camera intrinsic parameters from the calibration camera intrinsic parameters and the reference camera intrinsic parameters based on the comparison results.

[0069] In this embodiment, the intrinsic parameters of the calibrated camera and the intrinsic parameters of the reference camera are compared. Based on the comparison results, the intrinsic parameters of the target camera are determined from the intrinsic parameters of the calibrated camera and the intrinsic parameters of the reference camera. The more accurate intrinsic parameters can be selected from them and subsequent camera calibration can be performed, thereby improving the accuracy of camera calibration.

[0070] In one embodiment, the target camera intrinsics are determined from the calibrated camera intrinsics and the reference camera intrinsics based on the comparison results, and camera calibration is performed based on the target camera intrinsics and the target matching point pair, including:

[0071] When the difference between the calibrated camera intrinsics and the reference camera intrinsics is within a preset range, the reference camera intrinsics are used as the target camera intrinsics, and camera calibration is performed based on the reference camera intrinsics and the target matching point pair.

[0072] Specifically, the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is within a preset range, meaning the absolute value of the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is less than or equal to a preset difference, or the ratio of the difference between the two to the calibrated camera intrinsic parameters is less than or equal to a preset ratio, etc. The opposite applies if the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is not within the preset range, which will not be elaborated upon here.

[0073] When the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is within the preset range, it indicates that the deviation of the calibrated camera intrinsic parameters from the reference camera intrinsic parameters is not large, and the camera still maintains system stability. At this time, the reference camera intrinsic parameters at the factory default are more accurate. Based on the reference camera intrinsic parameters and the target matching point pair, the relative pose parameters of the camera can be determined, and the camera extrinsic parameters can also be determined.

[0074] If the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is not within the preset range, it indicates that the calibrated camera intrinsic parameters deviate significantly from the reference camera intrinsic parameters. In this case, there is a high probability that there is a problem with the camera internally. It is recommended to return the camera to the factory for single-camera parameter calibration and not to perform subsequent camera calibration.

[0075] In this embodiment, when the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is within a preset range, the reference camera intrinsic parameters are more accurate. Therefore, the camera calibration can be performed accurately based on the reference camera intrinsic parameters and the target matching point pair.

[0076] In one embodiment, this application found that the purpose of the entire bi-camera calibration is to serve subsequent epipolar correction. During epipolar correction, most researchers prefer to use libraries like OpenCV and often choose the Zhang Zhengyou calibration method. However, the Zhang Zhengyou calibration method involves large rotations and translations of both cameras during epipolar correction, thus requiring high precision in the bi-camera calibration results. In this embodiment, however, it is noted that the principle of epipolar correction is to transform the distorted original image to the same epipolar line through distortion removal and transformation, forming a distortion-free corrected image where each corresponding point lies on an epipolar line. Therefore, how to rotate and transform the camera plane can be selected based on one's own model, requiring only subsequent image movement and cropping. Based on this, if the left camera plane remains almost unchanged while the right camera plane is transformed into the left camera coordinate system, the extrinsic parameter error of the left camera has almost zero impact on epipolar correction. Therefore, to simplify the bi-camera calibration model and accelerate computation, in each embodiment of this application, the intrinsic and extrinsic parameters of the left camera are fixed, and the relative pose of the two cameras and the extrinsic parameters of the right camera are processed in the adjustment model. However, it's important to note that the extrinsic parameters of the right camera can be replaced by the extrinsic parameters of the left camera and the relative pose of the cameras. Therefore, the entire model only needs to process the relative poses between the cameras, significantly reducing computational complexity. Since epipolar correction is primarily based on the left camera, the final error is very small, comparable to the results of all iterations. Understandably, depending on specific needs, it's also possible to prioritize the right camera and map the left camera plane into the right camera coordinate system.

[0077] like Figure 3 The diagram illustrates a process for camera calibration based on target matching point pairs in one embodiment, including:

[0078] Step 302: Substitute the target matching point pair into the reference collinearity equation to obtain the target collinearity equation.

[0079] The collinearity equation is a mathematical expression that states the object point, image point, and projection center lie on the same straight line. For single-camera calibration, the bundle adjustment model studies the collinearity equation, and the parameters of each camera must satisfy the following reference collinearity equation:

[0080]

[0081]

[0082] Where Δx and Δy are distortions, (x,y) are the pixel coordinates of the image, and (X,Y,Z) are the true coordinates of the marker on the calibration plate.

[0083] The target matching point pair contains the pixel coordinates (x, y) of the original image, as well as the coordinates (X, Y, Z) of the marker point on the matching object. Therefore, by substituting them, the collinearity equation of the target can be obtained.

[0084] Step 304: Based on the collinearity equation of the target, take the partial derivative of each matrix element in the first camera extrinsic matrix to obtain the first relational expression.

[0085] x0, y0 are the pixel coordinates of the principal point projected onto the image plane by the camera's optical center, and f is the camera's focal length. The extrinsic parameters a, b, and c are determined by the camera's attitude angles. ω, κ and camera position X s ,Y s Z s It's a decision. Understandably, distortion can be added to the collinearity equations, or it can be left out.

[0086] a, b, and c mentioned above, as well as the attitude angle ω, κ and camera position X s ,Y s Z s All are unknowns. The pixel coordinates of the image can be obtained by reading the image, and the actual coordinates of the calibration points on the calibration plate can also be obtained through other calibration methods.

[0087] The extrinsic parameter matrix of the first camera is determined by the extrinsic parameters of the first camera. Specifically, the R corresponding to the extrinsic parameter matrix of the first camera is... r for

[0088]

[0089] Where r represents the first camera. Actually, a in Rr... r1 This is equivalent to a1 in the collinearity equation… and so on, which will not be elaborated here. Similarly, all elements in the extrinsic parameter matrix of the first camera are also unknowns.

[0090] Since the collinear equations contain some matrix elements from the first camera extrinsic matrix, it is relatively easy and fast to find the partial derivatives.

[0091] Step 306: Calculate the partial derivatives of each relative pose parameter based on the relative pose parameter matrix, and determine the product with the preset second camera extrinsic parameter matrix to obtain the second relational expression.

[0092] Specifically, the relative pose parameter matrix is ​​based on the relative pose parameters. The relative pose parameter matrix Rc is determined to be...

[0093]

[0094] The preset second camera extrinsic matrix refers to the known camera extrinsic matrix pre-stored in the terminal device, as follows:

[0095]

[0096] R l The values ​​of parameters a, b, and c in the equation are all known.

[0097] The second relational expression represents the extrinsic parameter matrix R of the first camera. r The expression for the partial derivative of each matrix element with respect to the relative pose parameters, where the first camera extrinsic parameter matrix R... r Unknown. Then there is

[0098]

[0099] Step 308: Based on the product of the first relation and the second relation, determine the partial derivative function relation of the collinear equation with respect to the relative pose parameters.

[0100] Specifically, the expression of the collinearity equation is changed to

[0101]

[0102]

[0103] The partial derivatives of the collinear equations with respect to the relative pose parameters are expressed as follows:

[0104]

[0105] Among them, the partial derivative function relationship of the collinear equation with respect to the relative pose parameters can be decomposed based on the chain rule into the partial derivative of the collinear equation with respect to the first camera extrinsic matrix, and the partial derivative of the first camera extrinsic matrix with respect to the relative pose parameters.

[0106] Step 310: Based on the product of the increment and partial derivative function of the relative pose parameters, obtain the pixel deviation relationship between the true coordinates of the image and the collinearity equation of the target.

[0107] The pixel deviation formula can refer to either the pixel deviation formula in the x-direction or the pixel deviation formula in the y-direction. The calculation method for both is the same.

[0108] Specifically, based on the first-order Taylor expansion, partial derivatives with respect to each extrinsic parameter in the collinear equations can be obtained as follows:

[0109]

[0110]

[0111] in, △ξ is the extrinsic parameter of the first camera.r This refers to the increment of the first camera's extrinsic parameters. Furthermore, the relative pose parameter matrix has this relationship with the first camera's extrinsic parameter matrix and the unknown second camera's extrinsic parameter matrix.

[0112] R c =R l R' r

[0113] Therefore, the partial derivative of the above relationship with respect to the extrinsic parameters of the first camera can be transformed into taking the partial derivative with respect to the relative pose parameters between the two cameras, and can be rewritten as follows:

[0114]

[0115]

[0116] The left side of the equation represents the pixel deviation between the true x-coordinate of the image and the collinearity equation F, and the pixel deviation between the true y-coordinate of the image and the collinearity equation G.

[0117] Step 312: Iterate the pixel deviation relationship. When the iteration is complete, obtain the relative pose parameter values ​​of the first camera.

[0118] Specifically, the terminal device iterates the relative pose parameters in the pixel deviation relationship. When the iteration reaches a preset number of times or when the error of the pixel deviation relationship is less than or equal to a preset error, the relative pose parameter value of the first camera is obtained.

[0119] In this embodiment, analysis revealed that the coefficients in the pixel deviation relationship can be decomposed into a first relationship and a second relationship. Furthermore, the collinearity equation contains each matrix element of the first camera extrinsic parameter matrix, and the relative pose parameter matrix contains the relative pose parameters. By using matrix form to solve the problem, matrix partial differentials are used to replace the cumbersome calculation of partial derivatives of variables in the collinearity equation. The model is simple in form and very convenient to program, greatly reducing the computational complexity. It can quickly complete the calculation of the relative pose parameter values, and the results of binocular vision calibration are consistent with the calibration results of traditional methods.

[0120] In one embodiment, the pixel offset relationship is iterated, and when the iteration is complete, the relative pose parameter values ​​of the first camera are obtained, including:

[0121] Adjust the values ​​of the relative pose parameters in the pixel deviation formula. When the pixel deviation formula meets the error condition, obtain the values ​​of each relative pose parameter of the first camera.

[0122] Among them, satisfying the error condition can be that the value of the pixel deviation relationship is less than or equal to the preset error, etc.

[0123] Specifically, the terminal device adjusts the values ​​of the relative pose parameters in the pixel deviation relationship. When the value of the pixel deviation relationship is less than or equal to the preset error, or when the square value of the pixel deviation relationship is less than or equal to the preset error, the adjusted values ​​at this time are used as the relative pose parameter values ​​of the first camera.

[0124] In this embodiment, for the pixel deviation relationship, only the following problem needs to be optimized.

[0125]

[0126] The above formula uses the square of the pixel deviation relationship as an example for illustration, and the same applies to the y-coordinate values ​​of the image. It can be understood that the x and y coordinates can also be iterated independently using the pixel deviation relationship.

[0127] In this embodiment, the values ​​of the relative pose parameters in the pixel deviation relationship are adjusted. When the pixel deviation relationship satisfies the error condition, the values ​​of each relative pose parameter of the first camera are obtained. Only one model needs to be iterated and optimized, which can greatly reduce the computational complexity, improve the efficiency of determining the relative pose parameter values ​​of the camera, and thus improve the calibration efficiency.

[0128] In one embodiment, the method further includes: substituting the relative pose parameter values ​​into the relative pose parameter matrix to obtain the target relative pose matrix; and obtaining the target first camera extrinsic matrix based on the product of the transpose of the target relative pose matrix and a preset second camera extrinsic matrix.

[0129] Specifically, the values ​​in the target's first camera extrinsic parameter matrix are all known. The first camera extrinsic parameter matrix and the relative pose matrix can be converted to each other, i.e.:

[0130] R2=R' c R1

[0131] Then, by substituting the relative pose parameter values ​​into the relative pose parameter matrix, the target relative pose matrix is ​​obtained. Based on the product of the transpose of the target relative pose matrix and the preset second camera extrinsic matrix, the target first camera extrinsic matrix is ​​obtained.

[0132] In this embodiment, the relative pose parameter values ​​are substituted into the relative pose parameter matrix to obtain the target relative pose matrix; based on the product of the transpose of the target relative pose matrix and the preset second camera extrinsic parameter matrix, the target first camera extrinsic parameter matrix is ​​obtained, the extrinsic parameter matrices of the first camera and the second camera are obtained, the relative pose matrix between the first camera and the second camera is obtained, and combined with the known intrinsic parameters of the first camera and the second camera, all calibration parameters of the binocular camera are obtained.

[0133] In one embodiment, the relative pose parameter matrix contains relative pose parameters; the collinearity equation contains matrix elements of the first camera extrinsic parameter matrix; the partial derivative function relationship is determined based on the product of two partial derivative functions decomposed by the chain rule, one of which is the partial derivative of the collinearity equation with respect to the first camera extrinsic parameter matrix, and the other is the partial derivative of the first camera extrinsic parameter matrix with respect to the relative pose parameters.

[0134] Specifically, directly calculating the partial derivatives of the collinearity equations with respect to the relative pose parameters is computationally complex. However, analysis reveals that decomposing the equations using the chain rule yields two partial derivative functions, which are very simple to calculate, allowing for a rapid acquisition of the relative pose parameter values.

[0135] In one embodiment, such as Figure 4 The diagram shown illustrates a process for matching the calibrated point cloud with the scanned point cloud in one embodiment.

[0136] Step 402: Obtain the scan point cloud obtained by taking a picture of the calibration object from the current scanning viewpoint; the calibration object contains marker points.

[0137] The current scanning viewpoint refers to the viewpoint used when the multiple cameras to be calibrated are scanning. The scanned point cloud represents the three-dimensional coordinates of the marker points in the current camera coordinate system. The scanned point cloud also contains the positional and vector information of the marker points in the current scanning viewpoint.

[0138] Specifically, the binocular camera captures images of the calibration object from the current scanning viewpoint and synthesizes them to obtain a scanned point cloud. The terminal device then acquires the scanned point cloud obtained from the real-time capture of the calibration object from the current scanning viewpoint.

[0139] Step 404: Match the calibration point cloud with the scan point cloud to obtain a reference matching point pair; the reference matching point pair includes the calibration matching point and the matching scan matching point.

[0140] The calibration point cloud refers to the 3D coordinates of the marker points calculated based on binocular vision parameters after photographing the calibration object using precisely calibrated left and right cameras. Similarly, the calibration point cloud contains the positional and vector information of each marker point on the calibration object. The vector information can be the normal vector of the marker point.

[0141] A calibration object matching point is a marker point on a calibration object that matches a point in the scanned point cloud. A scan matching point is a marker point in the scanned point cloud that matches a marker point on the calibration object.

[0142] Specifically, the computer equipment uses a feature matching algorithm for marker points to match the calibrated point cloud with the scanned point cloud. When a match is successful, a reference matching point pair is obtained. Feature matching algorithms that can be referenced include ICP (Iterative Closest Point) algorithm, Fast-ICP algorithm, energy minimization feature matching, and topology-based point cloud matching.

[0143] Step 406: Project the scanned matching points onto the camera's image coordinate system to obtain the image mapping points.

[0144] In this context, projection refers to projecting the point cloud onto the image coordinate system. The points projected onto the image coordinate system from the scanned matching points are called image mapping points. The image coordinate system refers to the coordinate system used when the camera captures the image. The image coordinate system is also the coordinate system corresponding to the calibration object image.

[0145] Specifically, laser triangulation is generally used to calculate the coordinates of 3D points. Specifically, the ray connecting the pixels of the left camera and the ray connecting the corresponding pixels of the right camera intersect in 3D space. The intersection point is the 3D coordinate point corresponding to the two matching pixels. Then, using bundle adjustment, the scanned matching points of all reference matching point pairs can be projected onto the camera's image coordinate system to obtain image mapping points. For example, projecting the scanned matching points onto the left camera's image coordinate system yields the first matching point pair corresponding to the left camera. Similarly, projecting the scanned matching points onto the right camera's image coordinate system yields the first matching point pair corresponding to the right camera.

[0146] Step 408: Obtain the image of the calibration object taken by the camera.

[0147] Specifically, for a binocular camera, the image of the calibration object is acquired by each camera taking a picture of the calibration object.

[0148] Step 410: Match the image mapping points with the marker points on the calibration object image to obtain image matching point pairs; the image matching point pairs include the scan matching points and the matching marker points on the calibration object image.

[0149] The image matching point pair includes the scan matching point and the marker point on the matching calibration object image.

[0150] Specifically, the terminal device matches the image mapping points with the marker points on the calibration object image, and filters out unmatched point pairs to obtain image matching point pairs.

[0151] Step 412: For the camera, based on the reference matching point pair and the image matching point pair, determine the first matching point pair that matches the marker point on the calibration object image with the marker point on the calibration object.

[0152] The first matching point pair includes a matching marker point on the calibration object image and a marker point on the calibration object.

[0153] Specifically, the reference matching point pair includes matching scan matching points and calibration object matching points. The image matching point pair consists of scan matching points and matching marker points on the calibration object image. Therefore, the terminal device can determine the matching calibration object matching points from the reference matching point pair based on the scan matching points in the image matching point pair, thus obtaining a first matching point pair where the marker points on the calibration object image match with the marker points on the calibration object. Alternatively, the terminal device can determine the matching marker points on the calibration object image from the image matching point pair based on the scan matching points in the reference matching point pair, thus obtaining a first matching point pair where the marker points on the calibration object image match with the marker points on the calibration object.

[0154] In this embodiment, by matching the calibration point cloud with the scan point cloud, a successfully matched reference matching point pair is obtained. Then, based on the scan matching points in the reference matching point pair, the scan matching points are projected onto the corresponding camera, and the image mapping points are matched with the marker points on the calibration image to determine the image matching point pair. Based on the reference matching point pair and the image matching point pair, the first matching point pair is determined, which matches the marker points on the calibration image with the marker points on the calibration object. By matching the calibration object and the scan point cloud, and the scan point cloud and the calibration image separately, as many point pairs as possible can be obtained. Furthermore, through multiple matching, more accurate target marker point pairs can be obtained, optimizing the unstable factors of each device and obtaining better scanning results in the future.

[0155] In one embodiment, matching the calibration point cloud with the scanned point cloud to obtain a reference matching point pair includes: performing feature matching on the calibration point cloud and the scanned point cloud to obtain a point pair with successful feature matching; determining a first transformation relationship from the current scanning viewpoint to the calibration coordinate system based on the point pair with successful feature matching; mapping the marker point under the current scanning viewpoint to the calibration coordinate system based on the first transformation relationship to obtain the calibration mapping point; and matching the calibration mapping point with the marker point on the calibration object to obtain a reference matching point pair.

[0156] Feature matching can acquire features such as distance, normal vector, and the angle between the normal vectors of adjacent points. The first transformation relationship is the transformation from the current scanning viewpoint to the calibration object coordinate system. The first transformation relationship can include rotation matrix, translation matrix, and transformation vector.

[0157] Specifically, the terminal device acquires the features of marker points in the marker point cloud and the features of marker points in the scanned point cloud, and performs feature matching. When a match is successful, a point pair with successfully matched features is obtained. At this point, there are many point pairs with successfully matched features, including many noisy points. Several non-collinear point pairs can then be selected from the point pairs with successfully matched features, and a first transformation relationship from the current scanning viewpoint to the calibration object coordinate system can be calculated. Based on the first transformation relationship, the terminal device can map all marker points under the current scanning viewpoint to the calibration object coordinate system, that is, transform the coordinates of all marker points under the current scanning viewpoint to the coordinates under the calibration object coordinate system, obtaining the calibration object mapped points. The terminal device matches the calibration object mapped points with the marker points on the calibration object; when a match is successful, a reference matched point pair is obtained.

[0158] In this embodiment, feature matching is performed on the calibration point cloud and the scanned point cloud to obtain point pairs with successfully matched features. This means that a screening process is performed during the matching process to remove some noisy points with mismatched features. Based on the point pairs with successfully matched features, a first transformation relationship is determined, and mapping is performed again to map the marker points in the current viewpoint to the calibration coordinate system. The mapped points of the calibration object are matched with the marker points on the calibration object to obtain a successfully matched reference point pair. This allows previously unmatched marker points to be matched with their corresponding marker points, retaining as many valid points as possible. Furthermore, because the matching is performed through the first transformation relationship, more accurate matching point pairs are obtained, which greatly improves the accuracy of subsequent scanning.

[0159] In one embodiment, matching the calibrated object mapping points with the marker points on the calibrated object to obtain a reference matching point pair includes:

[0160] Match the mapping points of the calibration object with the marker points on the calibration object to obtain valid point pairs; valid point pairs include the marker points under the current scanning view and the matching marker points on the calibration object.

[0161] The second transformation relationship from the current scanning viewpoint to the calibration object coordinate system is determined based on the effective point pairs;

[0162] Based on the second transformation relationship, the marker point under the current scanning view is mapped to the coordinate system of the calibration object. When the mapped marker point matches the marker point on the calibration object, a reference matching point pair is obtained.

[0163] The valid point pairs include a marker point at the current scanning viewpoint and a marker point on a calibration object that matches the marker point at the current scanning viewpoint. The values ​​calculated by the second transformation relationship may differ from those calculated by the first transformation relationship. The first transformation relationship is calculated based on points with successfully matched features; the second transformation relationship is calculated based on valid point pairs with successfully matched features. The second transformation relationship provides more accurate values ​​compared to the first transformation relationship.

[0164] Specifically, the terminal device matches the mapped points of the calibration object with the marker points on the calibration object. When a match is successful, a valid point pair is obtained. Based on the valid point pair, the terminal device calculates the rotation and translation transformation relationship between the two point clouds using the Singular Value Decomposition (SVD) algorithm, i.e., the second transformation relationship. Based on the second transformation relationship, the terminal device maps the marker points from the current viewpoint to the calibration object coordinate system. The position of the mapped marker points in the calibration object coordinate system is more accurate than that mapped based on the first transformation relationship. Furthermore, matching the mapped marker points with the marker points on the calibration object yields a larger number of reference matching point pairs.

[0165] In this embodiment, by setting appropriate matching conditions and processes, the accuracy of matching point pairs can be improved while maintaining the number of point pairs.

[0166] In one embodiment, based on a second transformation relationship, the marker point under the current scanning view is mapped to the coordinate system of the calibration object. When the mapped marker point matches a marker point on the calibration object, a reference matching point pair is obtained, including:

[0167] Based on the second transformation relationship, the marker points under the current scanning view are mapped to the calibration object coordinate system, and the mapped marker points on the same plane are retained;

[0168] A reference matching point pair is obtained when the mapped marker points on the same plane match the marker points on the calibration object.

[0169] Specifically, after matching the marker points on the calibration object with the marker points under the current scanning viewpoint, and noting that the marker points are almost all located on the calibration object plane, the marker points under the current scanning viewpoint after rotation and translation should also be very close to this calibration object plane. Based on the second transformation relationship, the terminal device maps the marker points under the current scanning viewpoint to the calibration object coordinate system, discarding the mapped marker points not on the calibration object plane and retaining the mapped marker points on the same plane. The mapped marker points on the same plane are then matched with the marker points on the calibration object. When a mapped marker point on the same plane matches a marker point on the calibration object, the terminal device obtains a reference matching point pair.

[0170] In this embodiment, by filtering the mapped marker points and eliminating noise points that are not on the same plane, the obtained marker point pairs are more accurate, and the subsequent camera calibration is also more accurate.

[0171] In one embodiment, feature matching is performed between the calibration point cloud and the scanned point cloud to obtain point pairs with successfully matched features, including:

[0172] Obtain the normal vector of the calibration point cloud and the first distance between each point in the calibration point cloud and its adjacent points;

[0173] Determine the normal vector of each point in the scanned point cloud and the second distance between each point and its neighboring points;

[0174] The normal vector of the calibration point cloud is matched with the normal vector of each point in the scanned point cloud, and the first distance is matched with the second distance. When both the normal vector and the distance are successfully matched, a point pair with successfully matched features is obtained.

[0175] Specifically, the first distance is the distance between each point in the calibration point cloud and its adjacent points. The second distance refers to the distance between each point in the scanned point cloud and its adjacent points. The features of the calibration point cloud may include the normal vector of the calibration point cloud and the distance between each point in the calibration point cloud and its adjacent points; it may also include the angle between the normal vectors of each point in the calibration point cloud and its adjacent points.

[0176] The normal vectors of the calibration point cloud are matched with the normal vectors in the scanned point cloud. This matching may include matching the values ​​of the normal vectors, or matching the angles between the normal vectors of the matched point and its neighboring points. Generally, the normal vectors of all points in the calibration point cloud are identical. Optionally, the normal vectors of the points in the calibration point cloud may be different, in which case the normal vector of the calibration point cloud is the normal vector of each point. The normal vectors of the calibration point cloud and the distances between each point in the calibration point cloud and its neighboring points are pre-set and stored in the terminal device.

[0177] In this embodiment, the normal vector of the calibration point cloud is matched with the normal vector of each point in the scanned point cloud, and the first distance and the second distance are matched. When both are successfully matched, a point pair with successfully matched features is obtained, which can quickly filter out point pairs that meet the preliminary features.

[0178] In one embodiment, determining a first transformation relationship from the current scanning viewpoint to the calibration object coordinate system based on successfully matched point pairs includes:

[0179] Select three feature-matching point pairs from the feature-matching point pairs;

[0180] From three feature-matched point pairs, identify three marker points located in the same point cloud. When the three marker points located in the same point cloud are not collinear, determine whether the three feature-matched point pairs satisfy the congruent triangle condition.

[0181] When three feature-matched point pairs satisfy the condition of congruent triangles, the first transformation relationship from the current scanning viewpoint to the calibration object coordinate system is determined based on the three feature-matched point pairs.

[0182] Specifically, from the feature-matched point pairs, the terminal device randomly selects three feature-matched point pairs. Each feature-matched point pair includes a point in the scanned point cloud and a matching marker point on the calibration object. Therefore, the three marker points in the same point cloud can be either points in the scanned point cloud or points in the calibration object's point cloud. When the three marker points in the same point cloud are not collinear, it indicates that they can form a triangle. The terminal device determines whether the triangle formed by the three points in the scanned point cloud and the triangle formed by the three points in the calibration object's point cloud are congruent. When two triangles are congruent, meaning the three feature-matched point pairs satisfy the congruent triangle condition, the first transformation relationship from the current scanning viewpoint to the calibration object's coordinate system is determined based on the three feature-matched point pairs.

[0183] In this embodiment, the point pairs with successfully matched features may also include many noisy points, i.e., non-marking points, so further screening is required; the features of three collinear points are relatively few, so non-collinear points need to be selected for the determination of congruent triangles. When the three feature-matched point pairs satisfy the condition of congruent triangles, it indicates that the accuracy of the three feature-matched point pairs is high, so the accuracy of the calculated first transformation relationship is also high.

[0184] In one embodiment, taking the calibration object as the calibration board and 15 marker points as an example, feature matching is performed between the calibration object point cloud and the scanned point cloud, potentially resulting in 18 successfully matched point pairs. From these 18 pairs, any 3 non-collinear pairs are selected, and a first transformation relationship is calculated for mapping and matching, yielding 10 valid point pairs. Based on these 10 valid point pairs, a second transformation relationship is calculated, and the entire scanned point cloud is mapped to the calibration board coordinate system. Some points, after mapping, are not on the same calibration board plane and are discarded. Mapped points on the same calibration board plane are matched with marker points on the calibration board, ultimately obtaining 14-15 reference matched point pairs. In this embodiment, the accuracy of marker point matching is ensured while maximizing the number of matched point pairs.

[0185] In one embodiment, a marker matching method includes:

[0186] Step (a1): Obtain the scan point cloud obtained by taking a picture of the calibration object from the current scanning viewpoint; the calibration object contains marker points.

[0187] Step (a2) obtains the normal vector of the calibration point cloud and the first distance between each point in the calibration point cloud and its adjacent points.

[0188] Step (a3) ​​determines the normal vector of each point in the scanned point cloud and the second distance between each point and its neighboring points.

[0189] Step (a4) involves matching the normal vector of the calibration point cloud with the normal vector of each point in the scanned point cloud, and matching the first distance with the second distance. When both the normal vector and the distance are successfully matched, a point pair with successfully matched features is obtained.

[0190] Step (a5): Select three feature-matching point pairs from the feature-matching point pairs.

[0191] Step (a6): Determine three marker points located in the same point cloud from the three feature-matched point pairs. When the three marker points located in the same point cloud are not collinear, determine whether the three feature-matched point pairs satisfy the congruent triangle condition.

[0192] Step (a7): When the three feature-matching point pairs satisfy the congruent triangle condition, determine the first transformation relationship from the current scanning viewpoint to the calibration object coordinate system based on the three feature-matching point pairs.

[0193] Step (a8): Based on the first transformation relationship, the marker point under the current scanning view is mapped to the calibration object coordinate system to obtain the calibration object mapping point.

[0194] Step (a9) involves matching the mapping points of the calibration object with the marker points on the calibration object to obtain valid point pairs. Valid point pairs include the marker point in the current scanning view and the matching marker point on the calibration object.

[0195] Step (a10) determines the second transformation relationship from the current scanning viewpoint to the calibration object coordinate system based on the valid point pairs.

[0196] Step (a11): Based on the second transformation relationship, the marker points under the current scanning view are mapped to the calibration object coordinate system, and the mapped marker points on the same plane are retained.

[0197] Step (a12): When the mapped marker point on the same plane matches the marker point on the calibration object, a reference matching point pair is obtained; the reference matching point pair includes the calibration object matching point and the matching scan matching point.

[0198] Step (a13) involves projecting the scanned matching points onto the camera's image coordinate system to obtain the image mapping points.

[0199] Step (a14): Obtain the image of the calibration object taken by the camera.

[0200] Step (a15) involves matching the image mapping points with the marker points on the calibration object image to obtain image matching point pairs. Each image matching point pair includes the scanned matching point and the matching marker point on the calibration object image.

[0201] Step (a16): For each camera in the dual cameras, based on the reference matching point pair and the image matching point pair, determine the first matching point pair that matches the marker point on the calibration object image with the marker point on the calibration object.

[0202] Step (a17): Based on the epipolar correction inverse mapping relationship, the marker points on the epipolar corrected image are mapped to the original image to obtain the original image mapping points; the original image is the image captured by the camera.

[0203] Step (a18) involves adding distortion coefficients to the original image mapping points and adjusting the distortion coefficients until the original image mapping points match the adjacent marker points on the original images, thereby obtaining a second matching point pair; the second matching point pair includes the marker points on the epipolar corrected image and the matching marker points on the original images.

[0204] Step (a19) is to determine the target matching point pair for each camera based on the first matching point pair and the second matching point pair; the target matching point pair includes the marker points on the original image and the marker points on the matching calibration object.

[0205] Step (a20): For each camera, perform single-camera calibration based on the target matching point pair to obtain the intrinsic parameters of the calibrated camera.

[0206] Step (a21) compares the calibrated camera intrinsics with the reference camera intrinsics.

[0207] Step (a22): When the difference between the calibrated camera intrinsics and the reference camera intrinsics is within a preset range, the reference camera intrinsics are used as the target camera intrinsics.

[0208] Step (a23): When the difference between the calibrated camera intrinsics and the reference camera intrinsics is not within a preset range, the calibrated camera intrinsics are used as the target camera intrinsics.

[0209] Step (a24) involves substituting the target matching point pair and the target camera intrinsic parameters into the reference collinearity equation to obtain the target collinearity equation.

[0210] Step (a25): Based on the collinearity equation of the target, take the partial derivative of each matrix element in the first camera extrinsic matrix to obtain the first relational expression.

[0211] Step (a26) involves taking the partial derivative of each relative pose parameter based on the relative pose parameter matrix and determining the product with the preset second camera extrinsic parameter matrix to obtain the second relational expression.

[0212] Step (a27): Based on the product of the first relation and the second relation, determine the partial derivative function relation of the collinear equation with respect to the relative pose parameters.

[0213] Step (a28) is to obtain the pixel deviation relationship between the image true coordinates and the collinearity equation of the target based on the product of the increment of the relative pose parameters and the partial derivative function.

[0214] Step (a29) iterates the pixel deviation relationship. When the iteration is complete, the relative pose parameter values ​​of the first camera are obtained.

[0215] I. 3D Point Cloud Matching

[0216] Specifically, 3D point cloud matching is an essential step in scanning. The rigid body transformation matrix of the scanner needs to be calculated between every two frames (in multi-frame cases, the position of the first frame is generally chosen as the reference value, and the remaining frames are rotated and translated with the first frame as the target). There are many methods for 3D point cloud matching, with the most well-known and widely used being the ICP and Fast-ICP algorithms. These focus on feature matching between two unknown point clouds, such as distance features, normal vector features, local divergence features, matching rate, and matching error. In addition, these algorithms perform well in local precise matching, but the SVD and quaternion algorithms are more efficient in coarse matching. Often, many scanning devices adopt a combination of both methods. That is, coarse matching is first performed using SVD and quaternions, and then iterative matching is performed locally using ICP to improve matching accuracy. This method has been used in many papers. However, the embodiments in this application aim to match the data after post-calibration processing, not during the scanning stage, so the ICP algorithm is not required. Therefore, this paper uses the SVD method to match the marker points of unknown relationships.

[0217] Before marker matching, data preparation is required; in other words, useful features need to be calculated for matching. Figure 5 This is a schematic diagram of the marker features in one embodiment. The marker appearance used in this paper is as follows. Figure 5 Each marker is a circular surface, with a white circle inside and a black ring around it. It's clear that each marker's white circular area has a center and a radius, and a fixed normal vector for the entire marker's circular surface. Figure 5 The angle between n1 and n2 can also be used as a feature of the point.

[0218] In addition to the features of each marker point itself, the feature matching in this embodiment also focuses on the relative features between points. For example... Figure 5The distance between points is a necessary attribute, and the angle between the normal vectors of two marker points also affects the correspondence of matching points. According to the SVD algorithm, calculating the eigenvalues ​​and eigenvectors of the covariance matrix requires at least three non-collinear points. Therefore, every combination of three point clouds not only needs to be non-collinear, but also needs to form matching congruent triangles with the corresponding three point combinations. For example... Figure 6 The diagram shown is a feature diagram of three point pairs in one embodiment.

[0219] i) Calculate the coordinates and normal vector of each frame's marker point, then calculate the distance and angle between each marker point and its adjacent points and the normal vectors, and save all the calculated features.

[0220] ii) Perform feature filtering and matching on the scanned point cloud and the marker point cloud of the calibration object, and select point pairs that meet the conditions and have successfully matched features.

[0221] iii) Based on all the point pairs that have been successfully matched for features, select three points from the combination. First, determine whether they are collinear. If they are collinear, select three points again. If they are not collinear, determine whether the corresponding three point groups satisfy the condition of congruent triangles. If they are satisfied, calculate the rotation and translation matrix and vector through SVD.

[0222] iV) transforms all scan points to the calibration object space by calculating the rotation and translation matrices and vectors, and then filters all valid point pairs.

[0223] V) After filtering all valid points, use the SVD algorithm to recalculate the rotation and translation matrix using all valid points, then change the scan points again to filter out as many matching point pairs as possible, thus completing the process.

[0224] Before and during the matching process of 3D point clouds, there are some point screening processes. Some points are discarded during epipolar correction, some points cannot satisfy the collinearity equation well, and some points will not be on the calibration plane after matching.

[0225] For stereo systems after epipolar correction, laser triangulation is generally used to calculate the coordinates of 3D points. Specifically, the ray formed by connecting the pixels of the left camera's optical center and the ray formed by connecting the corresponding pixels of the right camera intersects in 3D space. The intersection point is the 3D coordinate point corresponding to the two matching pixels. Figure 7The diagram illustrates how the scanned matching points are projected onto the camera's image coordinate system in one embodiment. Therefore, after rotating and translating the matched point cloud, the pixels on the left camera (l) and the right camera (r) are obtained in reverse. These two points must satisfy the strong constraint of epipolar correction. This criterion can largely filter out points with large deviations or those exhibiting the same feature due to the shooting angle (i.e., two points projected to almost the same pixel on the left camera, but with large deviations in the right camera's pixel projection). After matching the scanned points with the calibration points, it is noted that the marker points are almost all located on the calibration plane. Therefore, the matched points after rotation and translation should also be very close to this calibration plane. This criterion can eliminate some points with large deviations and incorrectly extracted noise.

[0226] In simple terms, there are two conditions: First, the current viewpoint's scanned point cloud is rotated and translated to the calibration object's coordinate system, and all matching points should be very close to the calibration object's plane. Second, feature matching is used to obtain all possible matching points, and then the rotation and translation relationships are derived. The points of the calibration object are then transformed to the current scan viewpoint through this relationship. The matching points are then projected back to the pixels of the left and right cameras using laser triangulation, and these projections need to be very close to the pixel coordinates of the corresponding matching points in the current viewpoint's scanned point cloud.

[0227] The scanned point clouds m1, m2, and m3 are calculated from the matching pixel pairs (l1, r1), (l2, r2), and (l3, r3) of the left and right cameras, and they are matched with the three marker points M1, M2, and M3 on the calibration object. Clearly, in the diagram, (l4, r4) will yield P (in the following formula), which has a high probability of matching M4 on the calibration object. However, since this point will deviate significantly from the plane after rotation and translation, it will be discarded. For all matching points m... i We can calculate the plane S:(A,B,C,D) where Ax+By+Cz+D=0. That is, the meaning of =0 in the following formula is that they are on the same plane.

[0228] Based on the above explanation, all matching points should be close to this plane, and after rotating back to the calibration coordinate system, they should be close to the calibration plane. For this plane, the epipolar correction process will yield the left camera intrinsic parameter matrix PL in two corrected coordinate systems. Through geometric relationships, the following equation can be obtained:

[0229]

[0230]

[0231] Where l1 refers to the marker point on the calibration object image captured by the left camera, and w1 is the transformation relationship during projection.

[0232] The above process can generate sufficient correspondences between the pixels of the epipolar corrected image captured from the current viewpoint and the marker points of the calibration object:

[0233] Pair L}=(P l1 M j1 )∪(P l2 M j2 )∪...∪(P li M ji )∪....,i=1,2...n1,j i ∈{1,2,3,...,num},

[0234] and {Pair R}=(P r1 M k1 )∪(P r2 M k2 )∪...∪(P ri M ki )∪...,i=1,2...n2,k i ∈{1,2,3,...,num}

[0235] These two correspondences are crucial in subsequent processing. The inverse mapping of epipolar correction will be used to find the correspondence between the pixels of the original image captured by the camera at the current viewpoint and the marker points of the calibration object, thereby forming calibration data for secondary calibration.

[0236] II. Polar Correction Inverse Mapping

[0237] Epipolar correction generates four tables, representing the sub-pixel coordinates of each pixel in the corrected image corresponding to the original captured image. Therefore, based on the logic of epipolar correction, a mapping table can be built in reverse, where the pixels of the original image correspond to the sub-pixels after epipolar correction. Through two-dimensional bilinear interpolation, the coordinates of the sub-pixel markers in the original image can be mapped to the coordinate system of the epipolar corrected image, thus finding the correspondence between each pixel in the epipolar corrected image and the pixel in the original image.

[0238] In epipolar correction, matrix C' is used to measure the transformation relationship from the corrected image coordinate system to the original image coordinate system. That is...

[0239] P ori =C'P rect

[0240]

[0241] P here l P represents the sub-pixel coordinates of a point after epipolar correction. oΔx represents the coordinates of the image center after epipolar correction (the coordinates of the camera principal point in the new coordinate system), f is the focal length, and Δx and Δy are the distortions.

[0242] Then, for P ori The distortion is increased because the epipolar-corrected image has undergone distortion removal. If it is necessary to find the relationship between the marker points in the original image and the epipolar-corrected image, distortion needs to be increased. Specifically, by combining the above formula and formula (2) below, we can obtain...

[0243]

[0244] Repeatedly use formula (11) to iteratively calculate P ori When it almost stops changing, it reaches a critical value, denoted as P. final Therefore, based on the coordinate deviation, the correspondence between the pixel coordinates of the original image and the epipolar-corrected image can be obtained. Combining the matching point pairs of the marker points on the epipolar-corrected image and the marker points on the calibration object obtained in Part I, 3D point cloud matching, the target matching point pairs of the marker points on the original image and the marker points on the calibration object can be obtained.

[0245] III. Calibration Analysis

[0246] In dual-target calibration based on adjustment models, many papers adopt the approach of fixing the intrinsic parameters of the cameras, using the extrinsic parameters of the two cameras and the relative pose between the cameras as the required variables, and substituting all of them into the adjustment model for iteration. Finally, a set of variables is calculated as the final value by minimizing the residuals. Because it involves three iterative models that are optimized together, the whole process is quite cumbersome.

[0247] In this application, the study found that the purpose of the entire bi-camera calibration is to serve the subsequent epipolar correction. During epipolar correction, most researchers prefer to use libraries like OpenCV and often choose the Zhang Zhengyou calibration method. However, the Zhang Zhengyou calibration method involves large rotations and translations of both cameras during epipolar correction, thus requiring high precision in the bi-camera calibration results. This paper notes that the principle of epipolar correction is to transform and adjust the distorted original image to the same epipolar line, forming a distortion-free corrected image where each corresponding point lies on an epipolar line. Therefore, the rotation and transformation of the camera plane can be selected based on the model, requiring only subsequent image movement and cropping. Based on this, if the left camera plane remains almost unchanged while the right camera plane is transformed into the left camera coordinate system, the extrinsic parameter error of the left camera has almost zero impact on epipolar correction. Therefore, to simplify the bi-camera calibration model and accelerate computation, this paper fixes the intrinsic and extrinsic parameters of the left camera and processes the relative pose of the two cameras and the extrinsic parameters of the right camera in the adjustment model. However, it's important to note that the extrinsic parameters of the right camera can be replaced by the extrinsic parameters of the left camera and the relative pose of the cameras. Therefore, the entire model only needs to process the relative pose between the cameras, significantly reducing computational complexity. Furthermore, since epipolar correction primarily focuses on the left camera, the final error is very small, comparable to the results of all iterations.

[0248] The following detailed discussion begins with the collinearity equations of the cameras and is further elaborated using the two-dimensional DLT algorithm.

[0249] 1. Global minimum reprojection residual optimization with all parameters

[0250] For single-camera calibration, the bundle adjustment model studies the collinearity equations, and the parameters of each camera must satisfy the following model:

[0251]

[0252] Where Δx and Δy are distortions, (x,y) are the pixel coordinates of the image, and (X,Y,Z) are the true coordinates of the marker point on the calibration board. The selected distortion model is as follows:

[0253] Δx=(x-x0)(K1r 2 +K2r 4 )+P1[r 2 +2(x-x0) 2 ]+2P2(x-x0)(y-y0)+P3(x-x0)+P4(y-y0)

[0254] Δy=(y-y0)(K1r 2 +K2r 4 )+P2[r 2+2(y-y0) 2 ]+2P1(x-x0)(y-y0) (2)

[0255] K1, K2, P1, P2, P3, P4 are the camera distortion coefficients, x0, y0 are the pixel coordinates of the principal point of the camera's optical center projected onto the image plane, and f is the camera's focal length. These nine parameters are the intrinsic parameters of the camera model used in this paper. Based on model (1), the extrinsic parameters correspond to a, b, c, which are determined by the camera's attitude angles. ω, κ and camera position X s ,Y s Z s It was decided.

[0256] For a fully parameter-optimized bi-objective positioning model, the goal is to ensure that both cameras not only satisfy model (1), but also meet the following condition.

[0257] R c =R l R' r ,T c =R l (T r -T l (3)

[0258] Let the two equations in model (1) be F(x) and G(y), respectively. At this point, the camera's intrinsic parameters do not need to be considered, therefore the variables are... and These are the extrinsic parameters of the left camera and the right camera, respectively. In this embodiment, l represents the left camera, corresponding to the second camera; and r represents the right camera, corresponding to the first camera. During the optimization process, model (1) obtains the iterative equation (first-order Taylor expansion) for each point by taking the partial derivatives of the extrinsic parameters:

[0259]

[0260] Combining equations (3) and (4), the Lagrange multiplier optimization model for the global parameters is as follows:

[0261]

[0262] The traditional model (5) has high computational complexity, requires multiple iterations, and generates a large amount of computation. The embodiments of this application use another method to solve the same problem.

[0263] 2. The binocular residual optimization in this application embodiment is mainly based on monocular vision.

[0264] Based on the above description, this application proposes a binocular residual optimization model primarily based on monocular vision, and transforms the extrinsic parameters of a single camera to form a model that is only related to the relative pose of the two cameras. However, it should be noted that this operation is related to the subsequent epipolar correction step. That is, if the reference plane selected for epipolar correction is the pixel plane of the left camera, then the model in this paper is mathematically almost identical to step 1, but it is faster and less complex. If another plane is selected for epipolar correction, then the extrinsic parameters of both cameras are involved in the optimization, which is an aspect that needs to be considered in pursuit of accuracy.

[0265] Based on model (1), the partial derivatives of each extrinsic parameter in the adjustment model of the right camera can be obtained.

[0266]

[0267] in This is a simple transformation of model (1).

[0268] Note that in equation (3), the partial derivatives of equation (6-1) with respect to the extrinsic parameters of the right camera can be transformed into the partial derivatives of model (1) with respect to the relative pose between the two cameras, given that the relative pose is...

[0269] Therefore, equation (6-1) can be rewritten as follows:

[0270]

[0271] For equation (6-2), it is not necessary to deal with the complex problems in model (5), but only to optimize the following problem.

[0272]

[0273] Solving this problem is quite simple. In this embodiment, we only need to iterate over equation (7) to find the relative pose that satisfies the condition of very small error.

[0274] 3. Traditional methods for calculating binocular models

[0275] As explained above, optimizing model (7) is equivalent to iteratively obtaining the true value of equation (6-2). For equation (6-2), it is necessary to adjust the relative pose for model (1). Calculate the partial derivative for each component. Note that solving this problem requires knowing R in equation (3). c and T c ,in ω c and κ c The corresponding R c Translation vector T c They are respectively

[0276]

[0277] Given the extrinsic parameters of the right camera The corresponding R r and T r for

[0278]

[0279] Left camera external reference The corresponding R l and T l for

[0280]

[0281] The values ​​mentioned above are fixed constant values.

[0282] By combining formulas (3), (8), and (9-2) in the traditional way, we can obtain that each element of the right camera extrinsic parameter matrix and vector satisfies the following system of equations.

[0283]

[0284] The traditional method involves substituting formula (10) into model (1), which can convert all a... r ,b r ,c r ,X r ,Y r Z r use The nonlinear combination of the equations can be used to obtain the partial derivative equations of the collinear model (1) with respect to all relative pose variables. The numerical values ​​close to the true values ​​can be calculated by iteration.

[0285] In this process, calculating the partial derivatives of each parameter of the collinear equations for the relative pose relationship variables is extremely complex. It can be noted that in equation (10), there are a large number of variables, and all of them are nonlinear combination equations.

[0286] 4. Optimization methods used in the embodiments of this application

[0287] Therefore, in actual programming, it is necessary to find methods to reduce computational complexity. Through observation of the model, the core of formula (6-2) is to obtain the coefficient of each variable before its increment during iteration. That is, to obtain the coefficient of each... and Based on the chain rule of differentials, the derivation can be performed.

[0288]

[0289] Note that R on the right side of equation (11) rij R is the extrinsic parameter matrix of the right camera. r For each element, therefore, we need to obtain the right side of equation (11). To find the value of , we only need to obtain the entire matrix R. r right The matrix partial derivatives of each variable can be obtained. Based on the transformation relation equation (3), we can obtain...

[0290]

[0291] Therefore, in equation (11) It becomes

[0292]

[0293] That is, in R' c For each Find the partial derivative matrix of the variables, and then multiply the right side of the matrix by the left camera extrinsic parameter matrix R. l Taking the element in the i-th row and j-th column is... The value.

[0294] Then solve equation (11) After the value, The value is obtained by taking the partial derivative of the first equation in the collinear equation (1) with respect to the element in the i-th row and j-th column of the right camera extrinsic matrix. Thus, the value of each coefficient in equation (11) is obtained. It is not necessary to solve the complex nonlinear equation system combining model (1) and equation (10). It is only necessary to find the elements in the matrix based on equations (11) and (13). It should be noted that the partial derivative of each of the above matrices is just the derivative of a very simple trigonometric function, so the complexity is very low and the model is very intuitive. Thus, the model is completed. The matrix model is based on the collinear equation, i.e., model (1), and is composed of equations (11), (13) and (6-2). In this embodiment, by understanding the model and combining the chain rule and simple partial differential calculation, the computational complexity of dual-target calibration is greatly reduced, the calibration efficiency is improved, and fast dual-target calibration is achieved. Furthermore, through the above analysis, the implementation method is improved, and the obtained calibration results are no different from those of the traditional method.

[0295] In this embodiment, by matching the calibration point cloud with the scan point cloud, a successfully matched reference matching point pair is obtained. Then, based on the scan matching points in the reference matching point pair, the scan matching points are projected onto the corresponding camera, and the image mapping points are matched with the marker points on the calibration image to determine the image matching point pair. Based on the reference matching point pair and the image matching point pair, the first matching point pair is determined, which matches the marker points on the calibration image with the marker points on the calibration object. By matching the calibration object and the scan point cloud, and the scan point cloud and the calibration image separately, as many point pairs as possible can be obtained. Furthermore, through multiple matching, more accurate target marker point pairs can be obtained, optimizing the unstable factors of each device and obtaining better scanning results in the future.

[0296] It should be understood that, although the above Figures 2 to 4 In the flowchart, the steps are shown sequentially according to the arrows, and the steps in steps (a1) to (a29) are shown sequentially according to the labels. However, these steps are not necessarily executed in the order indicated by the arrows or numbers. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps; they can be executed in other orders. Furthermore, Figures 2 to 4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0297] In one embodiment, such as Figure 8 The diagram shown is a structural block diagram of a camera calibration device in one embodiment. Figure 8 A camera calibration device is provided, which can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes: a first matching point pair acquisition module 802, an epipolar correction inverse mapping module 804, a second matching point pair determination module 806, a target matching point pair acquisition module 808, and a calibration module 810, wherein:

[0298] The first matching point pair acquisition module 802 is used to acquire, for each of the dual cameras, a first matching point pair that matches the marker point on the epipolar correction image with the marker point on the calibration object.

[0299] The epipolar correction inverse mapping module 804 is used to map the marker points on the epipolar correction image to the original image based on the epipolar correction inverse mapping relationship, so as to obtain the original image mapping points; the original image is the image captured by the camera.

[0300] The second matching point pair determination module 806 is used to match the mapping points of the original image with the marker points on the original image to obtain a second matching point pair; the second matching point pair includes the marker points on the epipolar correction image and the matching marker points on the original image.

[0301] The target matching point pair acquisition module 808 is used to determine the target matching point pair corresponding to each camera based on the first matching point pair and the second matching point pair; the target matching point pair includes the marker points on the original image and the marker points on the matching calibration object;

[0302] The calibration module 810 is used for camera calibration based on target matching point pairs.

[0303] In this embodiment, for each camera, a first matching point pair is obtained that matches the marker points on the epipolar correction image with the marker points on the calibration object. Then, based on the epipolar correction inverse mapping relationship, the marker points on the epipolar correction image are mapped to the original image to determine the second matching point pair. Based on the first and second matching point pairs, the target matching point pair corresponding to each camera is determined. By matching the calibration object, the epipolar correction image, and the original image separately, as many point pairs as possible can be obtained. Furthermore, through multiple matchings, more accurate target marker point pairs can be obtained, allowing for real-time detection and correction of equipment parameters, optimization of unstable factors in each equipment scan, and subsequent acquisition of better scanning results.

[0304] In one embodiment, the second matching point pair determination module 806 is used to add distortion coefficients to the original image mapping points and adjust the distortion coefficients until the original image mapping points match the adjacent marker points on the original images to obtain the second matching point pair.

[0305] In this embodiment, since epipolar correction not only makes the images collinear but also corrects the image distortion, when mapping the epipolar corrected image to the original image, it is necessary to add distortion coefficients to the mapping points of the original image until they match the adjacent points of that point. This allows us to find the correct matching point pair and obtain the distortion coefficients, which can be used for other subsequent processing.

[0306] In one embodiment, the calibration module 810 is configured to perform single-camera calibration for each camera based on target matching point pairs to obtain calibrated camera intrinsic parameters; compare the calibrated camera intrinsic parameters with reference camera intrinsic parameters; determine the target camera intrinsic parameters based on the comparison results; and perform camera calibration based on the target camera intrinsic parameters and target matching point pairs.

[0307] In this embodiment, the intrinsic parameters of the calibrated camera and the intrinsic parameters of the reference camera are compared. Based on the comparison results, the intrinsic parameters of the target camera are determined from the intrinsic parameters of the calibrated camera and the intrinsic parameters of the reference camera. The more accurate intrinsic parameters can be selected from them and subsequent camera calibration can be performed, thereby improving the accuracy of camera calibration.

[0308] In one embodiment, the calibration module 810 is used to use the reference camera intrinsics as the target camera intrinsics when the difference between the calibrated camera intrinsics and the reference camera intrinsics is within a preset range, and to perform camera calibration based on the reference camera intrinsics and the target matching point pair.

[0309] In this embodiment, when the difference between the calibrated camera intrinsic parameters and the reference camera intrinsic parameters is within a preset range, the reference camera intrinsic parameters are more accurate. Therefore, the camera calibration can be performed accurately based on the reference camera intrinsic parameters and the target matching point pair.

[0310] In one embodiment, the calibration module 810 is used to substitute the target matching point pair into the reference collinearity equation to obtain the target collinearity equation; to obtain a first relational expression by taking the partial derivative of each matrix element in the first camera extrinsic parameter matrix based on the target collinearity equation; to obtain a second relational expression by taking the partial derivative of each relative pose parameter based on the relative pose parameter matrix and determining the product with the preset second camera extrinsic parameter matrix; to determine the partial derivative function relational expression of the collinearity equation with respect to the relative pose parameters based on the product of the increment of the relative pose parameters and the partial derivative function relational expression; to obtain the pixel deviation relational expression between the true coordinates of the image and the target collinearity equation; and to iterate the pixel deviation relational expression until the iteration is complete, thereby obtaining the values ​​of each relative pose parameter of the first camera.

[0311] In this embodiment, analysis revealed that the coefficients in the pixel deviation relationship can be decomposed into a first relationship and a second relationship. Furthermore, the collinearity equation contains each matrix element of the first camera extrinsic parameter matrix, and the relative pose parameter matrix contains the relative pose parameters. By using matrix form to solve the problem, matrix partial differentials are used to replace the cumbersome calculation of partial derivatives of variables in the collinearity equation. The model is simple in form and very convenient to program, greatly reducing the computational complexity. It can quickly complete the calculation of the relative pose parameter values, and the results of binocular vision calibration are consistent with the calibration results of traditional methods.

[0312] In one embodiment, the calibration module 810 is used to adjust the values ​​of the relative pose parameters in the pixel deviation relationship. When the pixel deviation relationship satisfies the error condition, the values ​​of each relative pose parameter of the first camera are obtained.

[0313] In this embodiment, the values ​​of the relative pose parameters in the pixel deviation relationship are adjusted. When the pixel deviation relationship satisfies the error condition, the values ​​of each relative pose parameter of the first camera are obtained. Only one model needs to be iterated and optimized, which can greatly reduce the computational complexity, improve the efficiency of determining the relative pose parameter values ​​of the camera, and thus improve the calibration efficiency.

[0314] In one embodiment, the calibration module 810 is further configured to substitute the relative pose parameter values ​​into the relative pose parameter matrix to obtain the target relative pose matrix; and to obtain the target first camera extrinsic matrix based on the product of the transpose of the target relative pose matrix and the preset second camera extrinsic matrix.

[0315] In this embodiment, the relative pose parameter values ​​are substituted into the relative pose parameter matrix to obtain the target relative pose matrix; based on the product of the transpose of the target relative pose matrix and the preset second camera extrinsic parameter matrix, the target first camera extrinsic parameter matrix is ​​obtained, the extrinsic parameter matrices of the first camera and the second camera are obtained, the relative pose matrix between the first camera and the second camera is obtained, and combined with the known intrinsic parameters of the first camera and the second camera, all calibration parameters of the binocular camera are obtained.

[0316] In one embodiment, the first matching point pair acquisition module 802 is used to acquire a scanned point cloud obtained by shooting a calibration object from the current scanning viewpoint; the calibration object contains marker points; the calibration object point cloud is matched with the scanned point cloud to obtain a reference matching point pair; the reference matching point pair includes a calibration object matching point and a matching scanned matching point; the scanned matching point is projected onto the camera's image coordinate system to obtain image mapping points; an image of the calibration object is acquired by the camera shooting the calibration object; the image mapping points are matched with marker points on the calibration object image to obtain an image matching point pair; the image matching point pair includes a scanned matching point and a matching marker point on the calibration object image; for the camera, based on the reference matching point pair and the image matching point pair, a first matching point pair is determined where the marker points on the calibration object image match the marker points on the calibration object.

[0317] In this embodiment, by matching the calibration point cloud with the scan point cloud, a successfully matched reference matching point pair is obtained. Then, based on the scan matching points in the reference matching point pair, the scan matching points are projected onto the corresponding camera, and the image mapping points are matched with the marker points on the calibration image to determine the image matching point pair. Based on the reference matching point pair and the image matching point pair, the first matching point pair is determined, which matches the marker points on the calibration image with the marker points on the calibration object. By matching the calibration object and the scan point cloud, and the scan point cloud and the calibration image separately, as many point pairs as possible can be obtained. Furthermore, through multiple matching, more accurate target marker point pairs can be obtained, optimizing the unstable factors of each device and obtaining better scanning results in the future.

[0318] In one embodiment, the first matching point pair acquisition module 802 is used to perform feature matching between the calibration point cloud and the scan point cloud to obtain point pairs with successful feature matching; determine a first transformation relationship from the current scanning viewpoint to the calibration coordinate system based on the point pairs with successful feature matching; map the marker points under the current scanning viewpoint to the calibration coordinate system based on the first transformation relationship to obtain the calibration mapping points; and match the calibration mapping points with the marker points on the calibration object to obtain reference matching point pairs.

[0319] In this embodiment, feature matching is performed on the calibration point cloud and the scanned point cloud to obtain point pairs with successfully matched features. This means that a screening process is performed during the matching process to remove some noisy points with mismatched features. Based on the point pairs with successfully matched features, a first transformation relationship is determined, and mapping is performed again to map the marker points in the current viewpoint to the calibration coordinate system. The mapped points of the calibration object are matched with the marker points on the calibration object to obtain a successfully matched reference point pair. This allows previously unmatched marker points to be matched with their corresponding marker points, retaining as many valid points as possible. Furthermore, because the matching is performed through the first transformation relationship, more accurate matching point pairs are obtained, which greatly improves the accuracy of subsequent scanning.

[0320] In one embodiment, the first matching point pair acquisition module 802 is used to match the calibration object mapping point with the marker point on the calibration object to obtain a valid point pair; the successfully matched valid point pair includes the marker point under the current scanning view and the matching marker point on the calibration object; based on the valid point pair, a second transformation relationship from the current scanning view to the calibration object coordinate system is determined; based on the second transformation relationship, the marker point under the current scanning view is mapped to the calibration object coordinate system, and when the mapped marker point matches the marker point on the calibration object, a reference matching point pair is obtained.

[0321] In this embodiment, by setting appropriate matching conditions and processes, the accuracy of matching point pairs can be improved while maintaining the number of point pairs.

[0322] In one embodiment, the first matching point pair acquisition module 802 is used to map the marker points under the current scanning view to the coordinate system of the calibration object based on the second transformation relationship, and retain the mapped marker points on the same plane; when the mapped marker points on the same plane match the marker points on the calibration object, a reference matching point pair is obtained.

[0323] In this embodiment, by filtering the mapped marker points and eliminating noise points that are not on the same plane, the obtained marker point pairs are more accurate, and the subsequent camera calibration is also more accurate.

[0324] In one embodiment, the first matching point pair acquisition module 802 is used to acquire the normal vector of the calibration point cloud and the first distance between each point in the calibration point cloud and its adjacent points; determine the normal vector of each point in the scanned point cloud and the second distance between each point and its adjacent points; match the normal vector of the calibration point cloud with the normal vector of each point in the scanned point cloud, and match the first distance with the second distance; when both the normal vector and the distance are successfully matched, a point pair with successfully matched features is obtained.

[0325] In this embodiment, the normal vector of the calibration point cloud is matched with the normal vector of each point in the scanned point cloud, and the first distance and the second distance are matched. When both are successfully matched, a point pair with successfully matched features is obtained, which can quickly filter out point pairs that meet the preliminary features.

[0326] In one embodiment, the first matching point pair acquisition module 802 is used to select three feature-matching point pairs from the feature-matching point pairs;

[0327] From three feature-matched point pairs, identify three marker points located in the same point cloud. When the three marker points located in the same point cloud are not collinear, determine whether the three feature-matched point pairs satisfy the congruent triangle condition.

[0328] When three feature-matched point pairs satisfy the condition of congruent triangles, the first transformation relationship from the current scanning viewpoint to the calibration object coordinate system is determined based on the three feature-matched point pairs.

[0329] In this embodiment, the point pairs with successfully matched features may also include many noisy points, i.e., non-marking points, so further screening is required; the features of three collinear points are relatively few, so non-collinear points need to be selected for the determination of congruent triangles. When the three feature-matched point pairs satisfy the condition of congruent triangles, it indicates that the accuracy of the three feature-matched point pairs is high, so the accuracy of the calculated first transformation relationship is also high.

[0330] For specific limitations regarding the camera calibration device, please refer to the limitations on the camera calibration method above, which will not be repeated here. Each module in the aforementioned camera calibration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0331] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a camera calibration method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0332] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0333] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0334] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0335] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0336] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes described in the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0337] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A camera calibration method, characterized in that, The method includes: For each of the two cameras, acquire the scan point cloud obtained by taking a picture of the calibration object at the current scanning viewpoint; the calibration object contains marker points; The calibration point cloud is matched with the scan point cloud to obtain a reference matching point pair; the reference matching point pair includes calibration matching points and matching scan matching points. The scanned matching points are projected onto the image coordinate system of the camera to obtain image mapping points; The calibration object image is obtained by the camera taking a picture of the calibration object; the calibration object image is an epipolar correction image. The image mapping points are matched with the marker points on the calibration object image to obtain image matching point pairs; the image matching point pairs include the scan matching points and the matching marker points on the calibration object image. Based on the reference matching point pair and the image matching point pair, a first matching point pair is determined that matches the marker point on the calibration object image with the marker point on the calibration object; Based on the epipolar correction inverse mapping relationship, the marker points on the epipolar corrected image are mapped to the original image to obtain the original image mapping points; the original image is the image captured by the camera. The mapping points of the original image are matched with the marker points on the original image to obtain a second matching point pair; the second matching point pair includes the marker points on the epipolar corrected image and the matching marker points on the original image. Based on the first matching point pair and the second matching point pair, a target matching point pair corresponding to each camera is determined; the target matching point pair includes the marker point on the original image and the marker point on the matching calibration object; Camera calibration is performed based on the target matching point pairs.

2. The method according to claim 1, characterized in that, The step of matching the mapping points of the original image with the marker points on the original image to obtain a second matching point pair includes: A distortion coefficient is added to the original image mapping point, and the distortion coefficient is adjusted until the original image mapping point matches the adjacent marker point on the original image, thus obtaining a second matching point pair.

3. The method according to claim 1, characterized in that, The camera calibration based on the target matching point pairs includes: Based on the target matching point pair, the camera is calibrated as a single camera to obtain the intrinsic parameters of the calibrated camera; The calibration camera intrinsic parameters and the reference camera intrinsic parameters are compared. Based on the comparison result, the target camera intrinsic parameters are determined from the calibration camera intrinsic parameters and the reference camera intrinsic parameters. Camera calibration is performed based on the target camera intrinsic parameters and the target matching point pair.

4. The method according to claim 3, characterized in that, The step of determining the target camera intrinsic parameters from the calibrated camera intrinsic parameters and the reference camera intrinsic parameters based on the comparison results, and performing camera calibration based on the target camera intrinsic parameters and the target matching point pair, includes: When the difference between the calibrated camera intrinsics and the reference camera intrinsics is within a preset range, the reference camera intrinsics are used as the target camera intrinsics, and camera calibration is performed based on the reference camera intrinsics and the target matching point pair.

5. The method according to claim 1, characterized in that, The camera calibration based on the target matching point pairs includes: Substitute the target matching point pair into the reference collinearity equation to obtain the target collinearity equation; Based on the target collinearity equation, partial derivatives are taken with respect to each element of the first camera extrinsic parameter matrix to obtain the first relational expression; The second relation is obtained by taking the partial derivative of each relative pose parameter with respect to the relative pose parameter matrix and determining the product with the preset second camera extrinsic parameter matrix. Based on the product of the first relation and the second relation, the partial derivative function relation of the target collinearity equation with respect to the relative pose parameters is determined; Based on the product of the increment of the relative pose parameter and the partial derivative function relationship, the pixel deviation relationship between the true coordinates of the image and the collinearity equation of the target is obtained; The pixel deviation relationship is iterated, and when the iteration is completed, the relative pose parameter values ​​of the first camera are obtained.

6. The method according to claim 1, characterized in that, The step of matching the calibrated point cloud with the scanned point cloud to obtain a reference matching point pair includes: Perform feature matching between the calibration point cloud and the scanned point cloud to obtain point pairs with successfully matched features; Based on the point pairs that are successfully matched by the features, a first transformation relationship is determined from the current scanning viewpoint to the calibration object coordinate system; Based on the first transformation relationship, the marker point under the current scanning view is mapped to the coordinate system of the calibration object to obtain the calibration object mapping point; Match the mapping points of the calibration object with the marker points on the calibration object to obtain a reference matching point pair.

7. A camera calibration device, characterized in that, The apparatus is used to implement the method according to any one of claims 1 to 6, the apparatus comprising: The first matching point pair acquisition module is used for: For each of the two cameras, acquire the scan point cloud obtained by taking a picture of the calibration object at the current scanning viewpoint; the calibration object contains marker points; The calibration point cloud is matched with the scan point cloud to obtain a reference matching point pair; the reference matching point pair includes calibration matching points and matching scan matching points. The scanned matching points are projected onto the image coordinate system of the camera to obtain image mapping points; The calibration object image is obtained by the camera taking a picture of the calibration object; the calibration object image is an epipolar correction image. The image mapping points are matched with the marker points on the calibration object image to obtain image matching point pairs; the image matching point pairs include the scan matching points and the matching marker points on the calibration object image; and Based on the reference matching point pair and the image matching point pair, a first matching point pair is determined that matches the marker point on the calibration object image with the marker point on the calibration object; The epipolar correction inverse mapping module is used to map the marker points on the epipolar corrected image to the original image based on the epipolar correction inverse mapping relationship, thereby obtaining the original image mapping points; the original image is the image captured by the camera. The second matching point pair determination module is used to match the original image mapping points with the marker points on the original image to obtain a second matching point pair; the second matching point pair includes the marker points on the epipolar correction image and the matching marker points on the original image. The target matching point pair acquisition module is used to determine the target matching point pair corresponding to each camera based on the first matching point pair and the second matching point pair; the target matching point pair includes the marker point on the original image and the marker point on the matching calibration object; The calibration module is used to perform camera calibration based on the target matching point pairs.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images