A camera model self-calibration method, device, medium and electronic equipment

Abstract

The application provides a camera model self-calibration method, device, medium and electronic equipment, and relates to the technical field of camera calibration. The method obtains a stereo image pair and a corresponding feature point matching pair based on an acquired image; obtains a three-dimensional point set according to the stereo image pair, the feature point matching pair, a rotation matrix and a translation vector corresponding to the stereo image pair; estimates virtual feature point coordinates based on the three-dimensional points, camera intrinsic parameters and preset hybrid distortion parameters; obtains a re-projection error value according to the virtual feature point coordinates and real feature point coordinates; calculates a total re-projection error value according to the re-projection error values corresponding to the images; calculates a corresponding regularization term error value based on the hybrid distortion parameters; determines a pixel loss function value based on the total re-projection error value and the regularization term error value, and updates the camera intrinsic parameters and the hybrid distortion parameters when the pixel loss function value does not converge until the pixel loss function converges. The method can effectively improve the efficiency of the camera calibration process.

Description

Technical Field

[0001] This application relates to the field of camera calibration technology, and in particular to a camera model self-calibration method, apparatus, medium and electronic device. Background Technology

[0002] A camera model describes the imaging process from a 3D point in the real world to a 2D point in an image. Broadly speaking, camera calibration includes two parts: determining the camera model and determining the camera parameters, which include intrinsic camera parameters and distortion parameters. Narrowly defined, camera calibration determines the camera parameters given a known camera model. Camera calibration is fundamental to vision-based 3D reconstruction techniques. Using camera parameters with large errors in 3D reconstruction will severely affect the accuracy of the generated 3D model; using a camera model that does not match the image can directly lead to reconstruction failure.

[0003] There are two main types of existing 3D reconstruction camera calibration methods: one is offline calibration, which uses traditional calibration methods and specific geometric information (such as a checkerboard pattern) to calibrate the camera parameters before 3D reconstruction begins; the other is online calibration, which involves adding the camera parameters as unknowns to the bundle adjustment (BA) solution process during 3D modeling. Most existing 3D modeling technologies include this type of online calibration function.

[0004] With the increasing prevalence of imaging devices such as mobile phones, digital cameras, and civilian drones, and the availability of a vast amount of image resources on the internet, it has become more difficult to manually determine camera models.

[0005] Existing camera model self-calibration methods cannot determine the inherent form of the camera model, requiring manual specification. However, determining the camera model is often difficult or incorrect, leading to low efficiency in the camera calibration process. Therefore, providing a camera model self-calibration method to solve the problem of inefficiency in camera calibration is of significant practical importance. Summary of the Invention

[0006] This application provides a camera model self-calibration method, apparatus, medium, and electronic device, which can improve the efficiency of the camera calibration process.

[0007] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a camera model self-calibration method, including:

[0009] Based on the images of the target object captured by the camera multiple times, stereo image pairs of the target object and corresponding feature point matching pairs are obtained;

[0010] Based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair, the three-dimensional point set of the target object is obtained;

[0011] Based on the 3D points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters, the coordinates of virtual feature points projected onto the image from the 3D points are estimated; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the calibration of the camera model.

[0012] The reprojection error value is obtained based on the virtual feature point coordinates and the real feature point coordinates;

[0013] Based on the reprojection error value corresponding to the image, calculate the total reprojection error value of the three-dimensional points with respect to the image;

[0014] Based on the hybrid distortion parameters, the corresponding regularization term error value is calculated;

[0015] The pixel loss function value is determined based on the total reprojection error value and the regularization term error value. When the pixel loss function value does not converge, the camera intrinsic parameters and the hybrid distortion parameters are updated to perform nonlinear optimization processing on the camera intrinsic parameters and the hybrid distortion parameters until the pixel loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters.

[0016] The camera model self-calibration method provided in this application embodiment obtains stereo image pairs and corresponding feature point matching pairs of the target object based on images captured multiple times by the camera on the target object; obtains a three-dimensional point set of the target object based on the stereo image pairs, the feature point matching pairs, and the rotation matrix and translation vector corresponding to the stereo image pairs; estimates the coordinates of virtual feature points projected onto the image based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the camera model calibration process; and estimates the coordinates of virtual feature points projected onto the image based on the real feature point coordinates and the true feature point set. The reprojection error value is obtained by taking the coordinates of real feature points. Based on the reprojection error value corresponding to the image, the total reprojection error value of the 3D points relative to the image is calculated. The corresponding regularization term error value is calculated based on the hybrid distortion parameters. The image point loss function value is determined based on the total reprojection error value and the regularization term error value. When the image point loss function value does not converge, the camera intrinsic parameters and the hybrid distortion parameters are updated to perform nonlinear optimization processing on the camera intrinsic parameters and the hybrid distortion parameters until the image point loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters. This method incorporates both the camera intrinsic parameters and distortion parameters into the optimization process, automatically calibrating camera parameters for images generated by different cameras and automatically calibrating the camera model, effectively improving the efficiency of the camera calibration process.

[0017] In one optional embodiment, obtaining stereo image pairs and corresponding feature point matching pairs of the target object based on images captured multiple times by the camera on the target object includes:

[0018] Acquire multiple images of the target object taken by the camera;

[0019] Affine matching point pair extraction and matching are performed on the image to obtain stereo image pairs of the target object and corresponding feature point matching pairs.

[0020] The method of this embodiment acquires images of the target object taken by the camera multiple times; performs affine matching point pair extraction and matching on the images to obtain stereo image pairs of the target object and corresponding feature point matching pairs. Affine matching point pair extraction and matching can be performed during camera model self-calibration to obtain stereo image pairs of the target object and corresponding feature point matching pairs accurately and efficiently, which can further improve the efficiency of the camera calibration process.

[0021] In an optional embodiment, obtaining the three-dimensional point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair includes:

[0022] Based on the feature point matching pairs, the rotation matrix and translation vector between the stereo image pairs of the target object are obtained;

[0023] The three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, the rotation matrix, and the translation vector.

[0024] In this embodiment, the rotation matrix and translation vector between the stereo image pairs of the target object are obtained based on the feature point matching pairs; the three-dimensional point set of the target object is obtained according to the stereo image pairs, the feature point matching pairs, the rotation matrix and the translation vector. By first determining the rotation matrix and translation vector, the three-dimensional point set of the target object can be determined more accurately, thereby improving the efficiency of the camera calibration process.

[0025] In one alternative embodiment, the hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters.

[0026] The method in this embodiment, wherein the hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters, can integrate multiple forms of camera model, making it more widely applicable. These camera parameters can be automatically calibrated for images generated by different cameras without maintaining a large database or requiring users to specify camera models, thereby more effectively improving the efficiency of the camera calibration process.

[0027] In one alternative embodiment, the regularization term is an L1 regularization term or an L2 regularization term.

[0028] In this embodiment, the regularization term is either an L1 regularization term or an L2 regularization term. By introducing an L1 or L2 regularization term for the distortion parameters, redundant parameters in the mixed distortion parameters can gradually approach 0 as the BA nonlinear optimization proceeds, thereby completing the update of the parameters in the mixed distortion parameters that match the camera and improving the efficiency of the camera calibration process.

[0029] In one alternative embodiment, the pixel loss function is obtained by summing the reprojection error with the regularization term error.

[0030] In this embodiment, the image point loss function is obtained by summing the reprojection error and the regularization term error. By adding a regularization term for the distortion parameter to the reprojection error as the loss function, the camera intrinsic parameters and mixed distortion parameters are efficiently coordinated and optimized, significantly improving the efficiency of the camera calibration process.

[0031] In one optional embodiment, calculating the corresponding regularization term error value based on the mixed distortion parameters includes:

[0032] Select the hybrid distortion parameters one by one. For each selected hybrid distortion parameter, calculate the parameter regularization term value corresponding to the selected hybrid distortion parameter.

[0033] The parameter regularization term values ​​are selected one by one. For each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficient corresponds one-to-one with the hybrid distortion parameter.

[0034] The regularization term factors of the parameters are summed to obtain the regularization term error value corresponding to the mixed distortion parameters.

[0035] The method of this embodiment selects the mixed distortion parameters one by one. For each selected mixed distortion parameter, a parameter regularization term value corresponding to the selected mixed distortion parameter is calculated. Then, for each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by a preset regularization term weight coefficient to obtain a parameter regularization term factor. The regularization term weight coefficients correspond one-to-one with the mixed distortion parameters. The parameter regularization term factors are summed to obtain the regularization term error value corresponding to the mixed distortion parameter. This method can accurately determine the corresponding regularization term error value based on the mixed distortion parameters, further improving the efficiency of the camera calibration process.

[0036] Secondly, embodiments of this application also provide a camera model self-calibration device, comprising:

[0037] The object information extraction module is used to obtain stereo image pairs of the target object and corresponding feature point matching pairs based on the images of the target object captured by the camera multiple times.

[0038] The three-dimensional point set calculation module is used to obtain the three-dimensional point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair.

[0039] The virtual coordinate determination module is used to estimate the coordinates of virtual feature points projected onto the image based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the calibration of the camera model;

[0040] The projection error calculation module is used to obtain the reprojection error value based on the coordinates of the virtual feature points and the coordinates of the real feature points;

[0041] The projection error integration module is used to calculate the total reprojection error value of the three-dimensional points relative to the image based on the reprojection error value corresponding to the image.

[0042] The regularization error calculation module is used to calculate the corresponding regularization term error value based on the hybrid distortion parameters.

[0043] The distortion parameter coordination module is used to determine the image point loss function value based on the total reprojection error value and the regularization term error value. When the image point loss function value does not converge, the camera intrinsic parameters and the hybrid distortion parameters are updated to perform nonlinear optimization processing on the camera intrinsic parameters and the hybrid distortion parameters until the image point loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters.

[0044] In one optional embodiment, the object information extraction module is specifically used for:

[0045] Acquire multiple images of the target object taken by the camera;

[0046] Affine matching point pair extraction and matching are performed on the image to obtain stereo image pairs of the target object and corresponding feature point matching pairs.

[0047] In one optional embodiment, the three-dimensional point set calculation module is specifically used for:

[0048] Based on the feature point matching pairs, the rotation matrix and translation vector between the stereo image pairs of the target object are obtained;

[0049] The three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, the rotation matrix, and the translation vector.

[0050] In one alternative embodiment, the hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters.

[0051] In one alternative embodiment, the regularization term is an L1 regularization term or an L2 regularization term.

[0052] In one alternative embodiment, the pixel loss function is obtained by summing the reprojection error with the regularization term error.

[0053] In one optional embodiment, the regularization error calculation module is specifically used for:

[0054] Select the hybrid distortion parameters one by one. For each selected hybrid distortion parameter, calculate the parameter regularization term value corresponding to the selected hybrid distortion parameter.

[0055] The parameter regularization term values ​​are selected one by one. For each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficient corresponds one-to-one with the hybrid distortion parameter.

[0056] The regularization term factors of the parameters are summed to obtain the regularization term error value corresponding to the mixed distortion parameters.

[0057] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the camera model self-calibration method of the first aspect.

[0058] Fourthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, the processor enables the processor to implement the camera model self-calibration method of the first aspect.

[0059] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A schematic flowchart illustrating a camera model self-calibration method provided in an embodiment of this application;

[0062] Figure 2 A schematic diagram illustrating the process of obtaining stereo image pairs and corresponding feature point matching pairs based on statistical data in a camera model self-calibration method provided in this application embodiment;

[0063] Figure 3 A schematic diagram illustrating the process of obtaining a three-dimensional point set using a camera model self-calibration method provided in this application embodiment;

[0064] Figure 4 A schematic diagram illustrating the process of calculating the regularization term error value based on hybrid distortion parameters in a camera model self-calibration method provided in this application embodiment;

[0065] Figure 5This is a schematic diagram of the structure of a camera model self-calibration device provided in an embodiment of this application;

[0066] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0069] The following are explanations of some of the words that appear in the text:

[0070] (1) BA (Bundle Adjustment): BA, also translated as bundle adjustment, refers to the process of extracting the optimal 3D model and camera parameters (intrinsic and extrinsic parameters) from visual reconstruction. The bundles of light rays reflected from each feature point are finally converged to the optical center after we make optimal adjustments to the camera pose and the position of the feature points.

[0071] (2) Reprojection error: Reprojection error is the distance between the image key points detected in the calibration image and the corresponding world points projected into the same image.

[0072] (3) Stereo pairs: Stereo pairs, or simply pairs, refer to a pair of images taken from two different locations of the same object or scene.

[0073] A camera model describes the imaging process from a 3D point in the real world to a 2D point in an image. Broadly speaking, camera calibration includes two parts: determining the camera model and determining the camera parameters, which include intrinsic camera parameters and distortion parameters. Narrowly defined, camera calibration determines the camera parameters given a known camera model. Camera calibration is fundamental to vision-based 3D reconstruction techniques. Using camera parameters with large errors in 3D reconstruction will severely affect the accuracy of the generated 3D model; using a camera model that does not match the image can directly lead to reconstruction failure.

[0074] There are two main types of existing 3D reconstruction camera calibration methods: one is offline calibration, which uses traditional calibration methods and specific geometric information (such as a checkerboard pattern) to calibrate the camera parameters before 3D reconstruction begins; the other is online calibration, which involves adding the camera parameters as unknowns to the bundle adjustment (BA) solution process during 3D modeling. Most existing 3D modeling technologies include this type of online calibration function.

[0075] With the increasing prevalence of imaging devices such as mobile phones, digital cameras, and civilian drones, and the availability of a vast amount of image resources on the internet, it has become more difficult to manually determine camera models.

[0076] Existing camera model self-calibration methods cannot determine the inherent form of the camera model, requiring manual specification. However, determining the camera model is often difficult or incorrect, leading to low efficiency in the camera calibration process. Therefore, providing a camera model self-calibration method to solve the problem of inefficiency in camera calibration is of significant practical importance.

[0077] To address existing technical problems, embodiments of this application provide a camera model self-calibration method, apparatus, medium, and electronic device. This method obtains stereo image pairs and corresponding feature point matching pairs of the target object based on multiple images captured by the camera. A three-dimensional point set of the target object is obtained based on the stereo image pairs, feature point matching pairs, and the rotation matrix and translation vector corresponding to the stereo image pairs. The coordinates of virtual feature points projected onto the image are estimated based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters. The hybrid distortion parameters characterize the image model calibration process based on multiple types of distortion models. The method involves revising the image; obtaining reprojection error values ​​based on virtual and real feature point coordinates; calculating the total reprojection error value of 3D points relative to the image based on the corresponding reprojection error value; calculating the corresponding regularization term error value based on the hybrid distortion parameters; determining the image point loss function value based on the total reprojection error value and the regularization term error value; updating the camera intrinsic parameters and hybrid distortion parameters when the image point loss function value does not converge, and performing nonlinear optimization on the camera intrinsic parameters and hybrid distortion parameters until the camera's image point loss function converges, thus completing the camera self-calibration; the regularization term error is the regularization term of the hybrid distortion parameters. This method incorporates both the camera intrinsic parameters and distortion parameters into the optimization process, automatically calibrating camera parameters for images generated by different cameras, and automatically calibrating the camera model, effectively improving the efficiency of the camera calibration process.

[0078] To make the inventive objectives, technical solutions, and advantages of the embodiments of this application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] The technical solutions provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0080] The camera model self-calibration method provided in this application can be applied to a terminal device. In the embodiments of this application, the terminal device can be an imaging device such as a camera. The following embodiments of this application illustrate the application of the camera model self-calibration method to a camera.

[0081] This application provides a camera model self-calibration method, such as... Figure 1 As shown, it includes the following steps:

[0082] Step S101: Based on the images of the target object captured by the camera multiple times, obtain stereo image pairs of the target object and corresponding feature point matching pairs.

[0083] In practice, based on the images of the target object captured by the camera multiple times, stereo image pairs of the target object and corresponding feature point matching pairs are obtained.

[0084] In some optional embodiments, the process in step S101 above, which involves obtaining stereo image pairs of the target object and corresponding feature point matching pairs based on the images of the target object captured multiple times by the camera, is as follows: Figure 2 As shown, this can be achieved through the following steps:

[0085] Step S201: Acquire images of the target object taken by the camera multiple times.

[0086] Understandably, the camera can be any type of camera used in image measurement, computer vision applications, and other scenarios involving camera applications in this field. Images can be obtained by the camera capturing images of the current object or the scene to be applied in real time or in advance. The camera can capture images of the target object from different positions and in different orientations.

[0087] Step S202: Perform affine matching point pair extraction and matching on the image to obtain stereo image pairs of the target object and corresponding feature point matching pairs.

[0088] The process of extracting and matching affine matching point pairs from multiple images of the target object captured by the camera can refer to the affine matching point pair extraction and matching methods used in camera calibration methods based on affine matching point pairs in this field. For example, the existing ASIFT algorithm can be used to extract affine matching point pairs, which will not be elaborated further in this application. The set of all affine matching point pairs found by the extraction and matching operations can constitute the affine matching point pair set corresponding to the target object. A stereo image pair refers to an image pair composed of two images, while an affine matching point pair is the affine matching point pair between the two images of a stereo image pair.

[0089] Step S102: Based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair, obtain the three-dimensional point set of the target object.

[0090] In practice, the three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair. Alternatively, the rotation matrix and translation vector corresponding to the stereo image pair can be determined based on the stereo image pair and the feature point matching pair; then, the three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair.

[0091] In some optional embodiments, the process of obtaining the three-dimensional point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair in step S102 above, such as... Figure 3 As shown, this can be achieved through the following steps:

[0092] Step S301: Based on the feature point matching pairs, obtain the rotation matrix and translation vector between the stereo image pairs of the target object.

[0093] In this embodiment, the camera parameters include intrinsic parameters, extrinsic parameters, and distortion parameters. Intrinsic parameters are parameters related to the camera's inherent characteristics, including but not limited to focal length and optical axis offset; the optical axis offset characterizes the offset of the camera's optical axis in the image coordinate system. Optical axis offset is typically measured in pixels. Extrinsic parameters refer to the camera's parameters in the world coordinate system, including but not limited to rotation matrices and translation vectors. Distortion parameters are distortions introduced by deviations in manufacturing precision and assembly processes, leading to distortion of the original image. During camera self-calibration, based on feature point matching pairs, the rotation matrix and translation vector between stereo image pairs of the target object can be obtained; these rotation matrix and translation vector can be used to determine the camera position and camera rotation direction.

[0094] Step S302: Based on the stereo image pairs, feature point matching pairs, rotation matrix, and translation vector, obtain the three-dimensional point set of the target object.

[0095] For example, taking a pinhole camera model used in 3D reconstruction as an example, the model equations of the following form are illustrated:

[0096]

[0097] in,

[0098] X c ,Y c Z c These are the coordinates of a 3D point in the camera coordinate system in reality;

[0099] u and v are the pixel coordinates of a three-dimensional point in reality as it is projected onto an image;

[0100] s represents the depth information lost during the projection from 3D to 2D.

[0101] f x It is the focal length of the camera in the x-direction;

[0102] f y It is the focal length of the camera in the y-direction;

[0103] c x It is the offset of the camera's optical axis in the x-direction;

[0104] c y It is the offset of the camera's optical axis in the x-direction.

[0105] f contained in the model equation x ,f y ,c x ,c y The four parameters can be called camera intrinsic parameters. The relationship between the camera coordinate system and the world coordinate system can be described by a rotation matrix R and a translation vector t. The rotation matrix R and the translation vector t are called camera extrinsic parameters.

[0106] Based on the camera's extrinsic parameters, the following formula can be used to calculate a 3D point (X) in the world coordinate system. w ,Y w Z w The coordinates (X) in the camera coordinate system c ,Y c Z c ):

[0107]

[0108] 3D reconstruction is the process of using a set of images from the same scene, estimating the camera intrinsic and extrinsic parameters corresponding to the images, and finally calculating the 3D coordinates of each key point in the scene.

[0109] Step S103: Based on the 3D points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters, estimate the coordinates of virtual feature points projected onto the image from the 3D points.

[0110] Among them, the hybrid distortion parameter characterization is used to revise the camera imaging based on multiple types of distortion models during the calibration of the camera model.

[0111] In some optional embodiments, the hybrid distortion parameters are distortion parameters included in a preset hybrid distortion equation; the hybrid distortion equation is a distortion equation constructed based on a variety of preset distortion parameters.

[0112] In the above embodiments, the hybrid distortion parameters are the distortion parameters included in the preset hybrid distortion equation, and the hybrid distortion equation is set as a distortion equation constructed based on multiple preset types of distortion parameters. This enables the camera imaging to be revised based on multiple types of distortion models during the calibration of the camera model, providing a mechanism for BA optimization process of 3D reconstruction without explicitly specifying a distortion equation.

[0113] In some alternative embodiments, the hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters.

[0114] Existing technical solutions require explicitly specifying a distortion equation before performing the BA (Balanced Image) optimization process for 3D reconstruction. To avoid specifying the form of the distortion equation in advance, an optional embodiment of this application proposes using an equation containing a certain amount of redundant parameters, i.e., a hybrid distortion equation, to describe camera distortion. The hybrid distortion equation is constructed based on multiple preset types of distortion parameters. During the BA optimization process of 3D reconstruction, the redundant parameters are constrained to iteratively tend towards 0, thereby degenerating the distortion equation into a simple form that matches the original image data.

[0115] In one optional embodiment of this application, the proposed preset hybrid distortion equation has the form shown below. This hybrid distortion equation considers radial distortion, tangential distortion, and thin lens distortion. The hybrid distortion equation contains a total of 12 parameters, including k1, k2, p1, p2, k3, k4, k5, k6, k7, k8, sx1, and sy1.

[0116]

[0117]

[0118]

[0119] Where u and v are the pixel coordinates in the x and y directions, respectively;

[0120] r is the distance from the pixel to the center of the image;

[0121] k1, k2, k3, k4, k5, k6, k7, and k8 are radial distortion parameters;

[0122] p1 and p2 are tangential distortion parameters;

[0123] sx1 and sy1 are the distortion parameters of the thin lens.

[0124] Thin lens distortion parameters are used to balance the imaging effects caused by the non-parallelism between the lens and the imaging element during the assembly process.

[0125] In some embodiments of this application, the reprojection error is calculated using the following process.

[0126] The key point (u0, v0) on the image is a 3D point (X) in the world coordinate system. w ,Y w Z w (Image is generated). In an optional embodiment, based on the current estimates of the camera's intrinsic and extrinsic parameters, the 3D points can be reprojected onto the image, as shown in the following equation;

[0127]

[0128] Where (u,v) are the coordinates of the projection point on the image.

[0129] By correcting the distortion of the projected points using the distortion model, we can obtain the coordinates of the virtual feature points projected onto the image from the 3D points, as shown in the following formula:

[0130]

[0131] Step S104: Obtain the reprojection error value based on the virtual feature point coordinates and the real feature point coordinates.

[0132] Specifically, reprojection error is the distance between image keypoints detected in an image and their corresponding world points projected into the same image during camera calibration. In some embodiments of this application, the image keypoints can be real feature point coordinates, while the corresponding world points projected onto the same image can be virtual feature point coordinates. Therefore, the reprojection error value can be obtained based on the virtual feature point coordinates and the real feature point coordinates.

[0133] In an optional embodiment of this application, the reprojection error of this key point can be expressed in the following form:

[0134]

[0135] Where (u0,v0) are the coordinates of the real feature points, representing the positions of key points on the image;

[0136] (u′,v′) are the coordinates of the virtual feature point, representing the position of the virtual feature point projected onto the image by the 3D point.

[0137] Step S105: Calculate the total reprojection error value of the 3D points to the image based on the reprojection error value corresponding to the image.

[0138] In some optional embodiments, the process of calculating the total reprojection error value of the three-dimensional points relative to the image based on the reprojection error value corresponding to the image can be to select images one by one, and for each selected image, sum the reprojection error values ​​of each three-dimensional point in the currently selected image to obtain the total reprojection error value of the three-dimensional points relative to the currently selected image.

[0139] Step S106: Calculate the corresponding regularization term error value based on the mixed distortion parameters.

[0140] In some alternative embodiments, the regularization term is an L1 regularization term or an L2 regularization term.

[0141] In one optional embodiment of this application, the 12 parameters in the distortion equation are added to the BA nonlinear optimization. The loss function of the BA optimization is constructed by adding a regularization term for the distortion parameters as a loss function on top of the reprojection error of the feature points. The regularization term can be L1 regularization or L2 regularization.

[0142] In some optional embodiments, the process of calculating the corresponding regularization term error value based on the mixed distortion parameters in step S106 above, such as... Figure 4 As shown, this can be achieved through the following steps:

[0143] Step S401: Select the mixed distortion parameters one by one. For each selected mixed distortion parameter, calculate the parameter regularization term value corresponding to the selected mixed distortion parameter.

[0144] For example, let's take L1 regularization as an example. We select the mixed distortion parameter k one by one. i Each time a mixed distortion parameter k is selected i Calculate and select the mixed distortion parameter k i The corresponding parameter regularization term value |k i |

[0145] Step S402: Select parameter regularization terms one by one. For each parameter regularization term selected, multiply the currently selected parameter regularization term value with the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficient and the mixed distortion parameter are in one-to-one correspondence.

[0146] For example, select the parameter regularization term value |k one by one. i |, for each selected parameter, the regularization term value|k i |, then the currently selected parameter regularization term value |k i | and the preset regularization term weight coefficient λ i Taking the product, we obtain the parameter regularization factor λ. i |k i |;Regularization term weight coefficient λ i With the mixed distortion parameter k i It's a one-to-one correspondence.

[0147] Step S403: Sum the regularization term factors of the parameters to obtain the regularization term error value corresponding to the mixed distortion parameters.

[0148] For example, the parameter regularization term factor λ i |k i Summing yields the mixture distortion parameter k. i The corresponding regularization term error value. The regularization term error value obtained in this process can be represented by the following formula:

[0149] loss L1 =∑λ i |k i |

[0150] Understandably, similar to the process described above, when the regularization term is L2 regularization, the resulting error value of the regularization term can be expressed as follows:

[0151]

[0152] Step S107: Determine the image point loss function value based on the total reprojection error value and the regularization term error value. If the image point loss function value does not converge, update the camera intrinsic parameters and hybrid distortion parameters to perform nonlinear optimization processing on the camera intrinsic parameters and hybrid distortion parameters until the image point loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters.

[0153] In practice, the image point loss function includes a regularization term error.

[0154] In some optional embodiments of this application, the image point loss function is constructed based on the reprojection error and the regularization term error.

[0155] In some optional embodiments, the image point loss function is obtained by summing the reprojection error and the regularization term error. The loss function used combines the reprojection error and the regularization term error, and the total loss function for all keypoints in the scene is shown in the following equation:

[0156] loss = ∑(loss p +loss L ),

[0157] Where, loss p This is due to reprojection error;

[0158] loss L This represents the error in the regularization term.

[0159] Under the influence of the regularization term error, as the BA nonlinear optimization proceeds, the redundant mixed distortion parameters in the distortion equation will gradually approach 0, thereby causing the distortion equation to degenerate into a simpler form that matches the actual camera.

[0160] In some embodiments, when parameters k4, k5, k6, k8 approach 0, the preset hybrid distortion equation degenerates into the form of the equation shown below:

[0161]

[0162] h2=k1*r2 +k2*r 4 +k3*r 6 +k4*r 8 ;

[0163] The above equation has a high order of radial distortion, which can describe highly nonlinear radial distortion, and also increases the distortion of thin lenses.

[0164] When the parameters k7, k8, sx1, sy1 approach 0, the preset mixed distortion equation degenerates into the form shown in the following equation:

[0165]

[0166]

[0167] The above equation has a high order of radial distortion and can describe highly nonlinear radial distortion.

[0168] Similarly, in one case, the pre-defined mixed distortion equation can degenerate into the form shown below:

[0169]

[0170] The above equation balances radial and tangential distortion.

[0171] In another case, the pre-defined hybrid distortion equation can degenerate into the form shown below:

[0172]

[0173] The above equation balances radial distortion using two radial distortion parameters.

[0174] In another case, the pre-defined hybrid distortion equation can degenerate into the form shown below:

[0175]

[0176] The above equation balances the low-order radial distortion.

[0177] The degradation result of the aforementioned preset hybrid distortion equation enables camera models that cover commonly used 3D reconstruction imaging devices.

[0178] Understandably, based on different preset hybrid distortion equations, in some other embodiments of this application, the preset hybrid distortion equations may also degenerate into other forms besides those described above. This application does not specifically limit the form of the hybrid distortion equations.

[0179] The camera model self-calibration method provided in this application embodiment obtains stereo image pairs and corresponding feature point matching pairs of the target object based on images captured multiple times by the camera on the target object; it obtains a three-dimensional point set of the target object based on the stereo image pairs, feature point matching pairs, and the rotation matrix and translation vector corresponding to the stereo image pairs; it estimates the coordinates of virtual feature points projected onto the image based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters; the hybrid distortion parameters characterize the revision of camera imaging based on multiple types of distortion models during the camera model calibration process; and it estimates the coordinates of virtual feature points projected onto the image based on the virtual feature points. The reprojection error value is obtained by combining the coordinates of the points and the coordinates of the ground truth feature points. Based on the reprojection error value corresponding to the image, the total reprojection error value of the 3D points to the image is calculated. The corresponding regularization term error value is calculated based on the hybrid distortion parameters. The image point loss function value is determined based on the total reprojection error value and the regularization term error value. If the image point loss function value does not converge, the camera's intrinsic parameters and hybrid distortion parameters are updated to perform nonlinear optimization processing on the camera's intrinsic parameters and hybrid distortion parameters until the camera's image point loss function converges, thus completing the camera's self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters. This method incorporates both the camera's intrinsic parameters and distortion parameters into the optimization process, automatically calibrating camera parameters for images generated by different cameras and automatically calibrating the camera model, effectively improving the efficiency of the camera calibration process.

[0180] Based on the same inventive concept, this application also provides a camera model self-calibration device. Since this device corresponds to the camera model self-calibration method provided in this application, and the principle of solving the problem by this device is similar to that of the method, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0181] Figure 5 This illustration shows a schematic diagram of a camera model self-calibration device according to an embodiment of this application. Figure 5 As shown, the camera model self-calibration device includes an object information extraction module 501, a three-dimensional point set calculation module 502, a virtual coordinate determination module 503, a projection error calculation module 504, a projection error integration module 505, a regularization error calculation module 506, and a distortion parameter coordination module 507.

[0182] Among them, the object information extraction module 501 is used to obtain stereo image pairs of the target object and corresponding feature point matching pairs based on the images of the target object captured by the camera multiple times;

[0183] The 3D point set calculation module 502 is used to obtain the 3D point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair;

[0184] The virtual coordinate determination module 503 is used to estimate the coordinates of virtual feature points projected onto the image based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the calibration of the camera model;

[0185] The projection error calculation module 504 is used to obtain the reprojection error value based on the coordinates of the virtual feature points and the coordinates of the real feature points;

[0186] The projection error integration module 505 is used to calculate the total reprojection error value of the three-dimensional points to the image based on the reprojection error value corresponding to the image.

[0187] The regularization error calculation module 506 is used to calculate the corresponding regularization term error value based on the mixed distortion parameters;

[0188] The distortion parameter coordination module 507 is used to determine the image point loss function value based on the total reprojection error value and the regularization term error value. When the image point loss function value does not converge, the camera intrinsic parameters and hybrid distortion parameters are updated to perform nonlinear optimization processing on the camera intrinsic parameters and hybrid distortion parameters until the image point loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters.

[0189] In some optional embodiments, the hybrid distortion parameters are distortion parameters included in a preset hybrid distortion equation; the hybrid distortion equation is a distortion equation constructed based on a variety of preset distortion parameters.

[0190] In an optional embodiment, the object information extraction module 501 is specifically used for:

[0191] Acquire multiple images of the target object taken by the camera;

[0192] Affine matching point pair extraction and matching are performed on the image to obtain stereo image pairs of the target object and corresponding feature point matching pairs.

[0193] In an optional embodiment, the three-dimensional point set calculation module 502 is specifically used for:

[0194] Based on the feature point matching pairs, the rotation matrix and translation vector between the stereo image pairs of the target object are obtained;

[0195] The three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, the rotation matrix, and the translation vector.

[0196] In one alternative embodiment, the hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters.

[0197] In one alternative embodiment, the regularization term is an L1 regularization term or an L2 regularization term.

[0198] In one alternative embodiment, the pixel loss function is obtained by summing the reprojection error with the regularization term error.

[0199] In an optional embodiment, the regularization error calculation module 506 is specifically used for:

[0200] Select the hybrid distortion parameters one by one. For each selected hybrid distortion parameter, calculate the parameter regularization term value corresponding to the selected hybrid distortion parameter.

[0201] The parameter regularization term values ​​are selected one by one. For each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficient corresponds one-to-one with the hybrid distortion parameter.

[0202] The regularization term factors of the parameters are summed to obtain the regularization term error value corresponding to the mixed distortion parameters.

[0203] The camera model self-calibration device provided in this application embodiment obtains stereo image pairs and corresponding feature point matching pairs based on the acquired image; obtains a three-dimensional point set based on the stereo image pairs, feature point matching pairs, and the rotation matrix and translation vector corresponding to the stereo image pairs; estimates the coordinates of virtual feature points based on the three-dimensional points, camera intrinsic parameters, and preset hybrid distortion parameters; obtains reprojection error values ​​based on the coordinates of virtual and real feature points; calculates the total reprojection error value based on the reprojection error value corresponding to the image; calculates the corresponding regularization term error value based on the hybrid distortion parameters; determines the image point loss function value based on the total reprojection error value and the regularization term error value; and updates the camera intrinsic parameters and hybrid distortion parameters when the image point loss function value does not converge until the image point loss function converges. This device can effectively improve the efficiency of the camera calibration process.

[0204] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. This electronic device can be used for camera model self-calibration. In this application, the electronic device can be a server or a terminal device. In one embodiment, the electronic device can be a terminal device. In this embodiment, the structure of the electronic device can be as follows... Figure 6 As shown, it includes a memory 601, a communication module 603, and one or more processors 602.

[0205] The memory 601 is used to store computer programs executed by the processor 602. The memory 601 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0206] Memory 601 may be volatile memory, such as random-access memory (RAM); memory 601 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 601 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 601 may be a combination of the above-described memories.

[0207] Processor 602 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 602 is used to implement the above-described camera model self-calibration method when calling the computer program stored in memory 601.

[0208] The communication module 603 is used to communicate with terminal devices and other servers.

[0209] This application embodiment does not limit the specific connection medium between the memory 601, communication module 603, and processor 602 described above. This application embodiment... Figure 6 The memory 601 and the processor 602 are connected via a bus 604, and the bus 604 is in Figure 6 The connections between other components are shown in thick lines only and are not intended to be limiting. The 604 bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0210] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the camera model self-calibration method described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A camera model self-calibration method, characterized in that, include: Based on the images of the target object captured by the camera multiple times, stereo image pairs of the target object and corresponding feature point matching pairs are obtained; Based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair, the three-dimensional point set of the target object is obtained; Based on the 3D points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters, the coordinates of virtual feature points projected onto the image from the 3D points are estimated; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the calibration of the camera model. The reprojection error value is obtained based on the virtual feature point coordinates and the real feature point coordinates; Based on the reprojection error value corresponding to the image, calculate the total reprojection error value of the three-dimensional points with respect to the image; Based on the hybrid distortion parameters, the corresponding regularization term error value is calculated; The image point loss function value is determined based on the total reprojection error value and the regularization term error value. When the image point loss function value does not converge, the camera intrinsic parameters and the hybrid distortion parameters of the camera are updated to perform nonlinear optimization processing on the camera intrinsic parameters and the hybrid distortion parameters until the image point loss function of the camera converges, thus completing the self-calibration of the camera. The regularization term error is the regularization term of the mixed distortion parameter; The step of calculating the corresponding regularization term error value based on the mixed distortion parameters includes: Select the hybrid distortion parameters one by one. For each selected hybrid distortion parameter, calculate the parameter regularization term value corresponding to the selected hybrid distortion parameter. The parameter regularization term values ​​are selected one by one. For each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficient corresponds one-to-one with the hybrid distortion parameter. The regularization term factors of the parameters are summed to obtain the regularization term error value corresponding to the mixed distortion parameters.

2. The method according to claim 1, characterized in that, The process of obtaining stereo image pairs and corresponding feature point matching pairs of the target object based on multiple images captured by the camera includes: Acquire multiple images of the target object taken by the camera; Affine matching point pair extraction and matching are performed on the image to obtain stereo image pairs of the target object and corresponding feature point matching pairs.

3. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair includes: Based on the feature point matching pairs, the rotation matrix and translation vector between the stereo image pairs of the target object are obtained; The three-dimensional point set of the target object is obtained based on the stereo image pair, the feature point matching pair, the rotation matrix, and the translation vector.

4. The method according to claim 1, characterized in that, The hybrid distortion parameters include radial distortion parameters, tangential distortion parameters, and thin lens distortion parameters.

5. The method according to claim 1, characterized in that, The regularization term is either an L1 regularization term or an L2 regularization term.

6. The method according to claim 1, characterized in that, The image point loss function is obtained by summing the reprojection error and the regularization term error.

7. A camera model self-calibration device, characterized in that, include: The object information extraction module is used to obtain stereo image pairs of the target object and corresponding feature point matching pairs based on the images of the target object captured by the camera multiple times. The three-dimensional point set calculation module is used to obtain the three-dimensional point set of the target object based on the stereo image pair, the feature point matching pair, and the rotation matrix and translation vector corresponding to the stereo image pair. The virtual coordinate determination module is used to estimate the coordinates of virtual feature points projected onto the image based on the three-dimensional points of the target object, camera intrinsic parameters, and preset hybrid distortion parameters; the hybrid distortion parameters represent the revision of camera imaging based on multiple types of distortion models during the calibration of the camera model; The projection error calculation module is used to obtain the reprojection error value based on the coordinates of the virtual feature points and the coordinates of the real feature points; The projection error integration module is used to calculate the total reprojection error value of the three-dimensional points relative to the image based on the reprojection error value corresponding to the image. The regularization error calculation module is used to calculate the corresponding regularization term error value based on the hybrid distortion parameters. The distortion parameter coordination module is used to determine the image point loss function value based on the total reprojection error value and the regularization term error value. When the image point loss function value does not converge, the module updates the camera intrinsic parameters and the hybrid distortion parameters to perform nonlinear optimization processing on the camera intrinsic parameters and the hybrid distortion parameters until the image point loss function of the camera converges, thus completing the camera self-calibration. The regularization term error is the regularization term of the hybrid distortion parameters. The regularization error calculation module is specifically used for: Select the hybrid distortion parameters one by one. For each selected hybrid distortion parameter, calculate the parameter regularization term value corresponding to the selected hybrid distortion parameter. The parameter regularization term values ​​are selected one by one. For each selected parameter regularization term value, the currently selected parameter regularization term value is multiplied by the preset regularization term weight coefficient to obtain the parameter regularization term factor. The regularization term weight coefficients correspond one-to-one with the mixed distortion parameters; The regularization term factors of the parameters are summed to obtain the regularization term error value corresponding to the mixed distortion parameters.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 6.

Images