Pinhole camera calibration method based on convolutional neural network

Abstract

The application discloses a pinhole camera calibration method based on a convolutional neural network, and comprises the following steps: constructing a sample set and an initial convolutional neural network, training the initial convolutional neural network based on the sample set, and obtaining a target convolutional neural network; performing distortion correction and elliptical conversion processing on distorted linear coordinates based on the target convolutional neural network, and obtaining target calibration parameters of the pinhole camera, wherein the target calibration parameters comprise a principal point and a focal length; and performing distortion correction on a pinhole image based on the target calibration parameters. The application combines the geometric prior that a straight line must be straight and that a spatial straight line is an ideal ellipse after spherical projection to construct a convolutional neural network, learns the mapping relationship between a pinhole camera and a corrected straight line and an ideal ellipse, obtains the internal spherical projection space relationship of the pinhole camera, and is thus favorable for complete and accurate estimation of pinhole camera parameters and has good scene transfer learning capability.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to a method for calibrating pinhole cameras based on convolutional neural networks. Background Technology

[0002] The process of checking and calibrating the internal parameters (internal orientation elements and optical distortion parameters) of a camera is called camera calibration. It is the prerequisite and foundation for various image vision tasks, and the completeness and accuracy of the calibration parameters directly affect the effectiveness of the vision task. Currently, common video surveillance cameras, mobile phone cameras, and various digital cameras used for various vision tasks widely adopt the "pinhole" imaging model. Based on the reference object used in the calibration calculation, existing pinhole camera calibration methods can be simply divided into two categories: manual reference object calibration methods and natural scene self-calibration methods. Manual reference object calibration methods require control points with known 3D coordinates or 2D coordinates in a specific reference plane. They often use high-precision 2D checkerboard patterns or indoor 3D control fields. However, the former requires multiple shots from different angles to ensure calibration accuracy, while the latter is more complex to construct and requires continuous maintenance. Natural scene self-calibration methods mainly utilize the geometric projection characteristics of the scene itself under the imaging model, such as "a straight line is still straight," as constraints to automatically establish calibration equations and optimize the solution parameters. Its advantages lie in its high degree of automation and simple application. With the development of deep learning, deep learning techniques, represented by convolutional neural networks (CNNs), have been introduced into the camera calibration process to achieve end-to-end output of camera orientation elements and optical distortion parameters. However, existing pinhole camera CNN calibration algorithms cannot fully obtain the camera's internal parameters, or although they can achieve image distortion pinholes relatively well, they cannot accurately give the distortion coefficients. In addition, due to the tedious sample annotation work and the incompleteness of the dataset itself, the network's generalization and transfer capabilities will be a potential problem. Summary of the Invention

[0003] The purpose of this invention is to provide a pinhole camera calibration method based on convolutional neural networks. This method has good scene transfer learning capabilities and good application value for artificial scenes with a large number of straight lines, thereby solving the problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a pinhole camera calibration method based on a convolutional neural network, comprising the following steps:

[0005] Construct a sample set and an initial convolutional neural network, and train the initial convolutional neural network based on the sample set to obtain the target convolutional neural network;

[0006] Based on the target convolutional neural network, distortion correction and elliptic transformation are performed on the distorted linear coordinates to obtain the target calibration parameters of the pinhole camera. The target calibration parameters include the principal point and the focal length.

[0007] Distortion correction is performed on the pinhole image based on the target calibration parameters.

[0008] Optionally, the process of constructing the sample set includes: obtaining an ideal elliptical arc by performing elliptical transformation on the pinhole image; mapping the pixel coordinates on the pinhole image to the coordinates of the ideal elliptical arc; and using the radius of the hemisphere for spherical projection as the focal length of the pinhole camera; constructing the sample set based on the elliptical transformation process; and managing the sample set in a file format.

[0009] Each file includes calibration parameters randomly generated within a preset value range, a sequence of coordinates for distorted line sampling points, a sequence of coordinates for corrected line sampling points, coordinates for the ideal elliptical arc, and parameters for the ideal elliptical arc.

[0010] Optionally, the process of distortion correction and elliptic transformation of the distorted line coordinates includes: the target convolutional neural network comprising several convolutional modules; acquiring the distorted line coordinates and inputting them into the first convolutional module for processing to obtain the corrected line coordinates, and simultaneously obtaining the corresponding distortion coefficients; inputting the corrected line coordinates into the second convolutional module for processing to obtain new corrected line coordinates; combining the new corrected line coordinates with the direction vector of the corresponding line and inputting them into the third convolutional module for elliptic transformation to obtain ideal elliptical arc coordinates; and obtaining the target calibration parameters of the pinhole camera based on the ideal elliptical arc coordinates.

[0011] Optionally, the process of obtaining the target calibration parameters of the pinhole camera based on the ideal elliptical arc coordinates includes the following steps: the elliptical parameters corresponding to the ideal elliptical arc are the major axis, minor axis, rotation angle, and eccentricity, and the major axis is the focal length of the pinhole camera; the new correction line coordinates are reconstructed based on the fourth convolution module to obtain the reconstructed correction line coordinates, and then the principal point coordinates of the pinhole camera are obtained.

[0012] Optionally, the process of obtaining the corrected line coordinates includes: converting the point coordinates in space into point coordinates in the camera coordinate system based on the rotation matrix and translation matrix; presetting the focal length and principal point of the pinhole camera, and projecting the point coordinates in the camera coordinate system onto the image plane to convert them into point coordinates on the image plane.

[0013] In this process, distortion occurs during imaging. A first distortion coefficient and a second distortion coefficient are preset. Based on the preset principal point coordinates and the coordinates of the distorted point, the distance from the coordinates of the distorted point to the preset principal point coordinates is obtained. Based on the first convolution module, the first distortion coefficient, the second distortion coefficient, the distance from the coordinates of the distorted point to the preset principal point coordinates, the preset principal point coordinates, and the coordinates of the distorted point are processed to obtain the coordinates of the distortion-free point. The coordinates of the distortion-free point are the coordinates of the corrected line.

[0014] Optionally, before performing the ellipse transformation, a hemisphere is defined with the focal length of the pinhole camera as its radius, the principal point of the pinhole camera is tangent to the hemisphere, and the straight line connecting the center of the hemisphere to the principal point coincides with the optical center of the camera.

[0015] Optionally, the process of obtaining the coordinates of the ideal elliptical arc includes converting the new corrected line on the image plane into the corresponding spatial line, mapping the points on the spatial line along the ray formed by the line and the center of the hemisphere onto the surface of the hemisphere to obtain a great circle arc, and then mapping the great circle arc onto the plane where the bottom surface of the hemisphere is located to obtain the corresponding ideal elliptical arc, wherein the center of the ellipse where the ideal elliptical arc is located is the center of the hemisphere.

[0016] The technical effects of this invention are as follows:

[0017] This invention combines the geometric priors that straight lines must be straight and that a straight line in space, when projected onto a sphere, forms an ideal ellipse to construct a convolutional neural network. The geometric prior of straight lines being straight is introduced into the line correction module. By learning the mapping relationship between the corrected straight line and the ideal ellipse, the intrinsic spherical projection spatial relationship of the pinhole camera is obtained. This facilitates the complete and accurate estimation of pinhole camera parameters and exhibits good scene transfer learning capabilities. Overall, it outperforms existing deep learning pinhole camera calibration algorithms and has significant application value for artificial scenes with a large number of straight lines.

[0018] This invention proposes a novel dataset generation method that automatically generates a sample set containing distorted lines, corrected lines, ideal ellipses, and internal parameters of the pinhole camera based on the pixel coordinate mapping relationship between the straight lines and ideal ellipses in a pinhole image under spherical projection. This not only avoids the large amount of sample annotation work required for traditional CCN training, but also, because it focuses on learning explicit scene geometry from the coordinate position information of the pixels where the lines are located, without involving pixel spectral information, the designed network has good scene transfer learning capabilities. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram of a pinhole camera imaging model in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the spherical projection transformation of a pinhole image in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the network structure of the pinhole camera calibration convolutional neural network in an embodiment of the present invention, wherein (a) is a schematic diagram of the convolutional module structure of the convolutional neural network; and (b) is a schematic diagram of the parameter estimation module and distortion correction module structure of the convolutional neural network.

[0023] Figure 4 This is a flowchart of the pinhole camera calibration method in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] Example 1

[0027] like Figure 1-4 As shown, this embodiment provides a pinhole camera calibration method based on a convolutional neural network, including:

[0028] Sample set construction

[0029] By using spherical projection transformation of the pinhole camera imaging model set, distorted straight lines can be converted into ideal elliptical arcs. This allows for a one-to-one correspondence between pixel coordinates p(x,y) on the pinhole image and pixel coordinates q(u,v) on the ideal elliptical arc. Furthermore, by setting the spherical projection radius to the focal length f of the pinhole camera, this transformation process completely encompasses the parameters of the pinhole camera. Based on this, a sample set is constructed, organized and managed in files. Each file records randomly generated lens parameters within a given value range. Original distorted straight line sampling point coordinate sequence Correcting the coordinate sequence of linear sampling points Ideal elliptical arc after spherical projection Regarding the parameters of the ideal elliptical arc, it should be noted that the three rotation angles in the rotation matrix can be selected at equal intervals within a certain range, and the convolutional neural network constructed in this embodiment does not specifically learn the three rotation angles.

[0030] Calibration Convolutional Neural Network Design

[0031] Assuming the distorted straight line on the pinhole image has been detected Using this as input, the structure constructed in this embodiment is as follows Figure 3 The convolutional neural network PinholeCalNet shown is used to achieve end-to-end complete estimation of pinhole camera parameters, with distortion lines. (Sampling points) are the input; the output is the ideal elliptic arc. (Sampling points) and internal parameters of the pinhole camera Furthermore, calibration parameters can be used to perform distortion removal (planar perspective) correction on pinhole images.

[0032] Specifically, the acquired distorted line coordinates (x”, y”) are first input, and the corrected line coordinates are obtained through a convolution module, while the corresponding distortion coefficients are estimated simultaneously. The corrected line coordinates (x’, y’) are then used by a new convolution module to output new corrected line coordinates (x, y). This step essentially learns the intrinsic translation relationship of the principal point offset. Then, (x, y) and the direction vector of the line are combined as input for learning spherical projection, transforming it into ideal elliptical arc coordinates (u, v). The elliptical parameters (a, b, θ, e) corresponding to this ideal elliptical arc are the major and minor axes, rotation angle, and eccentricity, respectively. Since the radius of the projection sphere is set to f, a = f, thus obtaining the camera's focal length. Another convolution module then uses the ideal elliptical arc to reconstruct the corrected line coordinates (x, y)(x’, y’). Using the reconstructed (x, y)(x’, y’), the translation relationship, i.e., the principal point coordinates of the camera, can be obtained, ultimately yielding all the camera intrinsic parameters. The convolutional modules consist of one-dimensional convolutions and multi-head self-attention mechanisms, forming a U-shaped structure similar to the UNet network, and incorporating step skips. The entire network contains multiple convolutional modules, but their inputs and outputs are all different.

[0033] The process of obtaining the corrected line coordinates through the convolution module and simultaneously estimating the corresponding distortion coefficients is as follows:

[0034] The entire pinhole camera imaging process is as follows Figure 1 As shown, through the pinhole imaging model, the point P(Xw,Yw,Zw) in space is projected onto the imaging plane as a pixel P'(u,v) through the optical center O.

[0035] The transformation from the world coordinate system to the camera coordinate system, using a rotation matrix R and a translation matrix T, from a spatial point Pw to Pc in the camera coordinate system, can be expressed as follows:

[0036]

[0037] The transformation from camera coordinates to pixel coordinates is mainly related to focal length and principal point. The relationship between the transformation from Pc in camera coordinates to pixel coordinates can be expressed as:

[0038]

[0039] Integrating the above two steps, assuming that the focal lengths in the x and y directions are the same and there is no translation, the final relationship between P and P' can be expressed as:

[0040]

[0041] Since camera distortion is also involved in the imaging process, this embodiment only considers radial distortion, and takes two distortion coefficients k1 and k2, let P C1 (x C1 ,y C1 P is a point without distortion. D1 (x D1 ,y D1 ) indicates a point of distortion. r is the distance from the distorted point to the principal point, and (u0, v0) are the principal point coordinates.

[0042]

[0043] In the above imaging process, the parameters (u0, v0, f, k1, k2) are related to the optical design and manufacturing of the pinhole camera itself, and are called internal parameters, which are the pinhole camera calibration parameters of this scheme.

[0044] The process of constructing an ideal elliptical arc includes:

[0045] Introducing the constraint that a straight line on a pinhole image becomes an ideal elliptical arc after spherical projection, the pinhole image is transformed by spherical projection. A hemisphere is defined with the camera focal length f as the radius. The principal point of the pinhole image is tangent to the hemisphere, and the straight line connecting the center of the hemisphere and the principal point of the image coincides with the optical center. The transformation process can be divided into the following three steps:

[0046] Spatial coordinate transformation, which utilizes the imaging principle of a pinhole camera, transforms a line segment P on the image plane J into a spatial coordinate transformation. C1 P C2 The transformation is converted into a corresponding spatial line, and the spatial coordinates of the spatial line are based on the O-XYZ image space auxiliary coordinate system. The transformation formula is as follows:

[0047]

[0048] like Figure 2 As shown, [x,y,-f] T Let P be a straight line segment on the image plane J. C1 P C2In the diagram, f is the equivalent focal length of the camera, and R1 represents the coordinate axes O-XYZ. The relative relationships between O-XYZ are given by... ω and κ are obtained, and a1, a2, a3, b1, b2, b3, c1, c2, c3 represent the 9 elements in the rotation matrix R1.

[0049] In spherical projection, after transforming a straight line segment on the image plane J into a spatial straight line, the points on the spatial straight line are mapped onto the hemisphere along the ray formed by the line and the center O of the hemisphere, becoming the great circle arc P. S1 P S2 Its projection formula is as follows:

[0050]

[0051] Orthographic projection, that is, the great circle P on the hemisphere. S1 P S2 Mapping onto the plane containing the OXY plane of the hemispherical base, we finally obtain a segment P. M1 P M2 The ideal elliptical arc is defined by the fact that the center of the ellipse containing this ideal elliptical arc is the center O of the hemisphere. The transformation relationship between the two is as follows:

[0052] x m =X s ,y m =Y s (7)

[0053] The above transformation process is related to the relative extrinsic parameter R1 between the camera and the hemisphere, as well as the camera focal length f. This embodiment mainly focuses on the camera intrinsic parameter, so the transformation only focuses on the focal length f, and successfully introduces the constraint that a straight line becomes an ideal ellipse after being transformed by spherical projection.

[0054] The camera calibration process in this embodiment first constructs a sample set, and then randomly generates pinhole internal parameters within a given value range. Where u0, v0, and f are all in pixels; i = 1, 2, ..., I, where I represents the number of parameter samples. Based on the given parameter value f... i Build size f i ×f i Randomly generate line segments SL on the plane j (j=1,2,...,J). Based on the given pinhole camera parameters... L is calculated using formulas (2), (3), (4), (5), (6), and (7). j Ideal elliptical arc after spherical projection The distortion formula (1) is used to obtain the distorted straight line. Where: (u n ,v n ), (xn ,y n ) respectively represent The coordinates of the same sampling point on the distorted line and the corrected line are given, where N is the number of sampling points. The generated sample set is then used to train the designed PinholeCalNet network. First, the distorted line is corrected, and the corrected line and distortion coefficients are estimated. Then, the corrected line is input to estimate the principal point and focal length of the pinhole camera. Finally, the trained network is applied to real images to estimate parameters, and the calibration parameters are used to perform distortion correction on the pinhole images.

[0055] This embodiment constructs a convolutional neural network, PinholeCalNet, for complete estimation of pinhole camera parameters. This network fully utilizes the geometric prior that straight lines must be straight and that a straight line in space becomes an ideal ellipse after spherical projection for training and learning. It includes two modules: pinhole image straight line correction and pinhole camera parameter estimation. The geometric prior that straight lines must be straight is introduced into the straight line correction module. By learning the mapping relationship between the corrected straight line and the ideal ellipse, the intrinsic spherical projection spatial relationship of the pinhole camera is obtained, which is beneficial for accurate estimation of pinhole camera parameters. This embodiment proposes a novel dataset generation method. Based on the pixel coordinate mapping relationship between the straight line and the ideal ellipse in the pinhole image under spherical projection, a sample set containing distorted / corrected straight lines, the ideal ellipse, and the intrinsic parameters of the pinhole camera is automatically generated. This not only avoids the large amount of sample annotation work required for traditional CCN training and learning, but also, because it focuses on learning explicit scene geometry from the coordinate position information of the pixels where the straight lines are located, without involving pixel spectral information, the designed network will have good scene transfer learning capabilities.

[0056] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pinhole camera calibration method based on a convolutional neural network, characterized in that, The method comprises the following steps: constructing a sample set and an initial convolutional neural network, training the initial convolutional neural network based on the sample set to obtain a target convolutional neural network; based on the target convolutional neural network, performing distortion correction and elliptical conversion processing on the distorted straight line coordinates to obtain target calibration parameters of the pinhole camera, wherein the target calibration parameters comprise a principal point and a focal length; based on the target calibration parameters, performing distortion correction on the pinhole image; the process of constructing the sample set comprises: obtaining an ideal elliptical arc by performing elliptical conversion processing on the pinhole image, one-to-one corresponding the pixel coordinates on the pinhole image to the ideal elliptical arc coordinates, and taking the radius of the spherical projection as the focal length of the pinhole camera; the sample set is constructed based on the process of elliptical conversion processing, and the sample set is managed in a file manner; each file comprises randomly generated calibration parameters, distorted straight line sampling point coordinate sequences, corrected straight line sampling point coordinate sequences, ideal elliptical arc coordinates, and parameters of the ideal elliptical arc in a preset value range; the process of performing distortion correction and elliptical conversion processing on the distorted straight line coordinates comprises: the target convolutional neural network comprises a plurality of convolutional modules; the distorted straight line coordinates are collected and input into a first convolutional module for processing to obtain corrected straight line coordinates and corresponding distortion coefficients; the corrected straight line coordinates are input into a second convolutional module for processing to obtain new corrected straight line coordinates; the new corrected straight line coordinates and the direction vectors of the corresponding straight lines are combined and input into a third convolutional module for elliptical conversion processing to obtain ideal elliptical arc coordinates; and the target calibration parameters of the pinhole camera are obtained based on the ideal elliptical arc coordinates; the process of obtaining the corrected straight line coordinates comprises: based on a rotation matrix and a translation matrix, converting the point coordinates in space into point coordinates in the camera coordinate system; presetting the focal length and the principal point of the pinhole camera; and projecting the point coordinates in the camera coordinate system on the image plane to convert them into point coordinates on the image plane; wherein distortion occurs during imaging, a first distortion coefficient and a second distortion coefficient are preset, the distance from the distorted point coordinates to the preset principal point coordinates is obtained based on the preset principal point coordinates and the distorted point coordinates; the coordinates of the non-distorted point are obtained by processing the first distortion coefficient, the second distortion coefficient, the distance from the distorted point coordinates to the preset principal point coordinates, the preset principal point coordinates, and the coordinates of the distorted point based on the first convolutional module; and the coordinates of the non-distorted point are the corrected straight line coordinates.

2. The pinhole camera calibration method based on the convolutional neural network according to claim 1, wherein the process of obtaining the target calibration parameters of the pinhole camera based on the ideal elliptical arc coordinates comprises: the elliptical parameters corresponding to the ideal elliptical arc are a major axis, a minor axis, a rotation angle, and an eccentricity, and the major axis is the focal length of the pinhole camera; the new corrected straight line coordinates are reconstructed by the fourth convolutional module to obtain reconstructed corrected straight line coordinates, and then the principal point coordinates of the pinhole camera are obtained.

3. The pinhole camera calibration method based on the convolutional neural network according to claim 1, wherein Further comprising, before the elliptical conversion, setting a hemisphere with the focal length of the pinhole camera as the radius, the principal point of the pinhole camera being tangent to the hemisphere, and the line connecting the principal point and the center of the hemisphere being coincident with the optical center of the camera.

4. The pinhole camera calibration method based on a convolutional neural network according to claim 3, wherein, The process of obtaining the ideal elliptical arc coordinates comprises converting the new rectified straight line coordinates on the image plane into a corresponding space straight line, mapping a point on the space straight line to a great circle arc on the hemisphere surface along a ray formed by the center of the hemisphere, and then mapping the great circle arc to a plane on which the bottom surface of the hemisphere is located to obtain a corresponding ideal elliptical arc, the center of the ideal elliptical arc being the center of the hemisphere.

Images