Camera calibration method and device

Abstract

The embodiment of the present application provides a camera calibration method and device, which relates to the field of terminal technology, and solves the problem of low image acquisition efficiency in the camera calibration method, which leads to low efficiency in estimating the model parameters of the camera. The specific scheme is: detect at least one first image to obtain multiple corner point coordinate information, wherein the first image is obtained by the camera shooting a chessboard three-dimensional structure, and the chessboard three-dimensional structure includes multiple first chessboard patterns located on at least two different planes; obtain the model parameters of the camera according to the multiple corner point coordinate information. The embodiment of the present application is used for the process of obtaining the internal and external parameters of the camera calibration.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a camera calibration method and device. Background Art

[0002] The camera lens model parameters can be obtained through camera calibration (camera calibration), and the model parameters include intrinsic parameters and extrinsic parameters. For cameras with radially distorted lenses, it is also necessary to estimate the distortion coefficient of the lens.

[0003] At present, the calibration methods used by most cameras are based on the calibration method proposed by Zhang Zhengyou. OpenCV and Matrix Laboratory (Matlab) also provide corresponding toolboxes and open source libraries to implement this calibration method. Zhang Zhengyou's calibration method uses a pinhole imaging model and a set of plane images to obtain calibration results (model parameters). Due to the interference of various noises during the acquisition process, camera calibration usually collects the corner coordinates of the checkerboard pattern in dozens of images. Each image includes a checkerboard pattern. The checkerboard pattern included in each image is the same, and only one image can be collected at a time. In order to simulate the radial distortion of the lens well, the angle and distance of the image are different each time to fully cover as many lens field of view (FOV) as possible. The corner coordinates of these images are then input into the optimization algorithm to obtain the model parameters.

[0004] Typically, this method requires at least 11 images to achieve sub-pixel performance. At the same time, in order to better model the distortion of the lens, the chessboard pattern needs to be placed to fully cover the corners and edges of the camera FOV, and the images located at the corners and edges of the lens will be severely distorted. This requirement poses a huge challenge to the shooting process, because the position of the camera (or chessboard) must be carefully set each time to ensure coverage of the FOV. Therefore, in the current calibration method, a large number of images need to be collected, and the process of collecting each image is complicated, resulting in low efficiency in collecting images and low efficiency in estimating the model parameters of the camera, making the current calibration method unsuitable for production purposes. Summary of the invention

[0005] The embodiments of the present application provide a camera calibration method and device, which solve the problem of low efficiency in image acquisition and low efficiency in estimating camera model parameters in the camera calibration method.

[0006] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0007] In a first aspect, a camera calibration method is provided, comprising: detecting at least one first image to obtain multiple corner point coordinate information, wherein the first image is obtained by photographing a checkerboard three-dimensional structure by a camera, and the checkerboard three-dimensional structure includes multiple first checkerboard patterns located on at least two different planes; and obtaining model parameters of the camera according to the multiple corner point coordinate information.

[0008] Therefore, when calculating the model parameters of camera calibration, the present application uses a first image containing a three-dimensional structure, and the first image includes multiple checkerboard patterns of the same structure. In this way, through one first image, the corner point coordinates of multiple checkerboard patterns in the first image on different planes can be obtained. In this way, the model parameters of the camera lens can be calculated through a small number of first images, such as 1-3 first images. The efficiency of collecting the corner point coordinates of the checkerboard pattern can be improved, thereby improving the efficiency of estimating the model parameters of the camera, which is suitable for production purposes.

[0009] In a possible design, the multiple first checkerboard patterns do not overlap each other, and the multiple first checkerboard patterns are located in different planes. Here, the multiple first checkerboard patterns are located in at least two different planes and are not parallel to each other.

[0010] In a possible design, the corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point; detecting at least one first image to obtain multiple corner point coordinate information includes: obtaining the two-dimensional coordinates of multiple corner points of each first checkerboard pattern in multiple first checkerboard patterns; the two-dimensional coordinates are used to represent the pixel point position of the corner point on the plane where the first image is located; and obtaining the three-dimensional coordinates of the multiple corner points according to the two-dimensional coordinates of the multiple corner points. The two-dimensional coordinates can be understood as the pixel coordinates of each corner point, and the three-dimensional coordinates can be understood as the world coordinates. When calculating the model parameters of the camera, the intrinsic parameters and extrinsic parameters of the camera can be obtained according to the pixel coordinates and the world coordinates.

[0011] In a possible design, obtaining the two-dimensional coordinates of multiple corner points of each first checkerboard pattern in the multiple first checkerboard patterns includes: selecting a first checkerboard pattern from the multiple first checkerboard patterns of the first image as the target checkerboard pattern; obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern; after obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, setting all pixels in the area where the target checkerboard pattern is located to a single value; selecting a second target checkerboard pattern from the remaining multiple first checkerboard patterns, and obtaining the two-dimensional coordinates of each corner point in the second target checkerboard pattern. In other words, one of the first checkerboard patterns can be selected, and the two-dimensional coordinates of the corner points of the first checkerboard pattern can be obtained, and then the color of the first checkerboard pattern for which the two-dimensional coordinates have been obtained can be set to a single color, that is, the mark has been detected, and one of the first checkerboard patterns can be selected from the remaining patterns to obtain the two-dimensional coordinates of its corner points.

[0012] In a possible design, before obtaining the model parameters of the camera based on multiple corner point coordinate information, the method also includes: detecting a second image to obtain multiple corner point coordinate information of a second checkerboard pattern; the second image is obtained by the camera shooting the second checkerboard pattern. Although the first image includes multiple first checkerboard patterns, there are gaps between the multiple first checkerboard patterns and between the first checkerboard patterns and the edges, and the edges of the first image are not covered, that is, the entire FOV of the camera lens is not completely covered. Therefore, the present application can also obtain the corner point coordinate information of a second checkerboard pattern based on the first image, and the second checkerboard pattern can be a full-screen checkerboard pattern, that is, it can cover the entire FOV.

[0013] In one possible design, obtaining the model parameters of the camera according to the multiple corner point coordinate information includes: obtaining the model parameters of the camera according to the multiple corner point coordinate information of the first checkerboard patterns and the multiple corner point coordinate information of the second checkerboard patterns.

[0014] In a possible design, when capturing the second image, the second checkerboard pattern fills the entire field of view FOV of the camera lens; or, the second checkerboard pattern fills the entire second image. The second checkerboard pattern may be a full-screen checkerboard pattern, in which the number of black and white grids may be greater than the number of black and white grids in the first checkerboard pattern.

[0015] In a possible design, the corner points are the intersection points of every four adjacent grids in the first checkerboard pattern or the second checkerboard pattern, that is, the intersection points of grids at the edge of the checkerboard are not regarded as corner points.

[0016] In a possible design, the number of first checkerboard patterns included in the checkerboard three-dimensional structure is 4, 5, 7, 11 or 15.

[0017] In a second aspect, a calibration device is provided, comprising: a detection unit, used to detect at least one first image to obtain multiple corner point coordinate information, wherein the first image is obtained by photographing a checkerboard three-dimensional structure by a camera, and the checkerboard three-dimensional structure includes multiple first checkerboard patterns located on at least two different planes; an acquisition unit, used to acquire model parameters of the camera based on the multiple corner point coordinate information.

[0018] In a possible design, the multiple first chessboard patterns do not overlap each other, and the multiple first chessboard patterns are located in different planes.

[0019] In one possible design, the corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point; the detection unit is used to: obtain the two-dimensional coordinates of multiple corner points of each first checkerboard pattern in multiple first checkerboard patterns; the two-dimensional coordinates are used to represent the pixel point position of the corner point on the plane where the first image is located; and obtain the three-dimensional coordinates of the multiple corner points based on the two-dimensional coordinates of the multiple corner points.

[0020] In one possible design, the detection unit is used to: select a first checkerboard pattern from multiple first checkerboard patterns in a first image as a target checkerboard pattern; obtain the two-dimensional coordinates of each corner point in the target checkerboard pattern; after obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, set all pixels in the area where the target checkerboard pattern is located to a single value; select a second target checkerboard pattern from the remaining multiple first checkerboard patterns, and obtain the two-dimensional coordinates of each corner point in the second target checkerboard pattern.

[0021] In a possible design, the detection unit is further used to: detect the second image to obtain coordinate information of multiple corner points of the second chessboard pattern; the second image is obtained by photographing the second chessboard pattern with a camera.

[0022] In a possible design, the acquisition unit is used to acquire the model parameters of the camera according to the coordinate information of multiple corner points of the first checkerboard patterns and the coordinate information of multiple corner points of the second checkerboard patterns.

[0023] In a possible design, when capturing the second image, the second checkerboard pattern fills the entire field of view FOV of the camera lens; or, the second checkerboard pattern fills the entire second image.

[0024] In a possible design, the corner point is an intersection point of every four adjacent grids in the first checkerboard pattern or the second checkerboard pattern.

[0025] In a possible design, the number of first checkerboard patterns included in the checkerboard three-dimensional structure is 4, 5, 7, 11 or 15.

[0026] According to a third aspect, a calibration device is provided, comprising a processor and a transmission interface, the transmission interface being used to receive or send image data; the processor being configured to call program instructions in a memory; when the program instructions are called, the processor being used to perform the following operations: detecting at least one first image to obtain multiple corner point coordinate information, wherein the first image is obtained by photographing a checkerboard three-dimensional structure by a camera, the checkerboard three-dimensional structure comprising multiple first checkerboard patterns located on at least two different planes; and obtaining the model parameters of the camera according to the multiple corner point coordinate information.

[0027] In a possible design, the multiple first chessboard patterns do not overlap each other, and the multiple first chessboard patterns are located in different planes.

[0028] In one possible design, the corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point; the operations performed by the processor include: obtaining the two-dimensional coordinates of multiple corner points of each first checkerboard pattern in multiple first checkerboard patterns; the two-dimensional coordinates are used to represent the pixel position of the corner point on the plane where the first image is located; and obtaining the three-dimensional coordinates of the multiple corner points based on the two-dimensional coordinates of the multiple corner points.

[0029] In one possible design, the operations performed by the processor include: selecting a first checkerboard pattern from multiple first checkerboard patterns in a first image as a target checkerboard pattern; obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern; after obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, setting all pixels in the area where the target checkerboard pattern is located to a single value; selecting a second target checkerboard pattern from the remaining multiple first checkerboard patterns, and obtaining the two-dimensional coordinates of each corner point in the second target checkerboard pattern.

[0030] In a possible design, the operation performed by the processor also includes: detecting the second image to obtain coordinate information of multiple corner points of the second chessboard pattern; the second image is obtained by photographing the second chessboard pattern with a camera.

[0031] In one possible design, the processor is configured to perform operations including: acquiring model parameters of the camera based on multiple corner point coordinate information of the first checkerboard patterns and multiple corner point coordinate information of the second checkerboard patterns.

[0032] In a possible design, when capturing the second image, the second checkerboard pattern fills the entire field of view FOV of the camera lens; or, the second checkerboard pattern fills the entire second image.

[0033] In a possible design, the corner point is an intersection point of every four adjacent grids in the first checkerboard pattern or the second checkerboard pattern.

[0034] In a possible design, the number of first checkerboard patterns included in the checkerboard three-dimensional structure is 4, 5, 7, 11 or 15.

[0035] In a fourth aspect, a readable storage medium is provided, comprising program instructions. When the program instructions are executed on a processor or a computer, the method described in the first aspect and any possible design of the first aspect is executed.

[0036] According to a fifth aspect, a computer program product is provided. When the computer program product runs on a computer or a processor, the method described in the first aspect and any possible design of the first aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a checkerboard pattern provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of corner points of a checkerboard pattern provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a camera calibration method flow provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a first checkerboard pattern in a first image provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a first checkerboard pattern in a first image provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of a first checkerboard pattern in a first image provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of a first checkerboard pattern in a first image provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a first checkerboard pattern in a first image provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of detecting a first chessboard pattern marked as having been detected in a first image provided by an embodiment of the present application;

[0046] Fig.10 A schematic diagram of a camera calibration method flow provided in an embodiment of the present application;

[0047] Fig.11A schematic diagram of a second checkerboard pattern in a second image provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0049] Fig.13 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0050] Fig.14 A schematic diagram of an application scenario of a camera calibration method provided in an embodiment of the present application;

[0051] Fig.15 A schematic diagram of the hardware architecture of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] For ease of understanding, some examples are provided for reference. They are as follows:

[0053] Lens calibration: In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are camera parameters (model parameters). Under most conditions, these parameters must be obtained through experiments and calculations. The process of solving parameters (intrinsic parameters, external parameters, and distortion parameters) is called camera calibration (or camera calibration). Whether in image measurement or machine vision applications, the calibration of camera parameters is a very critical link. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera.

[0054] Pixel coordinate system: used to describe the coordinates of the pixel points of an object on a digital image (photo). It can also be understood as a plane rectangular coordinate system (x f ,y f ), in one optional case, its origin is at the upper left corner of the image, x f Parallel to the X-axis of the image's physical coordinate system, y f Parallel to the Y-axis of the image's physical coordinate system, in units of (number of pixels).

[0055] Camera coordinate system: A coordinate system established on a camera, defined to describe the position of an object from the camera's perspective, as an intermediate link between the world coordinate system and the pixel coordinate system. The unit is m. The camera coordinate system is a three-dimensional rectangular coordinate system with the origin at the optical center of the lens. The x and y axes are parallel to the sides of the image plane, respectively, and the z axis is the optical axis of the lens, which is perpendicular to the image plane.

[0056] World coordinate system: A user-defined three-dimensional coordinate system introduced to describe the position of objects in the real world. The unit is m.

[0057] Internal parameters: used to determine the projection relationship of the camera from the three-dimensional space to the two-dimensional image. Including the focal length f(α x , α y ), radial distortion magnitude k, scaling factor (s), and the principal point of the image (u0, v0, the position of the center point of the image), etc.

[0058] External parameters: used to determine the relative position relationship between the camera coordinate system and the world coordinate system. If Pw is the world coordinate system and Pc is the camera coordinate system, the relationship between them is Pc=RPw+T. In the formula, T=(Tx,Ty,Tz) is the translation vector, and R=R(α,β,γ) is the rotation matrix. Among them, the rotation angle around the z-axis of the camera coordinate system is γ, the rotation angle around the y-axis is β, and the rotation angle around the x-axis is α. These 6 parameters (α, β, γ, Tx, Ty, Tz) are the camera external parameters.

[0059] Checkerboard pattern: In Zhang Zhengyou calibration method, the checkerboard pattern is usually composed of multiple black and white grids, such as Figure 1 shown.

[0060] Corner point: usually refers to Figure 1 The inner corners of the checkerboard pattern shown. Figure 2 , Figure 2 The point where the four adjacent 2×2 black and white squares (2 black squares + 2 white squares) intersect is called a corner point. Usually, the vertices at the corners of the outer edge squares of the chessboard pattern are not counted as corner points.

[0061] Field of view (FOV): In optical instruments, the angle formed by the two edges of the maximum range through which the image of the measured object can pass through the lens, with the lens of the optical instrument as the vertex, is called the field of view. The size of the field of view determines the field of view of the optical instrument. The larger the field of view, the larger the field of view and the smaller the optical magnification. In layman's terms, the target object will not be included in the lens if it exceeds the FOV.

[0062] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0063] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0064] The embodiments of the present application can be applied to the application scenario of camera lens calibration.

[0065] The embodiment of the present application can be used to execute on the terminal device side. When the terminal device obtains the checkerboard pattern captured by the camera, the terminal device can obtain the model parameters of the camera lens according to the checkerboard pattern, and the model parameters include internal parameters and external parameters. For example, the terminal device can be a personal computer (PC) or a mobile smart phone. The method of the embodiment of the present application can be an application executed on the PC side or the mobile smart phone side.

[0066] The basic principles of this application are introduced below.

[0067] In view of the problem that a large number of checkerboard images need to be collected in the current calibration method, the process of collecting each checkerboard image is complicated, resulting in low efficiency in collecting checkerboard images and low efficiency in estimating the model parameters of the camera, the embodiment of the present application provides a camera calibration method, in which an image containing a three-dimensional structure can be used, the image includes multiple checkerboard patterns of the same structure, and the corner point coordinates of the multiple checkerboard patterns in the image are obtained to calculate the model parameters of the camera lens according to the corner point coordinates. Compared with the current calibration method, which can only collect the corner point coordinates of one checkerboard pattern at a time, and the position of the checkerboard pattern to be placed at different times, resulting in low collection efficiency, the present application can improve the efficiency of collecting the corner point coordinates of the checkerboard pattern by setting multiple checkerboard patterns of a three-dimensional structure in one image, thereby improving the efficiency of estimating the model parameters of the camera, and is suitable for production purposes.

[0068] Based on the above principles, the present application embodiment provides a method for camera calibration, such as Figure 3 As shown, including:

[0069] 301. A terminal device detects at least one first image to obtain coordinate information of multiple corner points, wherein the first image is obtained by photographing a checkerboard three-dimensional structure by a camera, and the checkerboard three-dimensional structure includes multiple first checkerboard patterns located on at least two different planes.

[0070] It should be noted that the multiple first checkerboard patterns in the three-dimensional structure of the present application are located on at least two different planes and are not parallel to each other.

[0071] In some embodiments, in the first image, the plurality of first checkerboard patterns do not overlap with each other, and the plurality of first checkerboard patterns are located in different planes.

[0072] In some embodiments, in the first image, the number of the first checkerboard patterns may be 4 (eg Figure 4 As shown), 5 (as Figure 5 ), 7 (as shown Figure 6 ), or 11 (as shown Figure 7 as shown) or 15 (as shown Figure 8 The number may also be other numbers, which is not limited in this application.

[0073] For example, the number of first checkerboard patterns is 11. The size specifications of these 11 first checkerboard patterns are the same, and the number of squares and the width of the squares in each first checkerboard pattern are the same. These 11 first checkerboard patterns try to fill the entire field of view (FOV) of the camera lens, but in order to prevent distortion, blank areas are left between adjacent checkerboard patterns, and there is a gap between the first checkerboard pattern at the edge and the edge of the first image. In order to avoid the situation where the checkerboard pattern is too small to be detected, the area occupied by each first checkerboard pattern in the camera should be about 1 / 16 of the entire screen.

[0074] In some embodiments, before detecting at least one first image, a camera may be used to shoot the checkerboard three-dimensional structure to obtain at least one first image, wherein different first images correspond to different shooting positions.

[0075] In the embodiment of the present application, at least one first image may be 1 first image, 2 first images, or 3 first images. If 2 first images are used as calibration images, the shooting angles may be slightly different, and there are 11 first checkerboard patterns in each first image. It can be seen that there are 54 corner points in the first checkerboard pattern, and 11 first checkerboard patterns have 594 corner points. To obtain the corner point coordinate information of the 2 first images, a total of 1188 corner point coordinate information must be obtained.

[0076] In some embodiments, the corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point. The two-dimensional coordinates can be understood as the pixel coordinates of the corner point in the pixel coordinate system. The three-dimensional coordinates can be understood as the coordinates of the corner point in the world coordinate system.

[0077] In some embodiments, obtaining the two-dimensional coordinates of multiple corner points of each of the multiple first checkerboard patterns may include: selecting a first checkerboard pattern from the multiple first checkerboard patterns of the first image as a target checkerboard pattern; obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern; after obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, setting all pixels in the area where the target checkerboard pattern is located to a single value; selecting a second target checkerboard pattern from the remaining multiple first checkerboard patterns, and obtaining the two-dimensional coordinates of each corner point in the second target checkerboard pattern.

[0078] Exemplary, reference Figure 4 The first chessboard pattern can be used as a preset chessboard pattern in advance, and the number of grids and the size of the grids of the first chessboard pattern are pre-configured. Figure 4 When the first checkerboard pattern in the upper left corner is the target checkerboard pattern, after obtaining the first corner point coordinate information, the two-dimensional coordinates of each corner point in the target checkerboard pattern can be obtained according to the first corner point coordinate information and the preset checkerboard pattern.

[0079] The two-dimensional coordinates in the corner point coordinate information obtained here can be obtained by calling a function in opencv, for example, the function can be findchessboardcorners() function. It can also be obtained by other methods, which are not limited in this application. After the two-dimensional coordinates of each corner point in a target chessboard pattern are obtained according to the function, the area of ​​the target chessboard pattern in the first image can be detected according to the preset pattern structure of the target chessboard pattern. At this time, all pixels in the area where the target chessboard pattern is located can be set to a single value according to the detected area, for example, all pixels are set to a black value, such as Fig. 9 As shown. Then, a second first checkerboard pattern can be randomly selected from the remaining multiple first checkerboard patterns as the target checkerboard, and the function can be called again to obtain the two-dimensional coordinates of each corner point in the second target checkerboard pattern. Repeat this process until the function cannot detect the checkerboard pattern.

[0080] In this way, after obtaining the two-dimensional coordinates of the corner points of the 11 first checkerboard patterns in the first image, the three-dimensional coordinates of each corner point can be calculated through coordinate transformation based on the two-dimensional coordinates of all the corner points of the 11 first checkerboard patterns.

[0081] In some embodiments, the two-dimensional coordinates of the corner points can be converted into three-dimensional coordinates through the function cvReprojectImageTo3D in opencv.

[0082] 302. The terminal device obtains model parameters of the camera according to the coordinate information of multiple corner points.

[0083] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position (3D coordinates) of a point on the surface of a spatial object and its corresponding point (2D coordinates) in the image, the object in the space is imaged in the image, and the two-dimensional geometric position (2D coordinates) in the image is calculated based on the three-dimensional geometric position (3D coordinates) and the model parameters of the camera.

[0084] To obtain the model parameters of the camera, in some embodiments, camera calibration is a process of converting from a world coordinate system to an image coordinate system, that is, a process of obtaining the final projection matrix. The external and internal parameters of the camera can be obtained by the projection process.

[0085] Generally speaking, the calibration process is divided into two parts:

[0086] The first step is to convert from the world coordinate system to the camera coordinate system. This step is the conversion from 3D point to 3D point, including parameters such as R and T (external parameters, which determine the position and orientation of the camera in a certain 3D space);

[0087] The second step is to convert from the camera coordinate system to the imaging plane coordinate system (pixel coordinate system). This step is the conversion from three-dimensional points to two-dimensional points, including parameters such as K (intrinsic parameter, which is an approximation of the physical properties of the camera).

[0088] The conversion formula from the world coordinate system to the pixel coordinate system can be:

[0089]

[0090] in, Represents the three-dimensional coordinates of a point in the world coordinate system, Represents the two-dimensional coordinates of the pixel point in the pixel coordinate system. s is the scale factor, and M1 corresponds to the parameters (s, α x , α y , u0, v0) is the matrix of internal parameters, and the matrix including parameters (R, T) corresponding to M2 is the matrix of external parameters, and M is the projection matrix from the world coordinate system to the pixel coordinate system.

[0091] Since the lens usually has some distortion, the main distortion is radial distortion, and there will also be slight tangential distortion. If distortion is also considered, the model parameters also include distortion coefficients, which are used to correct distortion during the imaging process.

[0092] It should be noted that the calibration of the camera in the embodiments of the present application is described using a checkerboard pattern, but is not limited to a checkerboard pattern, and may also be a calibration pattern such as a dot calibration plate, or other calibration patterns.

[0093] Steps 301 and 302 are applicable to a camera lens with very little radial distortion or almost no radial distortion.

[0094] Therefore, when calculating the model parameters of camera calibration, the present application uses a first image containing a three-dimensional structure, and the first image includes multiple checkerboard patterns of the same structure. In this way, through one first image, the corner point coordinates of multiple checkerboard patterns in the first image on different planes can be obtained. In this way, the model parameters of the camera lens can be calculated through a small number of first images, such as 1-3 first images. The efficiency of collecting the corner point coordinates of the checkerboard pattern can be improved, thereby improving the efficiency of estimating the model parameters of the camera, which is suitable for production purposes.

[0095] In order to estimate the distortion coefficient of the lens more accurately, the corner points of the checkerboard pattern can fully cover the entire FOV of the camera lens. In particular, for the corners and edge areas where the lens distortion is most serious, the coordinate information of the corner points of these corners and edge areas can also be calculated.

[0096] Therefore, in the calibration process of the camera, the present application can also capture a full-screen second image through the camera lens. When capturing the second image, the second checkerboard pattern fills the entire FOV of the camera lens; or, the second checkerboard pattern fills the entire second image. That is, within the FOV of the camera lens, there is no other image information except the second checkerboard pattern. When capturing the second checkerboard pattern, care should be taken to avoid tilting the camera and the checkerboard at too large an angle to prevent the grid from being too small to be detected.

[0097] Therefore, the embodiment of the present application also provides a camera calibration method, such as Fig.10 As shown, including:

[0098] 801. A terminal device detects at least one first image to obtain coordinate information of multiple corner points, wherein the first image is obtained by photographing a checkerboard three-dimensional structure by a camera, and the checkerboard three-dimensional structure includes multiple first checkerboard patterns located on at least two different planes.

[0099] The implementation of step 801 may refer to the description in step 301 .

[0100] 802. The terminal device detects a second image to obtain coordinate information of multiple corner points of a second chessboard pattern; the second image is obtained by photographing the second chessboard pattern with a camera.

[0101] For example, Fig.11 As shown, when the second checkerboard pattern fills the second image, the size and density of the second checkerboard pattern can provide hundreds of corner points required for calibration by usually taking one second image. For example, when the second checkerboard pattern fills the second image, the number of grids can be 300*200.

[0102] In some embodiments, the number of captured second images may be 1, 2, or 3, etc.

[0103] In some embodiments, when detecting multiple corner point coordinate information of the second checkerboard pattern, detection can be started from the square at the center of the second checkerboard pattern. For example, when detecting the corner point coordinate information of the center square, the center square can be used as a reference to obtain the corner point coordinate information of multiple squares around the center square. For the pixels in the image area of ​​the square that has been detected, a single value can be set, and then continue to diffuse outward to obtain the corner point coordinate information of the squares around the previously detected square, until the square at the edge of the second checkerboard pattern can no longer obtain the corner point coordinates of the edge square (since the image distortion at the edge is the most serious, the corner point coordinate information of the edge square may not be obtained).

[0104] 803. The terminal device obtains model parameters of the camera according to the coordinate information of multiple corner points of the first chessboard patterns and the coordinate information of multiple corner points of the second chessboard patterns.

[0105] In some embodiments, the camera's intrinsic and extrinsic parameters can be calculated based on the algorithm of step 302 by combining the acquired information of multiple corner point coordinates of the first checkerboard patterns and the acquired information of multiple corner point coordinates of the second checkerboard patterns.

[0106] In this way, by adding a full-screen checkerboard pattern on the basis of the three-dimensional structure, the coverage of corner points at the edges and corners of the lens FOV can be improved, thereby improving the calibration effect.

[0107] It should be noted that the embodiments of the present application can be used for lens distortion correction, can also be used to provide more accurate intrinsic and extrinsic parameter information for digital image stabilization, can also be used to evaluate the characteristics of each lens, etc.

[0108] It is understandable that in order to implement the above functions, the terminal device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0109] In this embodiment, the terminal device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0110] In the case of dividing each functional module into corresponding functional modules, Fig.12 A possible schematic diagram of the composition of the terminal device 12 involved in the above embodiment is shown. Fig.12 As shown, the terminal device 12 may include: a detection unit 121 and an acquisition unit 122 .

[0111] Among them, the detection unit 121 can be used to support the terminal device 12 to execute the above-mentioned steps 301, 801, 802, etc., and / or other processes for the technology described in this document.

[0112] The acquisition unit 122 may be used to support the terminal device 12 in executing the above-mentioned steps 302 and 803, etc., and / or other processes of the technology described herein.

[0113] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0114] The terminal device 12 provided in this embodiment is used to execute the above-mentioned camera calibration method, and thus can achieve the same effect as the above-mentioned implementation method.

[0115] In the case of using an integrated unit, the terminal device 12 may include a processing module and a storage module. The processing module may be used to control and manage the actions of the terminal device 12, for example, it may be used to support the terminal device 12 to execute the steps executed by the above-mentioned detection unit 121 and acquisition unit 122. The storage module may be used to support the terminal device 12 to store program codes and data, etc.

[0116] The processing module may be a processor or a controller. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0117] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment may be a Fig.13 The terminal device 13 has the structure shown.

[0118] The embodiment of the present application further provides a terminal device 13, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the terminal device executes the above-mentioned related method steps to implement the camera calibration method in the above-mentioned embodiment.

[0119] like Fig.14 , which is an exemplary application scenario of the camera calibration method provided by the present application. In this scenario, the calibration process of the present application may include a detection process of the corner point coordinate information of two chessboards, and a parameter estimation process. Among them, the detection process of the corner point coordinate information of the two chessboards includes: detection of the corner point coordinate information of the standard chessboard three-dimensional structure, and detection of the corner point coordinate information of the full-screen chessboard three-dimensional structure. The parameter estimation process includes: calculating the information of the corner point coordinates detected by the two chessboards through an optimization algorithm to obtain model parameters, including internal parameters and external parameters.

[0120] In some embodiments, the above calibration process can be implemented by the calibration device in this application, or by Fig.13 The processor in the terminal device 13 is provided for implementation. For example, the terminal device 13 may be a PC, and the processor is a processor in the PC; or the terminal device 13 may also be a camera, and the processor may be a processor of the camera; or the terminal device 13 may also be a mobile smart phone, and the processor may be a processor of the mobile smart phone.

[0121] In some embodiments, the above-mentioned detection of the corner point coordinate information of the standard checkerboard three-dimensional structure may be the detection of the corner point coordinate information of multiple first checkerboard patterns in the first image in the present application; the above-mentioned detection of the corner point coordinate information of the full-screen checkerboard three-dimensional structure may be the detection of the corner point coordinate information of the second checkerboard pattern in the second image in the present application; the above-mentioned optimization algorithm may be the process of obtaining the model parameters of the camera according to the multiple corner point coordinate information of the multiple first checkerboard patterns and the multiple corner point coordinate information of the second checkerboard pattern in the present application.

[0122] refer to Fig.14 After the model parameters are obtained by the optimization algorithm, the model parameters can be used to correct lens distortion; they can also be used for digital video stabilization based on gyroscope sensors, etc.

[0123] In some embodiments, the correction of lens distortion may be performed by a processor on the camera side; or, the correction of lens distortion may be performed by a processor on the mobile smartphone side;

[0124] In some embodiments, the digital video stabilization may be electronic video stabilization, or digital video stabilization.

[0125] like Fig.15 FIG. 1 is a schematic diagram of a hardware architecture of an exemplary image processing device provided in an embodiment of the present application. The hardware architecture of the image processing device 15 is applicable to Fig.13 The processor in.

[0126] Exemplarily, the image processing device 15 includes at least one central processing unit (CPU), at least one memory, a graphics processing unit (GPU), a decoder, a dedicated video / graphics processor, a receiving interface, a sending interface, and the like.

[0127] Optionally, the image processing device 15 may also include a microprocessor and a microcontroller unit (MCU). In an optional case, the above-mentioned various parts of the image processing device 15 are coupled through a connector. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. This embodiment does not limit this. In an optional case, the above-mentioned various parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface and sending interface are integrated on a chip, and the various parts inside the chip access the external memory through the bus. The dedicated video / graphics processor can be integrated with the CPU on the same chip, or it can exist as a separate processor chip. For example, the dedicated video / graphics processor can be a dedicated image signal processor (image signal processor, ISP). The chip involved in the embodiments of the present application is a system manufactured on the same semiconductor substrate using an integrated circuit process, also called a semiconductor chip, which can be a collection of integrated circuits formed on the substrate (usually a semiconductor material such as silicon) using an integrated circuit process, and its outer layer is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various functional devices, each of which includes transistors such as logic gate circuits, metal-oxide-semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include other components such as capacitors, resistors or inductors. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

[0128] Optionally, the CPU may be a single-core processor or a multi-core processor; Optionally, the CPU may be a processor group consisting of multiple processors, and the multiple processors are coupled to each other through one or more buses. In an optional case, the processing of the image signal or video signal is partially completed by the GPU, partially completed by a dedicated video / graphics processor, and possibly completed by software code running on a general-purpose CPU or GPU.

[0129] The memory can be used to store computer program instructions, including an operating system (OS), various user applications, and various types of computer program codes including program codes for executing the scheme of the present application; the memory can also be used to store video data, image signal data, etc.; the CPU can be used to execute the computer program code stored in the memory to implement the method in the embodiment of the present application. Optionally, the memory may be a non-power-off volatile memory, such as an embedded multi media card (EMMC), universal flash storage (UFS) or read-only memory (ROM), or other types of static storage devices that can store static information and instructions. It may also be a power-off volatile memory (volatile memory), such as a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other computer-readable storage medium that can be used to carry or store program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0130] The receiving interface may be a data input interface of a processor chip. In an optional case, the receiving interface may be a high definition multimedia interface (HDMI).

[0131] Based on the hardware architecture of the above image processing device, if the PC includes the image processing device, the program instructions corresponding to the camera calibration method provided in this application can be stored in the memory of the PC; the CPU of the PC can run the program instructions through the OpenCV or Matlab program to implement the camera calibration method provided in this application. The memory can also be used to store the data of the first image and / or the data of the second image in this application; in some embodiments, if the calibration method provided in this application is executed on the PC, the first image and / or the second image of this application can be transmitted from the camera to the PC through the network, or can be copied from the camera to the PC.

[0132] In some embodiments, when the program instructions corresponding to the camera calibration method provided in the present application can be stored in a memory on the PC side, the program instructions can also be executed by a GPU or a dedicated video / graphics processor.

[0133] In some embodiments, when a terminal such as a video camera or a mobile smart phone includes the image processing device, the CPU of the terminal may execute the program instructions of the camera calibration method, and the memory of the terminal may store the program instructions.

[0134] In some embodiments, when a terminal such as a video camera or a mobile smart phone includes the image processing device, the GPU or a dedicated video / graphics processor of the terminal may execute program instructions of the camera calibration method.

[0135] An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the camera calibration method in the above-mentioned embodiment.

[0136] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the camera calibration method executed by the terminal device in the above-mentioned embodiment.

[0137] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor may execute the computer execution instructions stored in the memory so that the chip executes the camera calibration method executed by the terminal device in the above-mentioned method embodiments.

[0138] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0139] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0144] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A camera calibration method, characterized in that: include: Detecting at least one first image to obtain multiple corner point coordinate information, wherein the first image is obtained by the camera shooting a checkerboard three-dimensional structure, and the checkerboard three-dimensional structure includes a plurality of first checkerboard patterns located on at least two different planes; Detecting a second image to obtain coordinate information of multiple corner points of a second checkerboard pattern; the second image is obtained by the camera shooting the second checkerboard pattern, and when shooting the second image, the second checkerboard pattern fills the entire field of view FOV of the lens of the camera; or, the second checkerboard pattern fills the entire second image; The model parameters of the camera are obtained according to the coordinate information of the multiple corner points of the multiple first checkerboard patterns and the coordinate information of the multiple corner points of the second checkerboard patterns.

2. The method according to claim 1, characterized in that The multiple first chessboard patterns do not overlap each other, and the multiple first chessboard patterns are located in different planes.

3. The method according to claim 1 or 2, characterized in that: The corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point; The detecting at least one first image to obtain a plurality of corner point coordinate information comprises: Acquire two-dimensional coordinates of a plurality of corner points of each of the plurality of first checkerboard patterns; the two-dimensional coordinates are used to represent pixel positions of the corner points on the plane where the first image is located; The three-dimensional coordinates of the multiple corner points are acquired according to the two-dimensional coordinates of the multiple corner points.

4. The method according to claim 3, characterized in that The obtaining of the two-dimensional coordinates of a plurality of corner points of each of the plurality of first checkerboard patterns comprises: Selecting a first checkerboard pattern from a plurality of first checkerboard patterns in the first image as a target checkerboard pattern; Obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern; After obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, setting all pixels in the area where the target checkerboard pattern is located to a single value; A second target checkerboard pattern is selected from the remaining plurality of first checkerboard patterns, and the two-dimensional coordinates of each corner point in the second target checkerboard pattern are obtained.

5. The method according to claim 1, characterized in that: The corner points are intersection points of every four adjacent grids in the first checkerboard pattern or the second checkerboard pattern.

6. The method according to claim 1, characterized in that The number of the first checkerboard patterns included in the checkerboard three-dimensional structure is 4, 5, 7, 11 or 15.

7. A calibration device, characterized in that: include: A detection unit, configured to detect at least one first image to obtain coordinate information of a plurality of corner points, wherein the first image is obtained by photographing a chessboard three-dimensional structure by a camera, and the chessboard three-dimensional structure includes a plurality of first chessboard patterns located on at least two different planes; Detecting a second image to obtain coordinate information of multiple corner points of a second checkerboard pattern; the second image is obtained by the camera shooting the second checkerboard pattern; when shooting the second image, the second checkerboard pattern fills the entire field of view FOV of the lens of the camera; or, the second checkerboard pattern fills the entire second image; An acquisition unit is used to acquire the model parameters of the camera according to the coordinate information of the multiple corner points of the multiple first checkerboard patterns and the coordinate information of the multiple corner points of the second checkerboard patterns.

8. The calibration device according to claim 7, characterized in that: The multiple first chessboard patterns do not overlap each other, and the multiple first chessboard patterns are located in different planes.

9. The calibration device according to claim 7 or 8, characterized in that: The corner point coordinate information includes two-dimensional coordinates and three-dimensional coordinates of the corner point; The detection unit is used to: obtain the two-dimensional coordinates of a plurality of corner points of each of the plurality of first checkerboard patterns; the two-dimensional coordinates are used to represent the pixel position of the corner point on the plane where the first image is located; The three-dimensional coordinates of the multiple corner points are acquired according to the two-dimensional coordinates of the multiple corner points.

10. The calibration device according to claim 9, characterized in that: The detection unit is specifically used for: Selecting a first checkerboard pattern from a plurality of first checkerboard patterns in the first image as a target checkerboard pattern; Obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern; After obtaining the two-dimensional coordinates of each corner point in the target checkerboard pattern, setting all pixels in the area where the target checkerboard pattern is located to a single value; A second target checkerboard pattern is selected from the remaining plurality of first checkerboard patterns, and the two-dimensional coordinates of each corner point in the second target checkerboard pattern are obtained.

11. The calibration device according to claim 7, characterized in that: The corner points are intersection points of every four adjacent grids in the first checkerboard pattern or the second checkerboard pattern.

12. The calibration device according to claim 7, characterized in that: The number of the first checkerboard patterns included in the checkerboard three-dimensional structure is 4, 5, 7, 11 or 15.

13. A calibration device, characterized in that: include: A processor and a transmission interface, wherein the transmission interface is used to receive or send image data; The processor is configured to call program instructions in the memory to execute the method according to any one of claims 1 to 6.

14. A readable storage medium, comprising program instructions, characterized in that: When the program instructions are executed on a processor or a computer, the method according to any one of claims 1 to 6 is executed.

15. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed on a computer or a processor, the computer or the processor is enabled to perform the method according to any one of claims 1 to 6.

Images