Camera calibration method and device, storage medium and electronic equipment

Abstract

The invention discloses a camera calibration method and device, a storage medium and electronic equipment. The method comprises the steps that calibration images of a calibration plate are collected from different angles through a zoom camera at different calibration distances away from the calibration plate with code values corresponding to the calibration distances, and the calibration plate comprises a plurality of patterns; feature points are extracted from the calibration image corresponding to each calibration distance, and the features of the calibration pattern corresponding to at least part of the feature points corresponding to the farthest calibration distance are different from the features of the calibration pattern corresponding to at least part of the feature points corresponding to the nearest calibration distance; and determining an internal reference of the zoom camera corresponding to each calibration distance based on the feature points. According to the invention, the zoom camera can be calibrated.

Description

Technical Field

[0001] The present application belongs to the field of imaging technology, and in particular, relates to a camera calibration method, device, computer-readable storage medium, and electronic device. Background Art

[0002] With the improvement of people's living standards and the development of science and technology, taking pictures has become one of the important contents in people's daily life. For example, people often use electronic devices equipped with cameras to take pictures. At present, the camera needs to be calibrated before use. Summary of the invention

[0003] The embodiments of the present application provide a camera calibration method, device, storage medium and electronic device, which can realize the calibration of a zoom camera.

[0004] In a first aspect, an embodiment of the present application provides a camera calibration method, comprising:

[0005] Using a zoom camera, at different calibration distances from the calibration plate, and with a code value corresponding to each calibration distance, a calibration image of the calibration plate is collected from different angles, wherein the calibration plate includes a plurality of patterns;

[0006] Extracting feature points from the calibration image corresponding to each of the calibration distances, wherein features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance;

[0007] An internal parameter of the zoom camera corresponding to each of the calibration distances is determined based on the feature points.

[0008] In a second aspect, an embodiment of the present application provides a camera calibration device, including:

[0009] A camera calibration device, comprising:

[0010] An image acquisition module, used to acquire calibration images of the calibration plate from different angles at different calibration distances from the calibration plate using a zoom camera and using a code value corresponding to each calibration distance, wherein the calibration plate includes a plurality of patterns;

[0011] A feature point extraction module, used to extract feature points from the calibration images corresponding to each of the calibration distances, wherein the features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from the features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance;

[0012] A parameter determination module is used to determine the internal parameters of the zoom camera corresponding to each of the calibration distances based on the feature points.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the camera calibration method provided by the embodiment of the present application.

[0014] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute the camera calibration method provided in the embodiment of the present application by calling a computer program stored in the memory.

[0015] In an embodiment of the present application, calibration images of the calibration plate are collected from different angles by a zoom camera at different calibration distances from the calibration plate with code values ​​corresponding to each of the calibration distances, and the calibration plate includes multiple patterns; feature points are extracted from the calibration image corresponding to each of the calibration distances, wherein features corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features corresponding to at least some of the feature points corresponding to the closest calibration distance; and internal parameters of the zoom camera corresponding to each of the calibration distances are determined based on the feature points, so that the zoom camera can be calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0017] Figure 1 It is a flowchart of the camera calibration method provided in an embodiment of the present application.

[0018] Figure 2 It is a schematic diagram of a plane black and white chessboard calibration plate used in the related art.

[0019] Figure 3 Schematic diagram of a plane circular calibration plate used in the related art.

[0020] Figure 4 It is a schematic diagram of the design of the calibration plate provided in the embodiment of the present application.

[0021] Figure 5 It is a schematic diagram of the calibration plate provided in the embodiment of the present application.

[0022] Figure 6 It is a structural schematic diagram of a camera calibration device provided in an embodiment of the present application.

[0023] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] It should be noted that the terms "first", "second" and "third" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules that are not listed, or some embodiments also include other steps or modules inherent to these processes, methods, products or devices.

[0025] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] The embodiments of the present application provide a camera calibration method, a camera calibration device, a storage medium, and an electronic device. The execution subject of the camera calibration method may be the camera calibration device provided in the embodiments of the present application, or an electronic device integrating the camera calibration device, wherein the camera calibration device may be implemented in hardware or software. The electronic device may be a device such as a smart phone, a tablet computer, a PDA, a laptop computer, etc. that is equipped with a processor and has a camera calibration capability.

[0027] See also Figure 1 , Figure 1 : is a flowchart of a camera calibration method provided in an embodiment of the present application, and the process may include:

[0028] In 101, a calibration image of the calibration plate is collected from different angles at different calibration distances from the calibration plate by using a zoom camera and a code value corresponding to each calibration distance, wherein the calibration plate includes a plurality of patterns.

[0029] Among them, the zoom camera may include a main camera, a telephoto camera, a wide-angle camera or a TOF camera, etc.

[0030] Camera calibration refers to the process of determining the camera's internal parameters, i.e., intrinsic parameters, and external parameters, i.e., extrinsic parameters. The camera's internal parameters include the camera's focal length, principal point coordinates, distortion parameters, etc., while the external parameters include the camera's rotation matrix and translation vector. The purpose of camera calibration is to convert the pixel coordinates in the image into three-dimensional coordinates in the real world, or to project the three-dimensional coordinates into the pixel coordinates in the image.

[0031] In the related art, when performing camera calibration, a plane black and white checkerboard calibration plate or a plane circular calibration plate is usually used for camera calibration. For example, the plane black and white checkerboard calibration plate used in the related art can be as follows: Figure 2 As shown, the plane circle calibration plate used in the related art can be as follows Figure 3 shown.

[0032] By Figure 3 Taking the plane circle calibration plate shown in FIG. 1 as an example, in the related art, the zoom camera is usually used to collect a certain calibration distance such as Figure 3 The calibration image of the plane circular calibration plate shown is used to determine the internal parameters of the zoom camera based on the calibration image. However, the internal parameters obtained by this calibration method have high accuracy only when used at the calibration distance, and low accuracy when used at other distances.

[0033] Considering the above problems, in this embodiment, a zoom camera is used to collect calibration images of the calibration plate from different angles at different calibration distances from the calibration plate with code values ​​corresponding to each calibration distance, and the calibration plate includes multiple patterns.

[0034] For example, different calibration distances can be set in the following manner.

[0035] Set multiple different calibration distances according to the needs and the characteristics of the zoom camera itself. For example, if the telephoto camera related application needs to shoot long-distance scenes, the farthest calibration distance should be set to the minimum focal length of the telephoto camera, and the closest calibration distance should be set to the maximum focal length of the telephoto camera. The calibration distance here refers to the vertical distance from the camera to the calibration plate.

[0036] It is understandable that Figure 3 Taking the plane circle calibration plate shown as an example, for the plane circle pattern of the same size in the plane circle calibration plate, the pixels it occupies in the calibration image obtained by photographing the plane circle calibration plate at the farthest calibration distance are relatively few, while the pixels it occupies in the calibration image obtained by photographing the plane circle calibration plate at the closest calibration distance are relatively more. Therefore, for the calibration image obtained by photographing the plane circle calibration plate at the farthest calibration distance, the smaller plane circle pattern occupies fewer pixels, resulting in a larger error in the fitting circle points, which in turn leads to a larger error in the calibration result. For the calibration pattern obtained by photographing the plane circle calibration plate at the closest calibration distance, the larger plane circle pattern occupies more pixels, resulting in a relatively small number of feature points that can be extracted.

[0037] Based on this, this embodiment also provides a calibration plate generation solution, that is, Figure 3On the basis of the plane circle calibration plate shown, plane circle patterns of other sizes and / or other colors are added, so that for calibration images taken at different calibration distances, circle patterns with different features, such as different colors, different diameters, different circumferences and / or different areas, can be fitted with circle points.

[0038] For example, a computer program and a printing device that can input relevant parameters can be provided. The calibration plate is designed with a randomly distributed pattern of large and small circles. The input parameters include the parameters of the zoom camera. Among them, the zoom camera is designed with a maximum focal length and a minimum focal length, a pixel size of an image sensor in the zoom camera, a field of view (FOV) of the zoom camera, and a nearest calibration distance and a farthest calibration distance. Through these parameters, the computer program generates the required calibration plate image, and the printing device prints it out to obtain the calibration plate.

[0039] The above input parameters are necessary parameters and can meet general scenarios. In fact, other parameters are required to generate the calibration plate, which can be generated by a computer program or input from outside, that is, optional input parameters, which can provide more constraints for the calibration plate and generate a more suitable calibration plate, including: the maximum and minimum values ​​of the physical size of the calibration plate, the number of rows and columns of feature points, the type of random circle size, the type of plane circle pattern composition, the maximum number of pixels and the minimum number of pixels occupied by the plane circle pattern in the calibration image, and at least one of the maximum calibration tilt angle, etc. When the input parameters are unreasonable and the valid calibration plate image cannot be generated, the computer program will inform the user and provide recommended values ​​or prompt to change other parameters. When the input parameters are not provided, they are replaced by the default values ​​in the computer program.

[0040] The specific calibration board generation scheme can be as follows:

[0041] (1) According to the camera pinhole model and the relationship between similar triangles, the physical size w×h of the calibration plate at the calibration distance can be calculated based on the zoom camera design focal length, zoom camera field of view angle and calibration distance;

[0042] (2) The number of feature points s in the field of view is set by user input or computer program default value. For single-zoom camera calibration, the field of view here refers to the single camera field of view, and for multi-zoom camera calibration, the field of view here refers to the public field of view. For multi-zoom camera calibration, usually, for the same calibration plate, the number of feature points in the field of view of the camera with a longer focal length is smaller than the number of feature points in the field of view of the camera with a shorter focal length. Therefore, s should be set based on the number of feature points in the field of view of the camera with a longer focal length;

[0043] (3) Generate p kinds of random circle sizes according to the number of feature points s, the pixel size of the image sensor, the maximum number of pixels and the minimum number of pixels occupied by the plane circle pattern in the image, and the maximum calibration tilt angle γ, where p ≥ 2. At the nearest calibration distance, check whether the number of feature points s in the field of view of the zoom camera and the maximum number of pixels and the minimum number of pixels occupied by the plane circle pattern in the calibration image meet the optional input parameter requirements or the system default parameters. If not, then preferentially add white circle patterns of different diameters in the center of the larger black circle pattern. If it still does not meet the requirements, then add a black circle with a smaller diameter in the gap between the larger black circle pattern and the smaller black circle pattern. The obtained calibration plate can be as follows Figure 4 When the calibration distance is far, black circles with larger diameters, such as P1, P2, and P7, and concentric circles, such as P3 and P5, are used to extract feature points. When the calibration distance is near, black circles with smaller diameters, such as P6, and white circles, such as P4, are used to extract feature points.

[0044] (4) Verification: Select a relatively small random circular pattern sequence at a certain scale from the generated random circular pattern sequence, which is non-repeatable in the entire random circular pattern sequence, so as to locate the specific position of the feature point sequence in the field of view in the entire calibration plate, match and sort the feature points in the calibration images taken by the zoom camera at different angles, etc. The relatively small random circular pattern sequence should be determined according to the complexity of the entire calibration plate, and can be 3×3, 4×4, 5×5, etc. For example, for the convenience of illustration, Figure 3 Take the calibration plate shown as an example, the 3×3 random circle pattern sequence in the upper corner is [p1, p1, p1], [p2, p1, p1], [p2, p2, p1], as long as this random circle sequence appears in the field of view of the zoom camera, no matter what kind of translation or rotation the zoom camera undergoes, the position of each feature point in the calibration image in the entire calibration plate can be determined, so as to complete the corresponding matching of feature points in images at different angles. The purpose of the verification is: any relatively small random circle pattern sequence in the calibration plate can complete the function of feature point positioning. If there are repeated random circle pattern sequences in the relatively small random circle pattern sequence, or if there are repeated random circle pattern sequences after rotation, the random circle arrangement should be modified until it meets the requirements.

[0045] In an optional embodiment, for Figure 4The calibration plate formed by the calibration plate design shown in the figure should at least meet the following conditions when testing the non-repeatability of relatively small random circular pattern sequences: at a long distance, any relatively small random circular pattern sequence in the two types of black plane circular patterns with larger diameters, such as P1, P7 and concentric plane circular ring patterns P3, P5, is not repeated; at a medium distance, any relatively small random circular pattern sequence in the two types of plane circular patterns with larger diameters, such as P1, P2, P7 and concentric plane circular ring patterns P3, P5, is not repeated; at a close distance, any relatively small random circular pattern sequence in the white plane circular pattern with smaller diameter, such as P4 and a plane circular pattern with an even smaller diameter, such as P6, is not repeated. If any of the conditions is not met, the arrangement of the plane circular patterns should be adjusted until no repeated relatively small random circular pattern sequence appears.

[0046] It should be noted that Figure 4 The design of the calibration plate shown is only an example provided by the embodiment of the present application and is not intended to limit the present application. In practical applications, other calibration plate images can also be generated according to the same principle as the above method.

[0047] For example, the code value corresponding to each calibration distance of the zoom camera can be obtained in the following way.

[0048] The zoom camera faces the calibration plate, moves from near to far according to the set calibration distance and performs autofocus, and records the motor code value of the zoom camera at each calibration distance in turn, then moves from far to near according to the set calibration distance and performs autofocus, and records the motor code value again. The code values ​​of the zoom camera at each calibration distance are averaged as the final code value. Move from near to far again and perform autofocus. If the error between the code value obtained at each calibration distance and the final code value obtained before is within a certain range, the final code value is valid. The motor code value here corresponds to the focal length of the zoom camera one by one. At the bottom of the algorithm, the motor code value is usually used to represent the focal length of the camera.

[0049] For example, calibration images at different angles captured by a zoom camera can be obtained in the following way.

[0050] Place the zoom camera near each calibration distance, and lock the focus of the zoom camera according to the final camera code value obtained by the above process, and then collect the calibration plate image taken by the zoom camera. Change the angle of the zoom camera to collect several sets of image pairs. The changing angle includes translation, rotation, tilt, etc., but the tilt angle should not be greater than the maximum calibration tilt angle γ, and when changing the angle, the distance between the zoom camera and the calibration plate should be ensured to be at the corresponding calibration distance. At the same time, it should be observed whether the image is blurred. If blurred, adjust other angles. In actual application, at least 3 calibration images are required. For example, in a certain application, according to actual needs, only 3 calibration images are collected at three angles, including calibration images collected facing the calibration plate, calibration images collected at a tilt of 20° to the left, and calibration images collected at a tilt of 20° to the right.

[0051] In 102, feature points are extracted from the calibration images corresponding to each calibration distance, wherein features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance.

[0052] The feature points corresponding to the calibration distance are feature points extracted from the calibration image collected at the calibration distance, and the features of the calibration pattern corresponding to the feature points corresponding to the calibration distance are features of the calibration pattern corresponding to the feature points extracted from the calibration image collected at the calibration distance in the calibration plate.

[0053] The features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance may be different from the features of the calibration pattern corresponding to at least some of the feature points corresponding to the nearest calibration distance. For the farthest calibration distance and the nearest calibration distance, plane circle patterns with different features may be selected to extract feature points respectively. For example, for the farthest calibration distance, a plane circle pattern with a relatively large area or diameter may be selected to extract feature points; for the nearest calibration distance, a plane circle pattern with a relatively small area or diameter may be selected to extract features. In this way, feature points can be extracted more accurately from the calibration image collected at the farthest calibration distance and the calibration image collected at the nearest calibration distance.

[0054] For the calibration distances between the farthest calibration distance and the closest calibration distance, corresponding plane circle patterns may also be selected based on actual conditions to respectively extract feature points.

[0055] For example, Figure 4Taking the plane circle calibration plate shown in the figure as an example, when the zoom camera collects calibration images of the plane circle calibration plate at different calibration distances from the plane circle calibration plate, since the plane circle patterns P1, P2, P3, P5 and P7 in the calibration plate have larger areas, they occupy more pixels in the calibration image collected by the zoom camera at a longer calibration distance, and the center of the circle or the center of the ellipse is extracted more accurately, while the plane circle patterns P4 and P6 in the calibration plate have smaller areas, they occupy fewer pixels in the calibration image collected by the zoom camera at a longer calibration distance, and there is a possibility that the center of the circle or the center of the ellipse cannot be detected or even if it is detected, the extracted center of the circle or the center of the ellipse is inaccurate. Therefore, for the calibration images collected by the zoom camera at a longer calibration distance, the plane circle patterns P1, P2, P3, P5 and P7 can be used to extract feature points. For example, for the calibration images collected by the zoom camera at the farthest calibration distance, the plane circle patterns P1, P3, P5 and P7 can be used to extract feature points. Then, for the feature points extracted from the calibration images collected by the zoom camera at the farthest calibration distance, the features of the corresponding calibration patterns are the features of the plane circle patterns P1, P3, P5 and P7, such as the area, circumference and Color, etc.; Since the plane circular patterns P3, P5 and P7 in the calibration plate have a large area, incomplete circular patterns may appear in the calibration image collected by the zoom camera at a relatively central calibration distance. Therefore, the plane circular patterns P1 and P2 can be used to extract feature points. When the number of feature points is small, the plane circular patterns P3, P5 and P7 in the center area of ​​the calibration image can also be used to extract feature points; Since the plane patterns P1, P2, P3, P5 and P7 in the calibration plate have a relatively large area, incomplete circular patterns may appear in the calibration image collected by the zoom camera at a relatively close distance. Therefore, the plane circle patterns P4 and P6 can be preferentially used to extract feature points. When the number of feature points is small, a part of the plane circle patterns P1 and P2 in the center area of ​​the calibration image can also be used to extract feature points. For example, for the calibration image acquired by the zoom camera at the nearest calibration distance, the plane circle patterns P4 and P6 can be used to extract feature points. Then, for the feature points extracted from the calibration image acquired by the zoom camera at the nearest calibration distance, the features of the corresponding calibration patterns are the features of the plane circle patterns P4 and P6, such as the area, diameter, circumference, color, etc. of the plane circle patterns P4 and P6.

[0056] In 103 , an intrinsic parameter of the zoom camera corresponding to each calibration distance is determined based on the feature points.

[0057] For example, after obtaining the feature points in the calibration pattern corresponding to each calibration distance, the intrinsic parameters corresponding to the zoom camera and each calibration distance can be solved using the least squares method and maximum likelihood estimation based on the feature points in the calibration pattern corresponding to each calibration distance and the simultaneous equations of the mathematical model of the pinhole camera.

[0058] It is understandable that after obtaining the feature points in the calibration pattern corresponding to each calibration distance, the internal parameters of the zoom camera corresponding to each calibration distance can also be determined by other internal parameter determination methods provided in the relevant technology, and no specific limitation is made here.

[0059] In this embodiment, calibration images of the calibration plate are collected from different angles by a zoom camera at different calibration distances from the calibration plate with code values ​​corresponding to each calibration distance, and the calibration plate includes multiple patterns; feature points are extracted from the calibration image corresponding to each calibration distance, wherein features corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features corresponding to at least some of the feature points corresponding to the closest calibration distance; and internal parameters of the zoom camera corresponding to each calibration distance are determined based on the feature points, so that the zoom camera can be calibrated.

[0060] In an optional embodiment, a calibration image of the calibration plate is collected from different angles at different calibration distances from the calibration plate by a zoom camera with a code value corresponding to each calibration distance, including:

[0061] Using multiple zoom cameras, at different calibration distances from the calibration plate, to collect calibration images of the calibration plate from different angles with code values ​​corresponding to each calibration distance;

[0062] Extract feature points from the calibration image corresponding to each calibration distance, including:

[0063] Extracting feature points from calibration images corresponding to each calibration distance collected by each zoom camera;

[0064] After extracting feature points from the calibration image corresponding to each calibration distance, it also includes:

[0065] Extract common feature points from feature points corresponding to multiple zoom cameras at the same calibration distance and the same angle;

[0066] Based on the feature points, the internal parameters of the zoom camera corresponding to each calibration distance are determined, including:

[0067] Determine the internal parameters corresponding to each zoom camera and each calibration distance based on the feature points corresponding to each zoom camera;

[0068] After determining the internal parameters of the zoom camera corresponding to each calibration distance based on the feature points, the method further includes:

[0069] Based on the common feature points, external parameters corresponding to each calibration distance of the multiple zoom cameras are determined.

[0070] Taking multiple zoom cameras as the main camera and the telephoto camera as an example, when setting the calibration distance, the relative position of the main camera and the telephoto camera should be fixed. An appropriate calibration distance sequence should be set according to the needs and the characteristics of the multiple zoom cameras themselves.

[0071] For example, if telephoto camera-related applications need to shoot long-distance scenes, the farthest calibration distance should be set at the point where the telephoto camera has the smallest focal length, and the closest calibration distance should be set at the point where the telephoto or main camera has the largest focal length. Of course, there may be a large difference between the maximum calibration distance and the minimum calibration distance, such as greater than 3 meters, which may be difficult to mass produce. Therefore, this calibration distance should be combined with the characteristics of the camera itself. For example, after it is greater than 1.5 meters, the calibration distance gradually increases while the camera focal length changes very little. Within the error allowable range required by the application, there is no need to set the maximum calibration distance beyond 1.5 meters. In addition, the change in camera focal length and the calibration distance are roughly inversely proportional to the relationship between the function, and the spacing changes dramatically at close distances. Therefore, the calibration distance interval should be set to a small interval at close distances and a large interval at long distances.

[0072] For example, the calibration distances of a main camera and telephoto camera combination are: 20cm, 25cm, 30cm, 40cm, 55cm, 75cm, 100cm, 130cm, 180cm, 250cm, a total of 10 calibration distances. The calibration distance here refers to the vertical distance from the camera to the target.

[0073] For example, the code values ​​corresponding to each calibration distance of multiple zoom cameras can be obtained in the following way.

[0074] Multiple zoom cameras face the calibration plate and move from near to far in accordance with the set calibration distance. The motor code values ​​of the zoom cameras at each calibration distance are recorded in turn. Then the cameras move from far to near in accordance with the set calibration distance and the motor code values ​​are recorded again. The code values ​​of each zoom camera at each calibration distance are averaged as the final code value after two moves. The cameras move from near to far again. If the error between the code value obtained by each zoom camera at each calibration distance and the final code value obtained before is within a certain range, the final code value is valid. The motor code value here corresponds to the focal length of the zoom camera one by one. At the bottom of the algorithm, the motor code value is usually used to represent the focal length of the camera.

[0075] For example, calibration images at different angles captured by multiple zoom cameras can be obtained in the following manner.

[0076] Place multiple zoom cameras near each calibration distance, and lock the focus of the multiple zoom cameras according to the final camera code value obtained above, and then use the multiple zoom cameras to collect calibration images of the calibration plate at the same time. The common field of view of multiple zoom cameras should be covered by the calibration plate as much as possible, and change the angles of multiple zoom cameras to collect several sets of image pairs. The changing angle includes translation, rotation, tilt, etc., but the tilt angle should not be greater than the maximum calibration tilt angle γ, and when changing the angle, it should be ensured that the distance between the multiple zoom cameras and the calibration plate is at the corresponding calibration distance. At the same time, it should be observed whether the calibration image taken is blurred. If blurred, adjust other angles. In actual application, at least 3 groups are required. For example, in a certain application, according to actual needs, only 3 groups of calibration images are collected, including one group of calibration images collected facing the calibration plate, one group of calibration images collected at a 20° tilt to the left, and one group of calibration images collected at a 20° tilt to the right.

[0077] In an optional embodiment, if the zoom camera has an OIS optical image stabilization function, then when the zoom camera shakes when shooting the calibration plate, the optical axis will shift, which will affect the coordinates of the principal point in the zoom camera's internal parameters. Therefore, a better solution is to keep multiple zoom cameras stationary and change the angle of the calibration plate to complete the calibration image acquisition. Usually, at a longer calibration distance, it is usually difficult to change the angle of the calibration plate. Therefore, in most cases, the solution of changing the zoom camera angle is adopted. At this time, a delay program needs to be set. The zoom camera shoots the calibration image after a short period of time after changing the angle. The specific delay time should be related to the zoom camera system.

[0078] It is understandable that the above is only to illustrate the calibration method of the multi-zoom camera provided in the embodiment of the present application by taking the main camera and the telephoto camera as examples, and is not intended to limit the present application. In practical applications, the calibration method of the multi-zoom camera provided in the embodiment of the present application can also be used to calibrate the main camera and the wide-angle camera, and can also be used to calibrate the main camera and the TOF camera, and so on.

[0079] In an optional embodiment, extracting feature points from a calibration image corresponding to each calibration distance acquired by each zoom camera includes:

[0080] determining characteristics of the calibration pattern, and determining a reference calibration pattern sequence from the calibration pattern;

[0081] Determine the center point and features of the fitting pattern in the calibration image corresponding to each calibration distance collected by each zoom camera, and use the features of the fitting pattern as the features corresponding to the center point of the fitting pattern;

[0082] Determining, from the center points, a sequence of reference center points whose corresponding features match features of a reference calibration pattern in the sequence of reference calibration patterns;

[0083] Determine, from the neighborhood center points in the neighborhood of the reference center point sequence, a matching center point where the corresponding feature matches the feature of the corresponding neighborhood calibration pattern in the neighborhood of the reference calibration pattern sequence;

[0084] Determine the target center point sequence according to the reference center point sequence and the matching center point;

[0085] The center point in the target center point sequence is determined as a feature point in the calibration image corresponding to each calibration distance collected by each zoom camera.

[0086] For example, if the calibration plate is Figure 5 As shown, it can be determined that the calibration plate includes 7 types of planar circular patterns, namely P1 to P7, and different planar circular patterns have different characteristics, such as different areas, different diameters, different colors, etc. Assuming that the reference calibration pattern sequence is a planar circular pattern sequence in area A, the reference calibration pattern sequence can be determined to be [P1, P1, P7, P7], [P6, P6, P6, P6], [P2(P3), P2(P3), P1, P1], [P6, P6, P6, P6], and by detecting and fitting the fitting patterns in the calibration image corresponding to a certain calibration distance, such as circular patterns and elliptical patterns, the center point and features of the fitting patterns in the calibration image corresponding to the calibration distance, such as area and color, can be obtained; based on the features of the fitting patterns in the calibration image corresponding to the calibration distance and the features of the calibration patterns in the calibration plate, the calibration patterns matched by the fitting patterns, i.e., the planar circular patterns, can be determined, and then, the features of the fitting patterns are used as the features corresponding to the center points of the fitting patterns, and a reference center point sequence whose corresponding features match the features of the reference calibration patterns in the reference calibration pattern sequence can be determined from the center points. Next, a matching center point or a matching center point sequence in which corresponding features match features of a corresponding neighborhood calibration pattern in the neighborhood of the reference calibration pattern sequence may be determined from the neighborhood center points in the neighborhood of the reference center point sequence.

[0087] For example, for the neighborhood calibration pattern sequence [P4(P5), P4(P5), P1] on the left side of the reference calibration pattern P1 in the first row and first column in the reference calibration pattern sequence, if the features of the neighborhood center points at the corresponding positions in the neighborhood center point sequence on the left side of the reference center point in the first row and first column in the reference center point sequence match the neighborhood calibration patterns at the corresponding positions in the neighborhood calibration pattern sequence, then the neighborhood center point sequence can be determined as a matching center point sequence. In other words, if the features corresponding to the neighborhood center point C1 closest to the reference center point on the left side of the reference center point in the first row and first column in the reference center point sequence match the features of the neighborhood calibration pattern P1, the features corresponding to the neighborhood center point C2 on the left side of the neighborhood center point C1 match the features of the neighborhood calibration pattern P4(P5), and the features corresponding to the neighborhood center point C3 on the left side of the neighborhood center point C2 match the features of the neighborhood calibration pattern P4(P5), then the neighborhood center point sequence [C3, C2, C1] can be determined as a matching center point sequence.

[0088] In practical applications, each calibration pattern in the calibration plate will have a unique number, such as the calibration pattern from the first row and first column in the calibration plate to the calibration pattern in the last row and last column in the calibration plate, which are numbered 1, 2, 3, ... N. The value of N is the total number of calibration patterns in the calibration plate. After the neighborhood center point sequence and the matching center point sequence are obtained, the number of the calibration pattern whose feature matches the center point in the reference center point sequence can be used as the number of the center point in the reference center point sequence. Similarly, the number of the calibration pattern whose feature matches the center point in the matching center point sequence can be used as the number of the center point in the matching center point sequence. Then, the center points in the reference center point sequence and the center points in the matching center point sequence are sorted in order from large to small or from small to large, and the sorted center point sequence is determined as the target center point sequence.

[0089] It is understandable that in some cases, due to reasons such as depth of field or distance, not every circle or ellipse in the calibration image can be effectively extracted, that is, isolated points will appear. After extending from the reference center point sequence to the isolated point, it is found that the isolated point does not match the neighborhood calibration pattern of the corresponding position. In this case, the corresponding reference center point sequence and the matching center point can be continued to be matched based on other reference calibration pattern sequences, such as reference calibration pattern sequences in other areas of the calibration plate, until all reference calibration pattern sequences are used or the number of effectively sorted feature points accounts for a certain threshold of the total number of detected feature points. Then, the search is stopped. A small number of isolated points that are not used do not affect the overall effect of the algorithm.

[0090] In an optional embodiment, for different calibration distances, the selected feature points may be feature points corresponding to calibration patterns corresponding to different calibration distances. For example, for calibration images acquired by a zoom camera at a longer calibration distance, the plane circular patterns P1, P2, P3, P5 and P7 may be preferentially used to extract feature points. Then, after all reference center point sequences and matching center points are obtained, the center point of the feature matching of the corresponding feature with any of the plane circular patterns P1, P2, P3, P5 and P7 may be used as the feature point in the calibration image.

[0091] In an optional embodiment, after all corresponding feature point sequences are extracted from a set of calibration images taken by a zoom camera, common point correspondence can be performed, that is, a common feature point sequence is found for the feature point sequences corresponding to the calibration images at different angles at the same calibration distance, and the feature points in the common feature point sequence are used as valid feature points corresponding to the calibration images at different angles at the same calibration distance collected by the zoom camera. Based on these valid feature points, the internal parameters of the zoom camera are completed, including the calibration of the focal length, principal point and distortion coefficient at the same calibration distance.

[0092] It should be noted that Figure 5 This is only an example of the calibration plate provided in the embodiment of the present application and is not intended to limit the present application. For example, the number and characteristics of the planar circular patterns in the calibration plate provided in the embodiment of the present application are only used to assist in explaining the camera calibration method provided in the embodiment of the present application and are not intended to limit the present application.

[0093] The above is a schematic description of a reference calibration pattern sequence including all types of plane circles and plane circular ring patterns, such as P1 to P7. In an optional embodiment, the types of calibration patterns included in the reference calibration pattern sequence corresponding to different calibration distances may also be different. For example, at a long distance, the reference calibration pattern sequence may include only two types of black plane circular patterns with larger diameters, such as P1, P7 and concentric plane circular ring patterns P3, P5, that is, the reference calibration pattern sequence does not include a white plane circular pattern P2 with a larger diameter and a plane circular pattern with a smaller diameter, such as P4 and P6. At a medium distance, the reference calibration pattern sequence may include only two types of plane circular patterns with larger diameters, such as P1, P2, P7 and concentric plane circular ring patterns P3, P5, that is, it does not include plane circular patterns with smaller diameters, such as P4 and P6. At a close distance, the reference calibration pattern sequence may include only plane circular patterns with smaller diameters, such as P4 and P6, that is, it does not include two types of plane circular patterns with larger diameters, such as P1, P2, P7 and concentric plane circular ring patterns P3, P5.

[0094] In an optional embodiment, extracting common feature points from feature points corresponding to multiple zoom cameras at the same calibration distance and the same angle includes:

[0095] Determine the same center point sequence from target center point sequences corresponding to calibration images at the same calibration distance and the same angle corresponding to multiple zoom cameras;

[0096] The center points in the same center point sequence are determined as common feature points corresponding to multiple zoom cameras at the same calibration distance and the same angle.

[0097] It should be noted that, taking multiple zoom cameras including a main camera and a telephoto camera as an example, both zoom cameras need to complete the calibration of a single zoom camera first, and then perform calibration between the two zoom cameras, that is, determine the external parameters corresponding to the two zoom cameras and each calibration distance. The calibration between the two zoom cameras requires retrieving the valid feature points in the common field of view of the two zoom cameras. That is, the calibration images taken by the two zoom cameras at the same time at a certain calibration distance and a certain angle, after obtaining the corresponding target feature point sequences, the common sequence is retrieved, that is, the same feature point sequence in the corresponding target feature point sequences. The feature points in the common sequence are the valid feature points in the common field of view of the two zoom cameras at the calibration distance, that is, the common feature points. Repeat the above operation for the calibration images taken at all angles at the calibration distance to obtain the common feature points of the two zoom cameras at different angles at the calibration distance; repeat the above operation for the calibration images taken at all angles at each calibration distance to obtain the common feature points of the two zoom cameras at different angles at each calibration distance; based on the obtained common feature points, the external parameters corresponding to multiple zoom cameras and each calibration distance can be determined.

[0098] For example, based on the solved intrinsic parameters and the actual physical distance between the common feature points and the plane circle patterns corresponding to the common feature points on the calibration plate (for example, the distance between any two adjacent plane circle patterns is 1 cm), the system of equations can be combined again, and the least squares method and maximum likelihood estimation can be used to solve the external parameters corresponding to the two cameras and each calibration distance, including the rotation matrix and the translation vector.

[0099] It is understandable that the external parameters corresponding to the multiple zoom cameras and each calibration distance may also be determined by other external parameter determination methods provided in the relevant technology, and no specific limitation is made here.

[0100] In an optional embodiment, the calibration result may also be verified, including verification during the calibration process and verification after the calibration is completed.

[0101] Among them, the verification in the calibration process can be a. the mean, variance and extreme value of the projection error of the feature points at different angles of the single zoom camera, and the mean, variance and extreme value of the projection error of the feature points in the common field of view for the multi-zoom camera calibration. It can also be b. At a certain calibration distance, select a part of the multiple sets of calibration images collected at several angles, repeat the calibration of the single zoom camera and the multi-zoom camera, and detect the difference between the calibration result and the calibration result using all the calibration images. If the difference is too large, it means that some calibration images have poor shooting quality, which will affect the overall calibration result. This test can be repeated multiple times, and a different combination of calibration images is selected each time. The test after the calibration is completed is to select one or more distances after setting the calibration distance, shoot calibration images, and solve the projection accuracy of the dual cameras and the error between the physical distance obtained by binocular ranging and the real physical distance obtained by other higher-precision measurement equipment.

[0102] In an optional embodiment, after determining the external parameters corresponding to each calibration distance of the plurality of zoom cameras based on the common feature points, the method further includes:

[0103] Get the current code values ​​of multiple zoom cameras;

[0104] Determine a matching code value that matches the current code value from the code values ​​corresponding to each calibration distance of the plurality of zoom cameras;

[0105] Determine the calibration distance corresponding to the matching code value as the matching calibration distance;

[0106] Determine the internal parameter corresponding to each zoom camera and the matching calibration distance as the first current internal parameter corresponding to each zoom camera;

[0107] Determine the external parameters corresponding to the multiple zoom cameras and the matching calibration distance as first current external parameters corresponding to the multiple zoom cameras;

[0108] Based on the first current internal parameter and the first current external parameter, multiple zoom cameras are controlled to shoot a shooting scene to obtain a first target image.

[0109] For example, after obtaining the code value at each calibration distance and the corresponding internal and external parameters of multiple zoom cameras, the current code value of each zoom camera can be obtained in actual use, and the code value recorded at the calibration distance can be retrieved, and the code value closest to the current code value is selected as the code value matching the current code value, and the corresponding internal parameter is used as the first current internal parameter corresponding to each zoom camera, and the corresponding external parameter is used as the first current external parameter corresponding to each zoom camera, and then based on the first current internal parameter and the first current external parameter, the multiple zoom cameras are controlled to shoot the shooting scene to obtain the first target image.

[0110] For example, assuming that multiple zoom cameras are respectively a main camera and a telephoto camera, at a certain calibration distance, the code value corresponding to the main camera is closest to the current code value of the main camera, and the code value corresponding to the telephoto camera is closest to the current code value of the telephoto camera, then the calibration distance can be determined to be the target calibration distance, then, the internal parameter corresponding to the main camera and the target calibration distance can be determined as the first current internal parameter corresponding to the main camera, the internal parameter corresponding to the telephoto camera and the target calibration distance can be determined as the first current internal parameter corresponding to the telephoto camera, and the external parameters corresponding to the main camera and the telephoto camera and the target calibration distance can be determined as the first current external parameters corresponding to the main camera and the telephoto camera.

[0111] Among them, after starting a shooting application (such as the system application "camera" of an electronic device) according to user operation, the scene that the camera is aimed at is the shooting scene. For example, after the user clicks the icon of the "camera" application on the electronic device to start the "camera application", if the user uses the camera of the electronic device to aim at a scene including XX object, then the scene including XX object is the shooting scene. According to the above description, those skilled in the art should understand that the shooting scene does not refer to a specific scene, but the scene that is aimed at in real time following the direction of the camera.

[0112] In an optional embodiment, after obtaining the current code values ​​of the multiple zoom cameras, the method further includes:

[0113] Obtain a first fitting function and a second fitting function, wherein the first fitting function is obtained by fitting the code value and the internal parameter corresponding to each zoom camera and each calibration distance, and the second fitting function is obtained by fitting the coed value and the external parameter corresponding to multiple zoom cameras and each calibration distance;

[0114] According to the current code value and the first fitting function, a second current intrinsic parameter corresponding to each zoom camera is obtained, and according to the current code value and the second fitting function, a second current extrinsic parameter corresponding to the plurality of zoom cameras is obtained;

[0115] Based on the second current internal parameter and the second current external parameter, the multiple zoom cameras are controlled to shoot the shooting scene to obtain a second target image.

[0116] For example, the code values ​​at different distances may be fitted with calibration parameters to generate a fitting function. The current code value obtained in actual use may be used to calculate the internal and external parameters according to the fitting function. Based on the calculated internal and external parameters, multiple zoom cameras may be controlled to shoot the scene to obtain a second target image.

[0117] For example, assuming that the multiple zoom cameras are respectively a main camera and a telephoto camera, the code value and the internal parameter corresponding to each calibration distance of the main camera can be pre-fitted to obtain a first fitting function corresponding to the main camera, the code value and the internal parameter corresponding to each calibration distance of the telephoto camera can be pre-fitted to obtain a first fitting function corresponding to the telephoto camera, and the code value and the external parameter corresponding to each calibration distance of the main camera and the telephoto camera can be pre-fitted to obtain a second fitting function corresponding to the main camera and the telephoto camera. After the current code value of the main camera and the current code value of the telephoto camera are obtained, the current code value of the main camera can be input into the first fitting function corresponding to the main camera to obtain a second current internal parameter corresponding to the main camera, the current code value of the telephoto camera can be input into the first fitting function corresponding to the telephoto camera to obtain a second current internal parameter corresponding to the telephoto camera, and the current code value of the main camera and the current code value of the telephoto camera can be input into the second fitting function corresponding to the main camera and the telephoto camera to obtain a second current external parameter corresponding to the main camera and the telephoto camera.

[0118] In an optional embodiment, in some video applications, intrinsic and extrinsic parameters that may be needed are usually calculated in advance during initialization, so only a limited number of discrete intrinsic and extrinsic parameters can be used.

[0119] In an optional embodiment, determining the internal parameters of the zoom camera corresponding to each calibration distance based on the feature points includes:

[0120] Determining a target calibration distance and a non-target calibration distance from the calibration distance;

[0121] Determining internal parameters of the zoom camera corresponding to the target calibration distance based on feature points corresponding to the target calibration distance, wherein the internal parameters corresponding to the target calibration distance include a focal length, a principal point coordinate, and a distortion coefficient corresponding to the target calibration distance;

[0122] The focal length and distortion coefficient of the zoom camera corresponding to the non-target calibration distance are determined based on the feature points corresponding to the non-target calibration distance, and the principal point coordinates corresponding to the target calibration distance are used as the principal point coordinates corresponding to the non-target calibration distance.

[0123] Due to some factors: when the camera motor changes the motor position for focusing, the position of the motor may not be exactly the same at the same distance each time, and it cannot be guaranteed that the lens optical axis remains unchanged every time the motor moves. Most modern cameras include OIS function. Therefore, whether it is during calibration or actual use, the real-time principal point coordinates of the camera are actually constantly changing, that is, when the code value is the same, the principal point coordinates and the corresponding internal parameters are actually different. Moreover, whether it is during calibration or actual use, the change law of the principal point coordinates has a certain randomness, and there is no good mathematical model fitting. The focal length changes relatively regularly, the focus distance changes from near to far, and the focal length changes from large to small, and can be fitted with an inverse proportional function relationship. Therefore, in other applications, the principal point of each zoom camera and the external parameters between multiple zoom cameras use a set of fixed values, and only the focal length value is changed, that is, the focal length corresponding to the calibration distance is selected only according to the real-time code value, and the principal point coordinates and the external parameters between the two cameras are fixed to use the parameters of a certain calibration distance. In these applications, the law of focal length variation with code value of one or several zoom cameras can be applied to a batch of cameras of the same model in the same batch. To apply it more accurately to a single zoom camera, shoot the calibration plate at the corresponding code value to ensure clear shooting, use this law to project the extracted feature points, and use the relationship between the projection error and the true value to make certain corrections to the law.

[0124] There are also some applications, such as zoom cameras with anti-shake function. When solving the internal parameters of a single zoom camera, a calibration distance is directly selected from multiple calibration distances as the target calibration distance, and the remaining calibration distances are determined as non-target calibration distances. The principal point coordinates of the target calibration distance are selected, and when the least squares method and maximum likelihood estimation are used to solve the internal parameters at the non-target calibration distance, the principal point coordinates are artificially fixed and only the focal length and / or distortion coefficient are optimized. In this way, when the internal parameters corresponding to different calibration distances are obtained, the principal point coordinates are unchanged, and only the focal length and / or distortion coefficient change.

[0125] The target calibration distance may be any calibration distance among a plurality of calibration distances.

[0126] In an optional embodiment, professionals may also select a suitable calibration distance to calibrate the coordinates of the principal point, so that the principal point calibration is sufficiently accurate.

[0127] In an optional embodiment, the rotation matrix and translation vector between the multi-zoom cameras also apply fixed values ​​at a certain calibration distance, and only the focal length that changes with the code value is changed at different distances.

[0128] In an optional embodiment, in some applications such as map positioning, ranging, path planning, and three-dimensional reconstruction using a combination of an RGB camera and a TOF camera, more precise internal and external parameters may be required. In this case, the principal point coordinates can be updated and corrected in real time by combining the information such as the angular velocity and displacement in the three-dimensional direction provided by the IMU.

[0129] See also Figure 6 , Figure 6 The camera calibration device 200 includes an image acquisition module 201 , a feature point extraction module 202 and a parameter determination module 203 .

[0130] The image acquisition module 201 is used to acquire calibration images of the calibration plate from different angles at different calibration distances from the calibration plate using a zoom camera and with code values ​​corresponding to each calibration distance, wherein the calibration plate includes a plurality of patterns.

[0131] The feature point extraction module 202 is used to extract feature points from the calibration images corresponding to each calibration distance, wherein the features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from the features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance.

[0132] The parameter determination module 203 is used to determine the internal parameters of the zoom camera corresponding to each of the calibration distances based on the feature points.

[0133] In an optional embodiment, the image acquisition module 201 may be used to: acquire calibration images of the calibration plate from different angles at different calibration distances from the calibration plate using a plurality of zoom cameras and with code values ​​corresponding to each calibration distance;

[0134] The feature point extraction module 202 may be used to: extract feature points from the calibration images corresponding to each calibration distance collected by each zoom camera; extract common feature points from the feature points corresponding to multiple zoom cameras at the same calibration distance and the same angle;

[0135] The parameter determination module 203 may be used to: determine the internal parameters corresponding to each zoom camera and each calibration distance based on the feature points corresponding to each zoom camera; and determine the external parameters corresponding to multiple zoom cameras and each calibration distance based on the common feature points.

[0136] In an optional embodiment, the feature point extraction module 202 can be used to: determine the features of the calibration pattern, and determine a reference calibration pattern sequence from the calibration pattern; determine the center point and features of the fitting pattern in the calibration image corresponding to each of the calibration distances acquired by each of the zoom cameras, and use the features of the fitting pattern as the features corresponding to the center point of the fitting pattern; determine a reference center point sequence from the center point whose corresponding features match the features of the reference calibration pattern in the reference calibration pattern sequence; determine a matching center point from the neighborhood center points in the neighborhood of the reference center point sequence whose corresponding features match the features of the corresponding neighborhood calibration pattern in the neighborhood of the reference calibration pattern sequence; determine a target center point sequence based on the reference center point sequence and the matching center point; and determine the center point in the target center point sequence as the feature point in the calibration image corresponding to each of the calibration distances acquired by each of the zoom cameras.

[0137] In an optional embodiment, the feature point extraction module 202 can be used to: determine the same center point sequence from the target center point sequences corresponding to the calibration images of the multiple zoom cameras at the same calibration distance and the same angle; determine the center point in the same center point sequence as the common feature point of the multiple zoom cameras at the same calibration distance and the same angle.

[0138] In an optional embodiment, the camera calibration device 200 may further include a shooting module, which may be used to: obtain current code values ​​of the multiple zoom cameras; determine a matching code value that matches the current code value from the code values ​​corresponding to the multiple zoom cameras and each calibration distance; determine the calibration distance corresponding to the matching code value as the matching calibration distance; determine the internal parameter corresponding to each zoom camera and the matching calibration distance as the first current internal parameter corresponding to each zoom camera; determine the external parameter corresponding to the multiple zoom cameras and the matching calibration distance as the first current external parameter corresponding to the multiple zoom cameras; and control the multiple zoom cameras to shoot a shooting scene based on the first current internal parameter and the first current external parameter to obtain a first target image.

[0139] In an optional embodiment, the shooting module can be used to: obtain a first fitting function and a second fitting function, wherein the first fitting function is obtained by fitting the code value and the internal parameter corresponding to each zoom camera and each calibration distance, and the second fitting function is obtained by fitting the coed values ​​and the external parameter corresponding to multiple zoom cameras and each calibration distance; obtain a second current internal parameter corresponding to each zoom camera according to the current code value and the first fitting function, and obtain a second current external parameter corresponding to multiple zoom cameras according to the current code value and the second fitting function; and control the multiple zoom cameras to shoot the shooting scene based on the second current internal parameter and the second current external parameter to obtain a second target image.

[0140] In an optional embodiment, the parameter determination module 203 can be used to: determine a target calibration distance and a non-target calibration distance from the calibration distance; determine the internal parameters of the zoom camera corresponding to the target calibration distance based on the feature points corresponding to the target calibration distance, and the internal parameters corresponding to the target calibration distance include the focal length and principal point coordinates corresponding to the target calibration distance; determine the focal length of the zoom camera corresponding to the non-target calibration distance based on the feature points corresponding to the non-target calibration distance, and use the principal point coordinates corresponding to the target calibration distance as the principal point coordinates corresponding to the non-target calibration distance.

[0141] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the camera calibration method provided in the embodiment.

[0142] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the camera calibration method provided in the present embodiment by calling a computer program stored in the memory.

[0143] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0144] The electronic device 300 may include components such as a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the electronic device 300 may also include a screen.

[0145] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 302 and calling data stored in the memory 302, thereby monitoring the electronic device as a whole.

[0146] The memory 302 may be used to store applications and data. The applications stored in the memory 302 include executable codes. The applications may be composed of various functional modules. The processor 301 executes various functional applications and data processing by running the applications stored in the memory 302.

[0147] In this embodiment, the processor 301 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby achieving:

[0148] Using a zoom camera, at different calibration distances from the calibration plate, and with a code value corresponding to each calibration distance, a calibration image of the calibration plate is collected from different angles, wherein the calibration plate includes a plurality of patterns;

[0149] Extracting feature points from the calibration image corresponding to each of the calibration distances, wherein features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance;

[0150] An internal parameter of the zoom camera corresponding to each of the calibration distances is determined based on the feature points.

[0151] In an optional embodiment, when the processor 301 executes the step of acquiring the calibration images of the calibration plate from different angles at different calibration distances from the calibration plate by using a zoom camera with a code value corresponding to each calibration distance, the following steps may be performed: acquiring the calibration images of the calibration plate from different angles at different calibration distances from the calibration plate by using multiple zoom cameras with a code value corresponding to each calibration distance; when the processor 301 executes the step of extracting feature points from the calibration images corresponding to each calibration distance, the following steps may be performed: extracting feature points from the calibration images corresponding to each calibration distance acquired by each zoom camera; and After extracting feature points from the calibration image corresponding to the zoom camera, the processor 301 may further execute: extracting common feature points from feature points corresponding to the multiple zoom cameras at the same calibration distance and the same angle; when the processor 301 executes the step of determining the internal parameters corresponding to the zoom camera and each calibration distance based on the feature points, the processor 301 may execute: determining the internal parameters corresponding to each zoom camera and each calibration distance based on the feature points corresponding to each zoom camera; after the processor 301 executes the step of determining the internal parameters corresponding to the zoom camera and each calibration distance based on the feature points, the processor 301 may further execute: determining the external parameters corresponding to the multiple zoom cameras and each calibration distance based on the common feature points.

[0152] In an optional embodiment, when the processor 301 executes the step of extracting feature points from the calibration images corresponding to each calibration distance collected by each zoom camera, the following steps may be performed: determining features of the calibration pattern, and determining a reference calibration pattern sequence from the calibration pattern; determining a center point and features of a fitting pattern in the calibration images corresponding to each calibration distance collected by each zoom camera, and using the features of the fitting pattern as features corresponding to the center point of the fitting pattern; determining a reference center point sequence from the center points whose corresponding features match features of the reference calibration pattern in the reference calibration pattern sequence; determining matching center points from neighborhood center points in a neighborhood of the reference center point sequence whose corresponding features match features of the corresponding neighborhood calibration pattern in a neighborhood of the reference calibration pattern sequence; determining a target center point sequence based on the reference center point sequence and the matching center points; and determining the center point in the target center point sequence as a feature point in the calibration image corresponding to each calibration distance collected by each zoom camera.

[0153] In an optional embodiment, when the processor 301 executes the extraction of common feature points from feature points corresponding to multiple zoom cameras at the same calibration distance and the same angle, it can execute: determining the same center point sequence from target center point sequences corresponding to calibration images corresponding to multiple zoom cameras at the same calibration distance and the same angle; and determining the center point in the same center point sequence as the common feature point corresponding to the multiple zoom cameras at the same calibration distance and the same angle.

[0154] In an optional embodiment, after the processor 301 executes the determining of the external parameters corresponding to the multiple zoom cameras and each calibration distance based on the common feature point, it may also execute: obtaining current code values ​​of the multiple zoom cameras; determining a matching code value that matches the current code value from the code values ​​corresponding to the multiple zoom cameras and each calibration distance; determining the calibration distance corresponding to the matching code value as the matching calibration distance; determining the internal parameter corresponding to each zoom camera and the matching calibration distance as the first current internal parameter corresponding to each zoom camera; determining the external parameter corresponding to the multiple zoom cameras and the matching calibration distance as the first current external parameter corresponding to the multiple zoom cameras; and controlling the multiple zoom cameras to shoot the shooting scene based on the first current internal parameter and the first current external parameter to obtain a first target image.

[0155] In an optional embodiment, after the processor 301 executes the step of obtaining the current code values ​​of the multiple zoom cameras, it may further execute: obtaining a first fitting function and a second fitting function, wherein the first fitting function is obtained by fitting the code value and the internal parameter corresponding to each zoom camera and each calibration distance, and the second fitting function is obtained by fitting the coed value and the external parameter corresponding to the multiple zoom cameras and each calibration distance; obtaining a second current internal parameter corresponding to each zoom camera according to the current code value and the first fitting function, and obtaining a second current external parameter corresponding to the multiple zoom cameras according to the current code value and the second fitting function; and controlling the multiple zoom cameras to shoot a shooting scene based on the second current internal parameter and the second current external parameter to obtain a second target image.

[0156] In an optional embodiment, when the processor 301 executes the determining of the internal parameters corresponding to the zoom camera and each of the calibration distances based on the feature points, the processor 301 may execute: determining a target calibration distance and a non-target calibration distance from the calibration distances; determining the internal parameters corresponding to the zoom camera and the target calibration distances based on the feature points corresponding to the target calibration distances, the internal parameters corresponding to the target calibration distances including the focal length and the principal point coordinates corresponding to the target calibration distances; determining the focal length corresponding to the zoom camera and the non-target calibration distances based on the feature points corresponding to the non-target calibration distances, and using the principal point coordinates corresponding to the target calibration distances as the principal point coordinates corresponding to the non-target calibration distances.

[0157] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the camera calibration method above, and will not be repeated here.

[0158] The camera calibration device provided in the embodiment of the present application belongs to the same concept as the camera calibration method in the above embodiment. Any method provided in the camera calibration method embodiment can be run on the camera calibration device. The specific implementation process is detailed in the camera calibration method embodiment, which will not be repeated here.

[0159] It should be noted that, for the camera calibration method of the embodiment of the present application, a person skilled in the art can understand that all or part of the process of implementing the camera calibration method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the camera calibration method. Among them, the computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0160] It is understandable that in the specific implementation of this application, user information is involved, such as application usage behavior data, logs and other related data. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] For the camera calibration device of the embodiment of the present application, its various functional modules can be integrated into one processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0162] The camera calibration method, device, storage medium and electronic device provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A camera calibration method, characterized in that: include: Using a zoom camera, at different calibration distances from the calibration plate, and with a code value corresponding to each calibration distance, a calibration image of the calibration plate is collected from different angles, wherein the calibration plate includes a plurality of patterns; Extracting feature points from the calibration image corresponding to each of the calibration distances, wherein features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance; An internal parameter of the zoom camera corresponding to each of the calibration distances is determined based on the feature points.

2. The camera calibration method according to claim 1, characterized in that: The method of collecting calibration images of the calibration plate from different angles at different calibration distances from the calibration plate by using a zoom camera and using a code value corresponding to each calibration distance includes: Using multiple zoom cameras to collect calibration images of the calibration plate from different angles at different calibration distances from the calibration plate with code values ​​corresponding to each calibration distance; The extracting feature points from the calibration image corresponding to each calibration distance includes: Extracting feature points from the calibration images corresponding to each of the calibration distances collected by each of the zoom cameras; After extracting feature points from the calibration images corresponding to each calibration distance, the method further includes: Extracting common feature points from feature points corresponding to the plurality of zoom cameras at the same calibration distance and the same angle; The determining, based on the feature points, an internal parameter of the zoom camera corresponding to each of the calibration distances comprises: Determine an internal parameter corresponding to each of the zoom cameras and each of the calibration distances based on the feature points corresponding to each of the zoom cameras; After determining the internal parameters of the zoom camera corresponding to each of the calibration distances based on the feature points, the method further includes: Based on the common feature points, external parameters corresponding to the plurality of zoom cameras and each of the calibration distances are determined.

3. The camera calibration method according to claim 2, characterized in that: The extracting feature points from the calibration images corresponding to each calibration distance collected by each zoom camera comprises: determining characteristics of the calibration pattern, and determining a reference calibration pattern sequence from the calibration pattern; Determine the center point and features of the fitting pattern in the calibration image corresponding to each calibration distance acquired by each zoom camera, and use the features of the fitting pattern as the features corresponding to the center point of the fitting pattern; determining, from the center points, a sequence of reference center points whose corresponding features match features of reference calibration patterns in the sequence of reference calibration patterns; Determining, from the neighborhood center points in the neighborhood of the reference center point sequence, a matching center point where a corresponding feature matches a feature of a corresponding neighborhood calibration pattern in the neighborhood of the reference calibration pattern sequence; Determine a target center point sequence according to the reference center point sequence and the matching center point; The center point in the target center point sequence is determined as a feature point in the calibration image corresponding to each calibration distance collected by each zoom camera.

4. The camera calibration method according to claim 3, characterized in that: The extracting common feature points from the feature points corresponding to the plurality of zoom cameras at the same calibration distance and the same angle comprises: Determine the same center point sequence from target center point sequences corresponding to calibration images at the same calibration distance and the same angle corresponding to the multiple zoom cameras; The center point in the same center point sequence is determined as a common feature point corresponding to a plurality of zoom cameras at the same calibration distance and the same angle.

5. The camera calibration method according to any one of claims 2 to 4, characterized in that: After determining the external parameters corresponding to the plurality of zoom cameras and each of the calibration distances based on the common feature points, the method further includes: Obtaining current code values ​​of the multiple zoom cameras; Determine a matching code value that matches the current code value from code values ​​corresponding to each of the calibration distances of a plurality of zoom cameras; Determine the calibration distance corresponding to the matching code value as the matching calibration distance; Determine an internal parameter corresponding to each of the zoom cameras and the matching calibration distance as a first current internal parameter corresponding to each of the zoom cameras; Determine the external parameters corresponding to the plurality of zoom cameras and the matching calibration distance as first current external parameters corresponding to the plurality of zoom cameras; Based on the first current internal parameter and the first current external parameter, the plurality of zoom cameras are controlled to shoot a shooting scene to obtain a first target image.

6. The camera calibration method according to claim 5, characterized in that: After obtaining the current code values ​​of the plurality of zoom cameras, the method further includes: Obtaining a first fitting function and a second fitting function, wherein the first fitting function is obtained by fitting the code value and the internal parameter corresponding to each zoom camera and each calibration distance, and the second fitting function is obtained by fitting the coed value and the external parameter corresponding to multiple zoom cameras and each calibration distance; Obtaining a second current intrinsic parameter corresponding to each of the zoom cameras according to the current code value and the first fitting function, and obtaining a second current extrinsic parameter corresponding to a plurality of the zoom cameras according to the current code value and the second fitting function; Based on the second current internal parameter and the second current external parameter, the plurality of zoom cameras are controlled to shoot the shooting scene to obtain a second target image.

7. The camera calibration method according to claim 1, characterized in that: The determining, based on the feature points, an internal parameter of the zoom camera corresponding to each of the calibration distances comprises: Determining a target calibration distance and a non-target calibration distance from the calibration distances; Determining internal parameters of the zoom camera corresponding to the target calibration distance based on the feature points corresponding to the target calibration distance, wherein the internal parameters corresponding to the target calibration distance include a focal length and a principal point coordinate corresponding to the target calibration distance; The focal length of the zoom camera corresponding to the non-target calibration distance is determined based on the feature points corresponding to the non-target calibration distance, and the principal point coordinates corresponding to the target calibration distance are used as the principal point coordinates corresponding to the non-target calibration distance.

8. A camera calibration device, characterized in that: include: An image acquisition module, used to acquire calibration images of the calibration plate from different angles at different calibration distances from the calibration plate using a zoom camera and with code values ​​corresponding to each calibration distance, wherein the calibration plate includes a plurality of patterns; A feature point extraction module, used to extract feature points from the calibration images corresponding to each of the calibration distances, wherein the features of the calibration pattern corresponding to at least some of the feature points corresponding to the farthest calibration distance are different from the features of the calibration pattern corresponding to at least some of the feature points corresponding to the closest calibration distance; A parameter determination module is used to determine the internal parameters of the zoom camera corresponding to each of the calibration distances based on the feature points.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed on a computer, the computer executes the camera calibration method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the camera calibration method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

Images