Camera calibration method, device, equipment and storage medium

Abstract

The present application discloses a calibration method, device, equipment and storage medium for a camera, belonging to the technical field of camera calibration. The method includes: obtaining a first image captured by the camera for a feature pattern in a three-dimensional space; obtaining the world coordinates and image coordinates respectively corresponding to a plurality of feature points in the first image; for the j-th iteration in multiple rounds of iteration, obtaining the j-th camera external parameters through a first fitting algorithm according to the j-th camera internal parameters, world coordinates and image coordinates; calculating the camera coordinates according to the j-th camera external parameters and world coordinates; in the case that the j-th iteration does not meet the iteration end condition, obtaining the (j + 1)-th camera internal parameters through a second fitting algorithm according to the camera coordinates and image coordinates; in the case that the j-th iteration meets the iteration end condition, determining the j-th camera internal parameters and the j-th camera external parameters as the calibration result of the camera. The above-provided camera calibration method has stronger universality compared with the related technology.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and particularly relates to a calibration method, device, equipment and storage medium for a camera. Background Art

[0002] In computer vision technology, in order to determine the mapping relationship between the three-dimensional position of a certain point in the world coordinate system and the two-dimensional position in the image coordinate system, it is necessary to establish a geometric model of camera imaging. The parameters of the geometric model are the camera parameters (including camera internal parameters and camera external parameters), and the process of solving the camera parameters is camera calibration.

[0003] In the related art, a method for solving camera internal parameters through two or three vanishing points is provided. The vanishing point refers to the intersection point when two lines parallel to each other in the world coordinate system are mapped to the image coordinate system. For the case of two vanishing points (that is, there are two sets of parallel lines in the world coordinate system), in the related art, if the two sets of parallel lines are perpendicular in the world coordinate system, then the vector from the camera focus c f to the vanishing point 1 is perpendicular to the vector from the camera focus c f to the vanishing point 2 . By using this property, a constraint equation is constructed, and the camera focal length f can be solved.

[0004] For the case of three vanishing points (that is, there are three sets of parallel lines in the world coordinate system, and the three sets of parallel lines are perpendicular to each other pairwise), in the same way as the case of two vanishing points, the related art can construct three constraint equations, and then the three-dimensional coordinates (u f , u x , -f) of the camera focus c y can be obtained. At this time, the three-dimensional coordinate system is obtained by establishing a right-handed rectangular coordinate system along the z-axis from the <camera focus> to the <optical center> direction under the 2D coordinate system of the camera imaging plane. According to the three-dimensional coordinates (u f , u x , -f) of the camera focus c y , the camera internal parameters can be determined.

[0005] However, the method for solving camera internal parameters through vanishing points can only calibrate wide-angle lenses (with a small zoom range). When the lens is closer to the telephoto range, the parallel lines in the world coordinate system are projected onto the imaging plane and are also closer to parallel lines, resulting in an inability to accurately calculate the vanishing point, and thus an inability to calibrate the camera internal parameters. Summary of the Invention

[0006] The present application provides a calibration method, device, equipment and storage medium for a camera, which has stronger universality compared with the related art. The technical solutions are as follows:

[0007] According to one aspect of the present application, a calibration method for a camera is provided. The method includes:

[0008] Obtaining a first image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0009] Obtaining the world coordinates corresponding to multiple feature points in the first image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively;

[0010] For the j-th iteration in multiple rounds of iteration, according to the j-th camera internal parameter of the camera, the world coordinates corresponding to the multiple feature points respectively, and the image coordinates corresponding to the multiple feature points respectively, the j-th camera external parameter of the camera is obtained through a first fitting algorithm; when j is equal to 1, the j-th camera internal parameter is the initial camera internal parameter, and j is a positive integer;

[0011] According to the j-th camera external parameter and the world coordinates corresponding to the multiple feature points respectively, the camera coordinates corresponding to the multiple feature points in the camera coordinate system are calculated respectively;

[0012] In the case that the j-th iteration does not meet the iteration end condition, according to the camera coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively, a (j + 1)-th camera internal parameter is obtained through a second fitting algorithm, and the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th iteration in the multiple rounds of iteration;

[0013] In the case that the j-th iteration meets the iteration end condition, the j-th camera internal parameter and the j-th camera external parameter are determined as the calibration result of the camera;

[0014] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

[0015] According to another aspect of the present application, a calibration method for a camera is provided. The method includes:

[0016] Obtaining a fourth image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0017] Obtaining the world coordinates corresponding to multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively;

[0018] For the k-th iteration in multiple rounds of iteration, according to the k-th camera external parameter of the camera and the world coordinates corresponding to the multiple feature points respectively, the camera coordinates corresponding to the multiple feature points in the camera coordinate system are calculated respectively; k is greater than or equal to 2, and k is a positive integer;

[0019] In the case that the iteration end condition is not satisfied in the k-th round of iteration, according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points, the k-th camera internal parameter is obtained by a second fitting algorithm; according to the k-th camera internal parameter, the world coordinates corresponding to the multiple feature points, and the image coordinates corresponding to the multiple feature points, the (k + 1)-th camera external parameter of the camera is obtained by a first fitting algorithm, and the (k + 1)-th camera external parameter is used to perform the (k + 1)-th round of iteration in the multiple rounds of iteration process;

[0020] In the case that the iteration end condition is satisfied in the k-th round of iteration, the k-th camera external parameter and the (k - 1)-th camera internal parameter are determined as the calibration result of the camera; when k is equal to 2, the (k - 1)-th camera external parameter used to fit the (k - 1)-th camera internal parameter is the initial camera external parameter;

[0021] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

[0022] According to one aspect of the present application, a calibration device for a camera is provided, and the device includes:

[0023] An acquisition module, configured to acquire a first image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0024] The acquisition module is further configured to acquire the world coordinates corresponding to the multiple feature points in the first image in a world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in an image coordinate system;

[0025] A processing module, configured to, for the j-th round of iteration in a multiple-round iteration process, obtain the j-th camera external parameter of the camera by a first fitting algorithm according to the j-th camera internal parameter of the camera, the world coordinates corresponding to the multiple feature points, and the image coordinates corresponding to the multiple feature points; when j is equal to 1, the j-th camera internal parameter is the initial camera internal parameter, and j is a positive integer;

[0026] The processing module is further configured to calculate the camera coordinates corresponding to the multiple feature points in a camera coordinate system according to the j-th camera external parameter and the world coordinates corresponding to the multiple feature points;

[0027] The processing module is further configured to, in the case that the iteration end condition is not satisfied in the j-th round of iteration, obtain the (j + 1)-th camera internal parameter by a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points, and the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th round of iteration in the multiple-round iteration process;

[0028] An output module, configured to determine the j-th camera internal parameter and the j-th camera external parameter as the calibration result of the camera when the j-th round of iteration satisfies the iteration end condition;

[0029] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the plurality of feature points.

[0030] According to one aspect of the present application, there is provided a calibration device for a camera, the device includes:

[0031] An acquisition module, configured to acquire a fourth image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0032] The acquisition module is further configured to acquire the world coordinates corresponding to the plurality of feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the plurality of feature points in the image coordinate system respectively;

[0033] A processing module, configured to, for the k-th round of iteration in a multi-round iteration process, calculate the camera coordinates corresponding to the plurality of feature points in the camera coordinate system according to the k-th camera external parameter of the camera and the world coordinates corresponding to the plurality of feature points respectively; k is greater than or equal to 2 and k is a positive integer;

[0034] The processing module is further configured to, when the k-th round of iteration does not satisfy the iteration end condition, obtain the k-th camera internal parameter through a second fitting algorithm according to the camera coordinates corresponding to the plurality of feature points and the image coordinates corresponding to the plurality of feature points respectively; according to the k-th camera internal parameter, the world coordinates corresponding to the plurality of feature points and the image coordinates corresponding to the plurality of feature points respectively, obtain the (k + 1)-th camera external parameter of the camera through a first fitting algorithm, and the (k + 1)-th camera external parameter is used to perform the (k + 1)-th round of iteration in the multi-round iteration process;

[0035] An output module, configured to determine the k-th camera external parameter and the (k - 1)-th camera internal parameter as the calibration result of the camera when the k-th round of iteration satisfies the iteration end condition; when k is equal to 2, the (k - 1)-th camera external parameter used to fit the (k - 1)-th camera internal parameter is the initial camera external parameter;

[0036] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the plurality of feature points.

[0037] According to one aspect of the present application, a computer device is provided. The computer device includes: a processor and a memory. The memory stores a computer program, and the computer program is loaded and executed by the processor to implement the camera calibration method as described above.

[0038] According to another aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the camera calibration method as described above.

[0039] According to another aspect of the present application, a computer program product is provided. The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the camera calibration method as described above.

[0040] The beneficial effects brought by the technical solutions provided in the embodiments of the present application at least include:

[0041] In the j-th iteration process, according to the world coordinates and image coordinates respectively corresponding to multiple feature points, combined with the j-th camera internal parameters, the j-th camera external parameters are obtained through the first fitting algorithm; according to the world coordinates respectively corresponding to multiple feature points and the j-th camera external parameters, the (j + 1)-th camera internal parameters are obtained through the second fitting algorithm, and the (j + 1)-th camera internal parameters are used for the (j + 1)-th iteration. Among them, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points, that is, the first fitting algorithm supports optimizing the camera internal parameters, and / or the second fitting algorithm supports optimizing the camera external parameters.

[0042] Through multiple rounds of iteration processes, the camera internal parameters and camera external parameters continuously approach the true values. By adopting the above method, the disadvantages of the related technology that it can only be applicable to wide-angle lenses, two or three groups of parallel lines in three-dimensional space need to be strictly parallel, and it can only calibrate camera lenses with the same horizontal focal length f x and vertical focal length f y are overcome. The camera calibration method provided by the present application has a wider usage scenario, that is, it has stronger universality. Description of the Drawings

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic diagram of the transformation process from three-dimensional space to two-dimensional image provided by an exemplary embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the principle provided by an exemplary embodiment of the present application;

[0046] Figure 3 It is a flowchart of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the real-time calibration method of the camera provided by an exemplary embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0050] Figure 7 It is a flowchart of the method for obtaining world coordinates provided by an exemplary embodiment of the present application;

[0051] Figure 8 It is a schematic diagram of the feature pattern after three-dimensional reconstruction provided by an exemplary embodiment of the present application;

[0052] Figure 9 It is a schematic diagram of the principle provided by an exemplary embodiment of the present application;

[0053] Figure 10 It is a flowchart of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0054] Figure 11 It is a schematic diagram of the real-time calibration method of the camera provided by an exemplary embodiment of the present application;

[0055] Figure 12 It is a schematic diagram of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0056] Figure 13 It is a schematic diagram of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0057] Figure 14 It is a schematic diagram of the effect of virtual-real fusion provided by an exemplary embodiment of the present application;

[0058] Figure 15 It is a schematic diagram of the effect of virtual-real fusion provided by an exemplary embodiment of the present application;

[0059] Figure 16 It is a schematic diagram of the calibration method of the camera provided by an exemplary embodiment of the present application;

[0060] Figure 17 It is a schematic diagram of a method for obtaining world coordinates provided by an exemplary embodiment of the present application;

[0061] Figure 18 It is a schematic diagram of a multi-round iteration process provided by an exemplary embodiment of the present application;

[0062] Figure 19 It is a structural block diagram of a calibration device for a camera provided by an exemplary embodiment of the present application;

[0063] Figure 20 It is a structural block diagram of a calibration device for a camera provided by an exemplary embodiment of the present application;

[0064] Figure 21 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0065] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0066] First, briefly introduce the nouns involved in the embodiments of the present application:

[0067] Camera calibration: In computer vision technology, to determine the mapping relationship between the three-dimensional position of a certain point in the world coordinate system and the two-dimensional position in the image coordinate system, it is necessary to establish a geometric model of camera imaging. The parameters of the geometric model are the camera parameters (including camera internal parameters and camera external parameters), and the process of solving the camera parameters is camera calibration.

[0068] Camera external parameters: Referring to Figure 1 , Figure 1 shows the process of mapping a feature point P in the three-dimensional space (world coordinate system) 101 to the feature point p' in the pixel plane (image coordinate system, or pixel coordinate system) 104. The coordinates of the feature point p in the three-dimensional space are in three-dimensional form, and the coordinates of the feature point p' in the pixel plane 104 are in two-dimensional form. The camera external parameters (or camera pose) are used to map the feature point p in the three-dimensional space to the camera plane (camera coordinate system) 102. The camera external parameters include two components: the rotation matrix R and the translation vector t. The rotation matrix R is used to describe the direction of the axes of the world coordinate system in the camera coordinate system; the translation vector t is used to describe the position of the origin of the world coordinate system in the camera coordinate system.

[0069] Camera internal parameters: Referring to Figure 1 , the camera internal parameters are used to map the feature point in the camera plane 102 to the feature point p' in the pixel plane 104.

[0070] Schematically, the matrix structure of the camera internal parameters is

[0071] Among them, f x represents the scaling coefficient of the x-axis scaling that occurs from the imaging plane 103 to the pixel plane 104 multiplied by the focal length f (abbreviated as the horizontal focal length), f y represents the scaling coefficient of the y-axis scaling that occurs from the imaging plane 103 to the pixel plane 104 multiplied by the focal length (abbreviated as the vertical focal length), c x represents the x-axis offset that occurs when the feature point on the imaging plane 103 is mapped to the pixel plane 104, c y represents the y-axis offset that occurs when the feature point on the imaging plane 103 is mapped to the pixel plane 104. From f x , f y , c x and c y the internal parameters of the camera in the above form can be obtained.

[0072] In the related art, a method for solving the internal parameters of a camera through two or three vanishing points is provided. The vanishing point refers to the intersection point when two mutually parallel lines in the world coordinate system are mapped to the image coordinate system.

[0073] For the case of only 2 vanishing points, the following steps are performed:

[0074] 1. Obtain the optical center coordinates (u x , u y ): Denote the width and height of the image as w and h respectively. Then the optical center of the camera is at the center of the image, that is, u x = w / 2, u y = h / 2, u x , u y are the horizontal and vertical axis coordinates of the optical center in the camera plane respectively.

[0075] 2. In the 2D xy coordinate system of the camera focal plane, take the direction from the camera focus to the optical center as the z-axis to establish a right-handed rectangular coordinate system.

[0076] 3. In this coordinate system, denote the camera focus c f coordinates as (u x , u y , -f), the coordinates of the optical center c are (u x , u y , 0), the vanishing point 1 coordinates p are (p x , p y , 0), and the vanishing point 2 coordinates q are (q x , q y , 0). u x , u y are the x-axis and y-axis coordinates of the camera focus c f and the optical center c in this right-handed rectangular coordinate system, p x, p y are the x-axis and y-axis coordinates of the vanishing point 1 of this right-handed rectangular coordinate system, respectively, and q x , q y are the x-axis and y-axis coordinates of the vanishing point 2 of this right-handed rectangular coordinate system, respectively.

[0077] 4. Obtain the constraint equation: Since there are two sets of parallel lines perpendicular to each other in three-dimensional space, the vector from the camera focus c f to the vanishing point 1 (this vector is parallel to a set of parallel lines) is perpendicular to the vector from the camera focus c f to the vanishing point 2 (this vector is parallel to another set of parallel lines), that is Expanding it gives the constraint equation:

[0078] (p x - u x )(q x - u x )+(p y - u y )(q y - u y )+ f 2 = 0;

[0079] 5. According to the above equation, the value of f can be calculated:

[0080]

[0081] For the case of three vanishing points, perform the following steps:

[0082] For the case of three vanishing points, it is similar to the case of two vanishing points. However, since there are three vanishing points, it means there are three sets of parallel lines perpendicular to each other in three-dimensional space. Therefore, three constraint equations can be formed based on the perpendicular relationship, and thus u x , u y , and f can be solved without defaulting u x = w / 2 and u y = h / 2 as in the case of two vanishing points.

[0083] 1. Establish the world coordinate system: In the two-dimensional xy coordinate system of the camera focal plane, take the direction from the camera focus c f towards the <optical center as the z-axis to establish a right-handed rectangular coordinate system.

[0084] 2. In this coordinate system, denote the coordinates of the camera focus c f as (u x , u y , -f), and the coordinates of the optical center as (u x , u y,0), the coordinates of vanishing point 1, p, are (p x ,p y ,0), the coordinates of vanishing point 2, q, are (q x ,q y ,0), and the coordinates of vanishing point 3, r, are (r x ,r y ,0); u x 、u y are the x-axis and y-axis coordinates of the camera focus c f and the optical center c in this right-handed rectangular coordinate system. p x 、p y are the x-axis and y-axis coordinates of vanishing point 1 in this right-handed rectangular coordinate system. q x 、q y are the x-axis and y-axis coordinates of vanishing point 2 in this right-handed rectangular coordinate system. r x 、r y are the x-axis and y-axis coordinates of vanishing point 3 in this coordinate system respectively.

[0085] 3. Obtain the constraint equations: Since every two sets of parallel lines are perpendicular to each other in three-dimensional space, so:

[0086] (A) The vector from the camera focus c f > to the <vanishing point 1> (this vector is parallel to a set of parallel lines) is perpendicular to the vector from the <camera focus> to the <vanishing point 2> (this vector is parallel to another set of parallel lines);

[0087] (B) The vector from the camera focus c f > to the <vanishing point 1> is perpendicular to the vector from the <camera focus> to the <vanishing point 3>

[0088] (C) The vector from the camera focus c f > to the <vanishing point 2> is perpendicular to the vector from the <camera focus> to the <vanishing point 3>

[0089] In summary, 3 constraint equations are obtained, namely:

[0090]

[0091] Expanding gives:

[0092]

[0093] 4. After simplification, the calculation method of [u x u y can be obtained: T The calculation method of

[0094]

[0095] 5. Obtain u x and u y After that, the focal length f can be calculated according to the following formula:

[0096]

[0097] However, the method for solving the camera internal parameters through the above two or three vanishing points has the following disadvantages:

[0098] First, during camera calibration, two or three sets of parallel lines in three-dimensional space need to be strictly parallel. When the requirement of parallelism cannot be strictly met, the calibration accuracy will decrease significantly, or calibration may not be possible directly.

[0099] Second, only the horizontal focal length f x and the vertical focal length f y of the camera lens with the same horizontal and vertical focal lengths can be calibrated. For a camera lens with different horizontal focal length f x and vertical focal length f y it is not applicable (such as an anamorphic widescreen lens).

[0100] Third, only wide-angle lenses can be calibrated, and only wide-angle lenses (with a small zoom range) can be calibrated. When the lens is closer to the telephoto range, the parallel lines in the world coordinate system projected onto the imaging plane are also closer to parallel lines, resulting in the inability to accurately calculate the vanishing point and thus unable to calibrate the camera internal parameters.

[0101] Based on the above disadvantages, the present application provides the camera calibration method described below.

[0102] Figure 2 The schematic diagram of the principle of the camera calibration method provided by an exemplary embodiment of the present application is shown. Figure 2 The shown camera calibration method is executed by the computer device 21. Figure 2 The multi-round iteration process 200 for the first image is shown. The first image is any image captured by the camera, and the multi-round iteration process 200 is used to calibrate the camera that captured the first image.

[0103] For any iteration in the multi-round iteration process 200, determine multiple feature points (sometimes also referred to as reference points, key points, etc.) in the first image. According to the image coordinates 201 respectively corresponding to the multiple feature points, the world coordinates 202 respectively corresponding to the multiple feature points, and the camera internal parameters 203, through the first fitting algorithm, obtain the camera external parameters 204. Optionally, the first fitting algorithm supports optimizing the camera external parameters so that the optimized camera external parameters are closer to the true external parameters. According to the world coordinates 202 respectively corresponding to the multiple feature points and the camera external parameters 204, calculate the camera coordinates 205 respectively corresponding to the multiple feature points.

[0104] Determine whether the current iteration round meets the iteration end condition. If it does not meet the iteration end condition, then according to the camera coordinates 205 respectively corresponding to the multiple feature points and the image coordinates 201 respectively corresponding to the multiple feature points, through the second fitting algorithm, obtain new camera internal parameters. The new camera internal parameters are used to execute the next round of the iteration process. Optionally, the second fitting algorithm supports optimizing the camera internal parameters so that the optimized camera internal parameters are closer to the true internal parameters. If the current round meets the iteration end condition, then use the current camera internal parameters and camera external parameters as the iteration results of the multi-round iteration process, and output the current camera internal parameters and camera external parameters.

[0105] It should be noted that the first fitting algorithm supports optimizing the camera external parameters, and / or the second fitting algorithm supports optimizing the camera internal parameters. That is, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points. The smaller the reprojection error, the closer the camera internal parameters and / or camera external parameters after iterative fitting are to the true values.

[0106] In one embodiment, the above computer device 21 may include one or more computer devices ( Figure 2 only the case of one computer device is shown). When only one computer device is included, the device can be a terminal or a server; when multiple computer devices are included, Figure 2The calibration method of the camera shown can be executed by multiple terminals, or multiple servers, or jointly executed by terminals and servers. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The above-mentioned terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, smart TVs, etc., but are not limited thereto. The terminals and servers can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0107] Figure 3 The flowchart of the camera calibration method provided by an exemplary embodiment of this application is shown. Taking the method being executed by Figure 2 the computer device 21 shown as an example, the method includes:

[0108] Step 310, obtaining a first image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0109] The camera is a real camera in a three-dimensional space, and there is at least one feature pattern in the three-dimensional space. Obtain the first image obtained by the camera photographing at least one feature pattern. Multiple feature points of the feature pattern are displayed on the first image. In the embodiments of this application, each feature pattern has multiple feature points. The feature points are points used to calculate the camera parameters for transforming from a three-dimensional space to a two-dimensional image, that is, the reference points for executing the camera calibration method of this application. The feature points correspond to three-dimensional coordinates in the world coordinate system of the three-dimensional space and two-dimensional coordinates on the image coordinate system of the two-dimensional image.

[0110] The feature pattern, in this application, the feature pattern is a carrier pattern carrying feature points, and the feature pattern is any pattern that supports the detection of feature points. Schematically, the feature pattern is a two-dimensional code. At this time, the four corner points of the two-dimensional code respectively correspond to a feature point, that is, a two-dimensional code has four feature points. Schematically, the feature pattern is a pentagram pattern. At this time, the five corner points of the pentagram respectively correspond to a feature point, that is, a pentagram pattern has five feature points.

[0111] Step 320, obtaining the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system of the three-dimensional space and the image coordinates corresponding to them in the image coordinate system respectively;

[0112] In one embodiment, perform image detection on the first image to determine multiple feature points in the first image and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively. Obtain the world coordinates of the multiple feature points in the first image in the world coordinate system. Optionally, perform three-dimensional reconstruction through SFM (Structure from Motion) to establish the world coordinate system corresponding to the three-dimensional space, and then obtain the world coordinates of the multiple feature points.

[0113] Step 330, for the j-th iteration in the multiple-round iteration process, obtain the j-th camera extrinsic parameter of the camera through the first fitting algorithm according to the j-th camera intrinsic parameter, the world coordinates corresponding to the multiple feature points respectively, and the image coordinates corresponding to the multiple feature points respectively;

[0114] After obtaining the image coordinates and world coordinates corresponding to the multiple feature points in the first image respectively, perform a multiple-round iteration process to continuously optimize and obtain the camera intrinsic parameter and the camera extrinsic parameter. Figure 3 The method shown will give an initial camera intrinsic parameter and perform a multiple-round iteration process starting from the initial camera intrinsic parameter.

[0115] Optionally, the initial camera intrinsic parameter is a preset initial value or is obtained by converting the readings of the camera lens. It should be noted that at this time, the initial camera intrinsic parameter does not need to be very accurate, and only needs to be approximate to the standard value (such as the camera intrinsic parameter obtained by the Zhang Zhengyou calibration method).

[0116] For the j-th iteration in the multiple-round iteration process, obtain the j-th camera extrinsic parameter of the camera through the first fitting algorithm according to the j-th camera intrinsic parameter, the world coordinates corresponding to the multiple feature points respectively, and the image coordinates corresponding to the multiple feature points respectively (intrinsic parameter → extrinsic parameter), where j is a positive integer. Optionally, the first optimization algorithm is the SolvePNP algorithm. The SolvePNP algorithm supports reducing the reprojection error, that is, optimizing the camera extrinsic parameter so that the optimized camera extrinsic parameter is closer to the camera extrinsic parameter when taking the first image.

[0117] When performing the first iteration, that is, when j is equal to 1, the j-th camera intrinsic parameter is the initial camera intrinsic parameter.

[0118] Step 340, calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system respectively according to the j-th camera extrinsic parameter and the world coordinates corresponding to the multiple feature points respectively;

[0119] For the j-th iteration in the multiple-round iteration process, calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system respectively according to the j-th camera extrinsic parameter and the world coordinates corresponding to the multiple feature points respectively.

[0120] Step 350, in the case that the iteration end condition is not met in the j-th iteration, according to the camera coordinates corresponding to multiple feature points and the image coordinates corresponding to multiple feature points respectively, the (j + 1)-th camera internal parameter is obtained through the second fitting algorithm, and the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th iteration in the multi-round iteration process;

[0121] When the iteration end condition is not met in the j-th iteration, continue to execute the optimization process from the internal parameter to the external parameter. According to the camera coordinates corresponding to multiple feature points and the image coordinates corresponding to multiple feature points respectively, the (j + 1)-th camera internal parameter is obtained through the second fitting algorithm. Optionally, the second fitting algorithm supports reducing the reprojection error, that is, optimizing the camera internal parameter so that the optimized camera internal parameter is closer to the camera internal parameter when the first image is taken. The obtained (j + 1)-th camera internal parameter is used to perform the next iteration.

[0122] In one embodiment, the iteration end condition includes that the number of iterations reaches the number threshold and / or the reprojection error of multiple feature points is less than the threshold. The reprojection error refers to the mean square error between the coordinates of the feature points on the imaging plane and the detected image coordinates. Optionally, the reprojection error of multiple feature points refers to the mean of the reprojection errors of multiple feature points.

[0123] In the embodiments of the present application, the first fitting algorithm supports optimizing the camera external parameter, and / or, the second fitting algorithm supports optimizing the camera internal parameter. That is, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points. The smaller the reprojection error, the closer the camera internal parameter and / or camera external parameter after iterative fitting is to the true value.

[0124] Step 360, in the case that the iteration end condition is met in the j-th iteration, determine the j-th camera internal parameter and the j-th camera external parameter as the calibration result of the camera.

[0125] When the iteration end condition is met in the j-th iteration, end the multi-round iteration process, and determine the j-th camera internal parameter and the j-th camera external parameter as the calibration result of the camera for taking the first image.

[0126] To sum up, in the j-th iteration process, according to the world coordinates and image coordinates corresponding to multiple feature points respectively, combined with the j-th camera internal parameter, the j-th camera external parameter is obtained through the first fitting algorithm; according to the world coordinates corresponding to multiple feature points and the j-th camera external parameter respectively, the (j + 1)-th camera internal parameter is obtained through the second fitting algorithm, and the (j + 1)-th camera internal parameter is used for the (j + 1)-th iteration. Among them, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points, that is, the first fitting algorithm supports optimizing the camera internal parameter, and / or, the second fitting algorithm supports optimizing the camera external parameter.

[0127] Through multiple rounds of iterative processes, the internal camera parameters and external camera parameters are continuously approximated to the true values. By adopting the above method, the defects of the related technology that it can only be applied to wide-angle lenses, two or three sets of parallel lines in three-dimensional space need to be strictly parallel, and it can only calibrate the horizontal focal length f x and the vertical focal length f y of the same camera lens are overcome. The camera calibration method provided by this application has a wider usage scenario, that is, it has stronger universality.

[0128] Moreover, the method provided by this application does not limit the posting method of the feature pattern in three-dimensional space, overcoming the defect that the related technology must strictly ensure parallelism. Moreover, the method provided by this application not only supports the ordinary camera lenses supported by the related technology, but also supports wide-screen lenses (lenses with different horizontal focal lengths fx and vertical focal lengths fy). Moreover, the method provided by this application can not only calibrate wide-angle zoom lenses, but also support the calibration of large zoom lenses (such as lenses zooming from wide angle to telephoto).

[0129] Based on Figure 3 In the optional embodiment shown, in step 350, "According to the camera coordinates corresponding to multiple feature points and the image coordinates corresponding to multiple feature points, the (j + 1)-th internal camera parameters are obtained through the second fitting algorithm", which includes: for any one of the multiple feature points, based on the camera coordinates of the feature point and the image coordinates of the feature point, a fitting equation corresponding to the feature point is constructed; the least squares method is used to solve the fitting equations corresponding to multiple feature points respectively, and the (j + 1)-th internal camera parameters are obtained.

[0130] Schematically, the matrix structure of the internal camera parameters is

[0131] where f x represents the scaling coefficient of the x-axis scaling from the imaging plane to the pixel plane multiplied by the focal length f (abbreviation: horizontal focal length), f y represents the scaling coefficient of the y-axis scaling from the imaging plane to the pixel plane multiplied by the focal length (abbreviation: vertical focal length), c x represents the x-axis offset when the feature point on the imaging plane is mapped to the pixel plane, c y represents the y-axis offset when the feature point on the imaging plane is mapped to the pixel plane.

[0132] For any one feature point, the following fitting equation can be constructed:

[0133]

[0134] where (x, y, z) are the camera coordinates of the feature point, and (u, v) are the image coordinates of the feature point.

[0135] In one embodiment, for the fitting equations corresponding to multiple feature points, the least square method can be used to solve f x 、f y 、c x and c y , and then the camera intrinsic parameters can be calculated.

[0136] In one embodiment, the least square method is used to take the jth camera intrinsic parameter as a constraint item, and the fitting equations corresponding to the plurality of feature points are combined to solve the j+1th camera intrinsic parameter;

[0137] The j-th camera intrinsic parameter is used to constrain the unique solution of the j+1-th camera intrinsic parameter to be obtained by fitting when an infinite set of solutions is obtained by the least squares method.

[0138] Indicatively, the following formula is used to calculate the intrinsic parameter f of the j+1th camera: x 、f y 、c x and c y .

[0139]

[0140] Among them, x1 to x n Represents the x-axis component of the camera coordinates of the n feature points (the horizontal axis component of the camera coordinate system), y1 to y n Represents the y-axis component of the camera coordinates of the n feature points (the vertical axis component of the camera coordinate system); u1 to u n Represents the u-axis component of the image coordinates of the n feature points (the horizontal axis component of the image coordinate system); v1 to v n Represents the v-axis component of the image coordinates of the n feature points (the vertical axis component of the image coordinate system). f represents the intrinsic parameter of the jth camera x , f represents the intrinsic parameter of the jth camera y , c represents the intrinsic parameter of the jth camera x , c represents the intrinsic parameter of the jth camera y . Among them, λ is the proportional coefficient hyperparameter.

[0141] In summary, the above provides a second fitting algorithm to support the iterative process of obtaining camera intrinsic parameters from camera extrinsic parameters. The second fitting algorithm also uses the jth camera intrinsic parameter as a constraint item to ensure that when the camera plane of the camera is parallel to the plane of the feature pattern, the least squares method can obtain a unique solution.

[0142] based on Figure 3 In the illustrated alternative embodiment, Figure 3The process of performing multiple rounds of iterative processes on the first image is shown. To achieve real-time continuous calibration of the camera, the internal camera parameters obtained by calibrating the first image are also determined as the initial internal camera parameters when calibrating the second image, so as to complete the continuous calibration of the camera. The second image is a subsequent frame image of the first image. Optionally, the second image is the next frame image of the first image. Figure 4 The schematic diagram of the camera calibration method provided by an exemplary embodiment of the present application is shown. Steps 310 to 360 show the calibration method for the camera that captures the first image, corresponding to Figure 4 the first calibration stage 401 in Figure 4 After step 360, the following operations are also performed, corresponding to

[0143] Obtain the second image captured by the camera for the feature pattern in the three-dimensional space. The second image is a subsequent frame image of the first image; use the internal camera parameters of the camera calibrated under the first image as the initial internal camera parameters for performing multiple rounds of iterative processes on the second image; based on the initial internal camera parameters of the second image, obtain the calibration result of the camera under the second image by performing multiple rounds of iterative processes under the second image;

[0144] After calibrating the camera that captures the first image, the camera that captures the second image will also be calibrated. The second image is a subsequent frame image of the first image. Optionally, the second image is the next frame image of the first image. The changes in the internal camera parameters and / or external camera parameters during the camera shooting process are coherent. The change in the internal camera parameters can be simply understood as the camera has been focused, and the change in the external camera parameters can be simply understood as the position of the camera has changed.

[0145] Using the calibration result of the previous frame image as the initial calibration result for performing multiple rounds of iterative processes on the current frame helps to achieve fast iteration. It can be understood that since the change process of the internal and external camera parameters is continuous, the real-time calibration method provided in this embodiment cleverly uses the calibration result of the previous frame as the iteration starting point of the current frame, taking advantage of the continuity of the camera change between the previous frame and the current frame, which helps to quickly solve the calibration result of the current frame. In the related art, real-time calibration is performed separately for each frame, and the speed of real-time calibration is slow and inaccurate. Compared with the related art, the calibration effect of this embodiment has been significantly improved.

[0146] Based on Figure 3 In the optional embodiment shown, there are at least the following two possible implementation manners for the execution order between steps 340 and 360.

[0147] The first possible implementation manner. Schematically, with reference to Figure 5, for any iteration in the multi-round iteration process, determine multiple feature points in the first image. According to the image coordinates 201 corresponding to the multiple feature points respectively, the world coordinates 202 corresponding to the multiple feature points respectively, and the camera internal parameters 203, through the first fitting algorithm, obtain the camera external parameters 204. According to the world coordinates 202 corresponding to the multiple feature points respectively and the camera external parameters 204, calculate the camera coordinates 205 corresponding to the multiple feature points respectively. Project the camera coordinates 205 corresponding to the multiple feature points respectively onto the imaging plane of the camera to obtain the reprojection coordinates 206 corresponding to the multiple feature points respectively; according to the reprojection coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively, calculate the reprojection error of the multiple feature points; determine whether the reprojection error is less than the threshold; schematically, calculate the reprojection error using the following formula:

[0148]

[0149] where ε p is the reprojection error of the multiple feature points, P p is the reprojection coordinates of the multiple feature points, P i is the image coordinates of the multiple feature points, and N is the number of feature points.

[0150] In the case where the reprojection error is less than the threshold, determine that the iteration end condition is satisfied; or, in the case where the reprojection error is less than the threshold and the number of iterations reaches the number threshold, determine that the iteration end condition is satisfied;

[0151] If the iteration end condition is not satisfied, according to the camera coordinates 205 corresponding to the multiple feature points respectively and the image coordinates 201 corresponding to the multiple feature points respectively, through the second fitting algorithm, obtain new camera internal parameters.

[0152] If the iteration end condition is satisfied, then take the current camera internal parameters and camera external parameters as the iteration results of the multi-round iteration process, and output the current camera internal parameters and camera external parameters.

[0153] The second possible implementation manner, schematically, in combination with reference Figure 6 , for any iteration in the multi-round iteration process, determine multiple feature points in the first image. According to the image coordinates 201 corresponding to the multiple feature points respectively, the world coordinates 202 corresponding to the multiple feature points respectively, and the camera internal parameters 203, through the first fitting algorithm, obtain the camera external parameters 204.

[0154] In the case where the number of iterations reaches the number threshold, determine that the iteration end condition is satisfied.

[0155] If the iteration end condition is not satisfied, according to the world coordinates 202 corresponding to multiple feature points and the extrinsic camera parameters 204, the camera coordinates 205 corresponding to multiple feature points are calculated. According to the camera coordinates 205 corresponding to multiple feature points and the image coordinates 201 corresponding to multiple feature points, a new intrinsic camera parameter is obtained through a second fitting algorithm. The new intrinsic camera parameter is used to perform the next round of iteration.

[0156] If the iteration end condition is satisfied, the current intrinsic camera parameter and extrinsic camera parameter are used as the iteration results of the multi-round iteration process, and the current intrinsic camera parameter and extrinsic camera parameter are output.

[0157] Based on Figure 3 In the optional embodiment shown, step 310 includes step S1.

[0158] Step S1, obtain a first image, where the first image is obtained by the camera photographing a first plane and a second plane in a three-dimensional space. The first image shows all or part of the images of the first plane and the second plane. The first plane and the second plane intersect and each has at least one feature pattern, and one feature pattern includes multiple feature points.

[0159] Schematically, the three-dimensional space includes the ground (the first plane) and the wall (the second plane). At least one feature pattern is posted on the ground, and at least one feature pattern is posted on the wall. The first image is an image obtained by the camera photographing the ground and the wall. At this time, the first image contains all or part of the images of the ground and the wall.

[0160] In one embodiment, the feature pattern includes a QR code pattern. The vertices of the QR code pattern correspond to the feature points. The multiple QR code patterns on the first plane in the three-dimensional space are in a non-overlapping state, and the multiple QR code patterns on the second plane in the three-dimensional space are in a non-overlapping state. It should be noted that the camera calibration method provided in this embodiment supports any state other than the non-overlapping state. This embodiment does not limit the spacing, rotation angle, etc. of the multiple QR code patterns.

[0161] Based on Figure 3 In the optional embodiment shown, "obtain the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system of the three-dimensional space" in step 320 includes step S2.

[0162] Step S2, based on multiple first feature points on the first plane and multiple second feature points on the second plane in the three-dimensional space, establish a world coordinate system of the three-dimensional space; determine the world coordinates corresponding to the multiple feature points of the first image in the world coordinate system.

[0163] Step S2 is used to implement 3D reconstruction of the three-dimensional space, that is, to determine the world coordinates of multiple feature points in the three-dimensional space. A method of 3D reconstruction will be given below. Figure 7 It shows a method of 3D reconstruction provided by an exemplary embodiment of the present application.

[0164] Step S2-1: Obtain a third image obtained by the camera photographing the first plane and the second plane in the three-dimensional space. The third image shows the entire pictures of the first plane and the second plane;

[0165] Schematically, the three-dimensional space includes the ground (the first plane) and the wall surface (the second plane). At least one feature pattern is posted on the ground, and at least one feature pattern is posted on the wall surface. Optionally, the feature pattern is a QR code, and the four corner points of the QR code are respectively used as feature points. The third image is an image obtained by the camera photographing the ground and the wall surface. At this time, the third image needs to include the entire pictures of the ground and the wall surface.

[0166] It should be noted that the QR code patterns in the three-dimensional space are all different. Moreover, the first plane and the second plane do not need to be perpendicular (only a certain included angle is required), and the QR codes on each plane can be placed arbitrarily, that is, there is no need to adopt the standard that any two sets of parallel lines need to be strictly perpendicular in the vanishing point calibration method above for placement.

[0167] Step S2-2: For the first plane, detect the first image coordinates respectively corresponding to multiple first feature points on the first plane in the third image; based on the first plane, establish a first local coordinate system; determine the first three-dimensional coordinates respectively corresponding to the multiple first feature points in the first local coordinate system; according to the first three-dimensional coordinates and the first image coordinates respectively corresponding to the multiple first feature points, obtain a first transformation matrix, and the first transformation matrix is used to represent the plane attitude of the first plane;

[0168] Schematically, perform feature point detection on the third image to obtain the first image coordinates of multiple feature points on the first plane. Take the first plane as the center and establish a first local coordinate system. Determine the first three-dimensional coordinates respectively corresponding to the multiple first feature points in the first local coordinate system. Use the SolvePNP algorithm to obtain a first transformation matrix M1 according to the first three-dimensional coordinates and the first image coordinates respectively corresponding to the multiple first feature points. The first transformation matrix M1 is the transformation matrix used to map the first local coordinate system to the image coordinate system.

[0169] Schematically, Figure 8 It shows the virtual three-dimensional space after 3D reconstruction, Figure 8 It shows a situation of one ground and two wall surfaces, that is, a situation including the first plane, the second plane and the third plane. Figure 8The middle ground contains multiple QR code panels, and one QR code panel contains multiple QR code patterns (for example, one QR code panel is a rectangular structure, and one QR code panel includes a 3×4 QR code pattern); the wall also contains multiple QR code panels, and one QR code panel contains multiple QR code patterns.

[0170] Step S2-3: For the second plane, detect the second image coordinates corresponding to multiple second feature points on the second plane in the third image; based on the second plane, establish a second local coordinate system; determine the second three-dimensional coordinates corresponding to the multiple second feature points in the second local coordinate system respectively; according to the second three-dimensional coordinates and the third image coordinates corresponding to the multiple second feature points respectively, obtain a second transformation matrix, and the second transformation matrix is used to represent the plane attitude of the second plane.

[0171] Schematically, perform feature point detection on the third image to obtain the first image coordinates of multiple feature points on the second plane. Take the second plane as the center and establish a second local coordinate system. Determine the second three-dimensional coordinates corresponding to the multiple second feature points in the second local coordinate system respectively. Use the SolvePNP algorithm to obtain a second transformation matrix M2 according to the second three-dimensional coordinates and the second image coordinates corresponding to the multiple second feature points respectively. The first transformation matrix M2 is the transformation matrix used to map the second local coordinate system to the image coordinate system.

[0172] Step S2-4: Take the first local coordinate system as the world coordinate system, and take the first three-dimensional coordinate as the world coordinate, so as to obtain the world coordinates corresponding to the multiple first feature points respectively.

[0173] Take the first local coordinate system as the world coordinate system, and take the first three-dimensional coordinate as the world coordinate. At this time, denote the first transformation matrix M1 as M f .

[0174] Step S2-5: Divide the second transformation matrix by the first transformation matrix to obtain a third transformation matrix, and the third transformation matrix is used to represent the transformation matrix for transforming the second local coordinate system into the first local coordinate system; multiply the second three-dimensional coordinates corresponding to the multiple second feature points by the third transformation matrix to obtain the world coordinates corresponding to the multiple second feature points respectively.

[0175] Schematically, use the following formula to calculate the third transformation matrix:

[0176] M ti =M f -1 ·M2;

[0177] where, M ti is the third transformation matrix.

[0178] Multiply the second three-dimensional coordinates by the third transformation matrix M tiThe world coordinates of the second feature point can be obtained.

[0179] Schematically, the following formula is used to calculate the world coordinates of the second feature point.

[0180] Q = M ti P;

[0181] where Q is the world coordinate and P is the second three-dimensional coordinate of the second feature point in the second local coordinate system.

[0182] It should be noted that the above only takes two planes, namely the first plane and the second plane, as examples for introduction. In fact, there may also be a third plane, a fourth plane, etc. The specific implementation method is similar to the three-dimensional reconstruction method in the case of two planes, that is, after constructing their respective local coordinate systems respectively, they are then converted to the world coordinate system.

[0183] It should also be noted that the local coordinate system described above is constructed based on the first plane. It is also possible to construct a local coordinate system based on a feature image (or multiple feature patterns) on the first plane. At this time, in addition to converting the local coordinate systems of other planes to the world coordinate system, it is also necessary to convert the other local coordinate systems in the same plane to the world coordinate system. The above only introduces one of the most basic conversion methods, and similar variant conversion methods based on this basic conversion method should only fall within the protection scope of this application.

[0184] In summary, a method for three-dimensional reconstruction (i.e., establishing a world coordinate system in a three-dimensional space) is provided to support obtaining the world coordinates of multiple feature points in the first image.

[0185] Figure 9 The schematic diagram of the principle of the camera calibration method provided by an exemplary embodiment of this application is shown. Figure 9 The shown camera calibration method is executed by the computer device 91. Figure 9 The multi-round iteration process 900 for the fourth image is shown. The fourth image is any image captured by the camera, and the multi-round iteration process 900 is used to perform camera calibration on the camera that captures the fourth image.

[0186] For any round of iteration in the multi-round iteration process 900, multiple feature points (sometimes also referred to as reference points, key points, etc.) in the fourth image are determined. The world coordinates 901 corresponding to the multiple feature points and the image coordinates 902 corresponding to the multiple feature points are obtained. According to the world coordinates 901 corresponding to the multiple feature points and the external camera parameters 903, the camera coordinates 904 corresponding to the multiple feature points are calculated.

[0187] Determine whether the current iteration round meets the iteration end condition. If it does not meet the iteration end condition, based on the image coordinates 902 corresponding to multiple feature points and the camera coordinates 904 corresponding to multiple feature points respectively, use the second fitting algorithm to obtain the camera internal parameters. Optionally, the second fitting algorithm supports optimizing the camera internal parameters so that the optimized camera internal parameters are closer to the true internal parameters. Based on the camera internal parameters 905, the image coordinates 902 corresponding to multiple feature points, and the world coordinates 901 corresponding to multiple feature points respectively, use the first fitting algorithm to obtain new camera external parameters. Optionally, the first fitting algorithm supports optimizing the camera external parameters so that the optimized camera external parameters are closer to the true external parameters. The new camera external parameters are used to perform the next iteration process.

[0188] If the current round meets the iteration end condition, then use the current camera internal parameters and camera external parameters as the iteration results of the multi-round iteration process, and output the current camera internal parameters and camera external parameters.

[0189] It should be noted that the first fitting algorithm supports optimizing the camera external parameters, and / or the second fitting algorithm supports optimizing the camera internal parameters. That is, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points. The smaller the reprojection error, the closer the camera internal parameters and / or camera external parameters after iterative fitting are to the true values.

[0190] In one embodiment, the above computer device 91 may include one or more computer devices ( Figure 9 only one computer device is shown). When only one computer device is included, the device can be a terminal or a server; when multiple computer devices are included, Figure 9 the shown camera calibration method can be executed by multiple terminals, or multiple servers, or jointly executed by terminals and servers. The above server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The above terminals can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, smart TVs, etc., but are not limited thereto. The terminals and servers can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0191] Figure 10 shows the flowchart of the camera calibration method provided by an exemplary embodiment of the present application. And Figure 3The difference between the calibration methods shown is that Figure 10 the external camera parameters are used as the starting point for the multi-round iterative process. Taking Figure 10 the camera calibration method shown is Figure 9 executed by the computer device 91 in

[0192] Step 1010, obtain a fourth image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0193] The camera is a real camera in a three-dimensional space. There is at least one feature pattern in the three-dimensional space. Obtain a fourth image obtained by the camera photographing at least one feature pattern. Multiple feature points of the feature pattern are displayed on the fourth image. In the embodiments of the present application, each feature pattern has multiple feature points. The feature points are points used to calculate the camera parameters for transforming from a three-dimensional space to a two-dimensional image, that is, the reference points for executing the camera calibration method of the present application. The feature points correspond to three-dimensional coordinates in the world coordinate system of the three-dimensional space and two-dimensional coordinates on the image coordinate system of the two-dimensional image.

[0194] The feature pattern. In the present application, the feature pattern is a carrier pattern carrying feature points, and the feature pattern is any pattern that supports the detection of feature points. Schematically, the feature pattern is a two-dimensional code. At this time, the four corner points of the two-dimensional code respectively correspond to a feature point, that is, a two-dimensional code has four feature points. Schematically, the feature pattern is a pentagram pattern. At this time, the five corner points of the pentagram respectively correspond to a feature point, that is, a pentagram pattern has five feature points.

[0195] Step 1020, obtain the world coordinates corresponding to the multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively;

[0196] In one embodiment, perform image detection on the fourth image to determine the multiple feature points in the fourth image and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively. Obtain the world coordinates of the multiple feature points in the fourth image in the world coordinate system. Optionally, perform three-dimensional reconstruction through SFM (Structure from Motion) to establish the world coordinate system corresponding to the three-dimensional space, and then obtain the world coordinates of the multiple feature points.

[0197] Step 1030, for the k-th round of iteration in the multi-round iterative process, calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system according to the k-th external camera parameters of the camera and the world coordinates corresponding to the multiple feature points respectively;

[0198] For the k-th iteration in the multi-round iteration process, the camera coordinates corresponding to multiple feature points in the camera coordinate system are calculated based on the external parameters of the k-th camera and the world coordinates corresponding to the multiple feature points respectively. k is greater than or equal to 2.

[0199] Step 1040, when the iteration end condition is not satisfied in the k-th iteration, the k-th camera internal parameters are obtained through the second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively; the (k + 1)-th camera external parameters of the camera are obtained through the first fitting algorithm according to the k-th camera internal parameters, the world coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively, and the (k + 1)-th camera external parameters are used to perform the (k + 1)-th iteration in the multi-round iteration process;

[0200] When the k-th iteration does not satisfy the iteration end condition, the optimization process from internal parameters to external parameters is executed. The k-th camera internal parameters are obtained through the second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively. Optionally, the second fitting algorithm supports reducing the reprojection error, that is, optimizing the camera internal parameters so that the optimized camera internal parameters are closer to the camera internal parameters when the fourth image is captured.

[0201] The (k + 1)-th camera external parameters of the camera are obtained through the first fitting algorithm according to the k-th camera internal parameters, the world coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively (that is, the optimization process from internal parameters to external parameters is executed), k is a positive integer, and k is greater than or equal to 2. Optionally, the first optimization algorithm is the SolvePNP algorithm, and the SolvePNP algorithm supports reducing the reprojection error, that is, optimizing the camera external parameters so that the optimized camera external parameters are closer to the camera external parameters when the first image is captured.

[0202] Optionally, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points.

[0203] In one embodiment, the iteration end condition includes that the number of iterations reaches the number threshold and / or the reprojection error of multiple feature points is less than the threshold. The reprojection error refers to the mean square error between the coordinates of the feature points on the imaging plane and the detected image coordinates. Optionally, the reprojection error of multiple feature points refers to the mean value of the reprojection errors of multiple feature points.

[0204] Step 1050, when the k-th iteration satisfies the iteration end condition, the k-th camera external parameters and the (k - 1)-th camera internal parameters are determined as the calibration result of the camera;

[0205] When the k-th round of iteration meets the iteration end condition, the multi-round iteration process ends, and the k-th camera extrinsic parameters and the (k - 1)-th camera intrinsic parameters are determined as the calibration results of the camera for capturing the first image. The (k - 1)-th camera intrinsic parameters are the camera intrinsic parameters optimized according to the (k - 1)-th camera extrinsic parameters during the (k - 1)-th round of iteration. When k is equal to 2, the (k - 1)-th camera extrinsic parameters used to fit the (k - 1)-th camera intrinsic parameters are the initial camera extrinsic parameters.

[0206] In summary, during the k-th round of iteration, according to the world coordinates corresponding to multiple feature points and the k-th camera extrinsic parameters, the k-th camera intrinsic parameters are obtained through the second fitting algorithm. According to the world coordinates and image coordinates corresponding to multiple feature points, combined with the k-th camera intrinsic parameters, the (k + 1)-th camera extrinsic parameters are obtained through the first fitting algorithm, and the (k + 1)-th camera extrinsic parameters are used to perform the (k + 1)-th round of iteration; wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points, that is, the first fitting algorithm supports optimizing the camera intrinsic parameters, and / or, the second fitting algorithm supports optimizing the camera extrinsic parameters.

[0207] Through the multi-round iteration process, the camera intrinsic parameters and the camera extrinsic parameters continuously approach the true values. By adopting the above method, the disadvantages of the related technology that it can only be applied to wide-angle lenses, two or three groups of parallel lines in three-dimensional space need to be strictly parallel, and it can only calibrate the horizontal focal length f x and the vertical focal length f y of the same camera lens are overcome. The camera calibration method provided by the present application has a wider usage scenario, that is, it has stronger universality.

[0208] Moreover, the method provided by the present application does not limit the posting method of the feature pattern in three-dimensional space, overcoming the defect that the related technology must strictly ensure parallelism. Moreover, the method provided by the present application not only supports the ordinary camera lenses supported by the related technology, but also supports wide-screen lenses (lenses with different horizontal focal lengths fx and vertical focal lengths fy). Moreover, the method provided by the present application can not only calibrate wide-angle zoom lenses, but also support the calibration of large zoom lenses (such as lenses zooming from wide-angle to telephoto).

[0209] Based on Figure 10 In the optional embodiment shown, in step 1040, "According to the camera coordinates corresponding to multiple feature points and the image coordinates corresponding to multiple feature points, the k-th camera intrinsic parameters are obtained through the second fitting algorithm", includes: for any one of the multiple feature points, based on the camera coordinates of the feature point and the image coordinates of the feature point, a fitting equation corresponding to the feature point is constructed; the least squares method is used to solve the fitting equations corresponding to multiple feature points respectively, and the k-th camera intrinsic parameters are obtained.

[0210] Schematically, the matrix structure of the camera intrinsic parameters is

[0211] Among them, f x represents the scaling coefficient of the x-axis scaling that occurs from the imaging plane to the pixel plane multiplied by the focal length f (abbreviated as the horizontal focal length), f y represents the scaling coefficient of the y-axis scaling that occurs from the imaging plane to the pixel plane multiplied by the focal length (abbreviated as the vertical focal length), c x represents the x-axis offset when the feature point on the imaging plane is mapped to the pixel plane, c y represents the y-axis offset when the feature point on the imaging plane is mapped to the pixel plane.

[0212] For any feature point, the following fitting equation can be constructed:

[0213]

[0214] Among them, (x, y, z) are the camera coordinates of the feature point, and (u, v) are the image coordinates of the feature point.

[0215] In one embodiment, for the fitting equations corresponding to multiple feature points, the least squares method can be used to solve for f x , f y , c x and c y , and then the camera internal parameters can be calculated.

[0216] In one embodiment, using the least squares method, with the (k - 1)th camera internal parameters as the constraint terms, combined with the fitting equations corresponding to multiple feature points, the kth camera internal parameters are solved;

[0217] Among them, the (k - 1)th camera internal parameters are used to constrain to obtain a unique solution for the kth camera internal parameters when an infinite number of solutions are obtained by the least squares method.

[0218] Schematically, the f of the kth camera internal parameters is calculated using the following formula x , f y , c x and c y .

[0219]

[0220] Among them, x1 to x n represent the x-axis components of the camera coordinates of n feature points (the horizontal axis components of the camera coordinate system), and y1 to y n represent the y-axis components of the camera coordinates of n feature points (the vertical axis components of the camera coordinate system); u1 to u n represent the u-axis components of the image coordinates of n feature points (the horizontal axis components of the image coordinate system); v1 to v n represent the v-axis components of the image coordinates of n feature points (the vertical axis components of the image coordinate system). The f representing the internal parameters of the (k - 1)-th camera x , The f representing the internal parameters of the (k - 1)-th camera y , The c representing the internal parameters of the (k - 1)-th camera x , The c representing the internal parameters of the (k - 1)-th camera y . Wherein, λ is a proportionality coefficient hyperparameter.

[0221] In summary, the above provides a second fitting algorithm to support the iterative process of obtaining the internal parameters of a camera from the external parameters of the camera. The second fitting algorithm also uses the internal parameters of the j-th camera as a constraint term to ensure that when the camera plane of the camera is parallel to the plane of the feature pattern, the least squares method can obtain a unique solution.

[0222] Based on Figure 10 the optional embodiment shown, Figure 10 shows the process of performing multiple rounds of iterative processes on the fourth image. To achieve real-time continuous calibration of the camera, the external parameters of the camera obtained by calibrating the fourth image are also determined as the initial external parameters of the camera when calibrating the fifth image to complete the continuous calibration of the camera. The fifth image is a subsequent frame image of the fourth image. Optionally, the fifth image is the next frame image of the fourth image. Figure 11 shows a schematic diagram of a camera calibration method provided by an exemplary embodiment of the present application. Steps 1010 to 1050 show the calibration method of the camera that captures the fourth image, corresponding to Figure 11 the third calibration stage 1101 in Figure 11 the fourth calibration stage 1102 in

[0223] Obtain the fifth image captured by the camera for the feature pattern in the three-dimensional space. The fifth image is a subsequent frame image of the fourth image; use the internal parameters of the camera calibrated under the fourth image as the initial internal parameters of the camera for performing multiple rounds of iterative processes on the fifth image; based on the initial internal parameters of the fifth image, obtain the calibration result of the camera under the fifth image by performing multiple rounds of iterative processes under the fifth image;

[0224] After calibrating the camera that captures the fourth image, the camera that captures the fifth image will also be calibrated. The fifth image is a subsequent frame image of the fourth image. Optionally, the fifth image is the next frame image of the fourth image. The changes in the internal parameters and / or external parameters of the camera during the shooting process are coherent. The change in the internal parameters of the camera can be simply understood as the camera has been focused, and the change in the external parameters of the camera can be simply understood as the position of the camera has changed.

[0225] Taking the camera calibration result of the previous frame image as the initial calibration result for the current frame to perform multiple rounds of iteration process helps to achieve fast iteration. It can be understood that since the change process of the internal and external camera parameters is continuous, the real-time calibration method provided in this embodiment cleverly uses the calibration result of the previous frame as the iteration starting point of the current frame, making use of the continuity of the camera change between the previous frame and the current frame, which helps to quickly solve the calibration result of the current frame. In contrast, the real-time calibration in the related art performs separate calibration for each frame, with slow and inaccurate real-time calibration. Compared with the related art, the calibration effect of this embodiment has been significantly improved.

[0226] Based on Figure 10 In the optional embodiment shown, there are at least the following two possible implementation manners for the execution order between step 1030 and step 1050.

[0227] The first possible implementation manner, schematically, with reference to Figure 12 , for any round of iteration in the multiple-round iteration process 900, determine multiple feature points in the fourth image. According to the world coordinates 901 and the camera external parameters 903 respectively corresponding to the multiple feature points, calculate the camera coordinates 904 respectively corresponding to the multiple feature points. Project the camera coordinates 904 respectively corresponding to the multiple feature points onto the imaging plane of the camera to obtain the reprojection coordinates 906 respectively corresponding to the multiple feature points; according to the reprojection coordinates 906 respectively corresponding to the multiple feature points and the image coordinates respectively corresponding to the multiple feature points, calculate the reprojection error of the multiple feature points; determine whether the reprojection error is less than the threshold; schematically, calculate the reprojection error using the following formula:

[0228]

[0229] where ε p is the reprojection error of the multiple feature points, P p is the reprojection coordinates of the multiple feature points, P i is the image coordinates of the multiple feature points, and N is the number of feature points.

[0230] In the case where the reprojection error is less than the threshold, determine that the iteration end condition is satisfied; or, in the case where the reprojection error is less than the threshold and the number of iterations reaches the number threshold, determine that the iteration end condition is satisfied;

[0231] If the iteration end condition is not satisfied, according to the image coordinates 902 respectively corresponding to the multiple feature points and the camera coordinates 904 respectively corresponding to the multiple feature points, obtain the camera internal parameters 905 through the second fitting algorithm. According to the camera internal parameters 905, the image coordinates 902 respectively corresponding to the multiple feature points, and the world coordinates 901 respectively corresponding to the multiple feature points, obtain the new camera external parameters through the first fitting algorithm. The new camera external parameters are used to perform the next round of iteration.

[0232] If the iteration end condition is satisfied, the current camera internal parameters and camera external parameters are used as the iteration results of the multi-round iteration process, and the current camera internal parameters and camera external parameters are output.

[0233] The second possible implementation manner, schematically, in combination with reference Figure 13 , for any round of iteration in the multi-round iteration process 900, it is determined whether the iteration end condition is satisfied. When the number of iterations reaches the number threshold, it is determined that the iteration end condition is satisfied.

[0234] If the iteration end condition is not satisfied, a plurality of feature points in the fourth image are determined. According to the camera external parameters 903 and the world coordinates 901 corresponding to the plurality of feature points respectively, the camera coordinates 904 corresponding to the plurality of feature points are calculated. According to the camera coordinates 904 corresponding to the plurality of feature points and the image coordinates 902 corresponding to the plurality of feature points respectively, through the second fitting algorithm, the camera internal parameters 905 are obtained. According to the camera internal parameters 905, the image coordinates 902 corresponding to the plurality of feature points and the world coordinates 901 corresponding to the plurality of feature points respectively, through the first fitting algorithm, new camera external parameters are obtained. The new camera external parameters are used to perform the next round of iteration.

[0235] If the iteration end condition is satisfied, the current camera internal parameters and camera external parameters are used as the iteration results of the multi-round iteration process, and the current camera internal parameters and camera external parameters are output.

[0236] Based on Figure 10 In the optional embodiment shown, step 1010 includes step S3.

[0237] Step S3, obtaining a fourth image, where the fourth image is obtained by the camera photographing a first plane and a second plane in a three-dimensional space, the fourth image shows all or part of the pictures of the first plane and the second plane, the first plane and the second plane intersect and each has at least one feature pattern, and one feature pattern includes a plurality of feature points.

[0238] Schematically, the three-dimensional space includes the ground (the first plane) and the wall surface (the second plane). At least one feature pattern is posted on the ground, and at least one feature pattern is posted on the wall surface. The fourth image is an image obtained by the camera photographing the ground and the wall surface. At this time, the fourth image includes all or part of the pictures of the ground and the wall surface.

[0239] In one embodiment, the feature pattern includes a QR code pattern. The vertices of the QR code pattern correspond to the feature points. Multiple QR code patterns on the first plane in the three-dimensional space are in a non-overlapping state, and multiple QR code patterns on the second plane in the three-dimensional space are in a non-overlapping state. It should be noted that the camera calibration method provided in this embodiment supports any state other than the non-overlapping state. This embodiment does not limit the spacing, rotation angle, etc. of multiple QR code patterns.

[0240] Based on Figure 10 In the optional embodiment shown, "obtaining the world coordinates corresponding to the multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space" in step 1020 includes step S4.

[0241] Step S4: Based on multiple first feature points on the first plane and multiple second feature points on the second plane in the three-dimensional space, establish the world coordinate system of the three-dimensional space; determine the world coordinates corresponding to the multiple feature points in the fourth image in the world coordinate system respectively.

[0242] Step S4 is used to implement the three-dimensional reconstruction of the three-dimensional space, that is, to determine the world coordinates of multiple feature points in the three-dimensional space. For the specific three-dimensional reconstruction method, please refer to the detailed introduction related to Figure 7 the above (i.e., steps S2-1 to S2-5).

[0243] In summary, a method for three-dimensional reconstruction (i.e., establishing the world coordinate system of the three-dimensional space) is provided to support obtaining the world coordinates of multiple feature points in the fourth image.

[0244] The introduction of the camera calibration method has been completed above. Through camera calibration, the virtual-real fusion of the real space and the virtual space can be realized. In one application mode, through the three-dimensional reconstruction of the real space, a reconstructed virtual three-dimensional space is obtained. Technicians can add virtual elements such as virtual objects and virtual items that do not exist in the real space through computer vision technology in the virtual three-dimensional space. At this time, camera calibration plays a role of mutual mapping between the virtual space and the real space. In another application mode, after constructing the virtual space, the real elements in the real space are mapped into the virtual space to achieve virtual-real fusion. Schematically, Figure 14 shows implanting a person in the real world into the virtual space.

[0245] In the process of virtual-real fusion, camera calibration plays the role of mutual mapping between the virtual space and the real space. The camera takes real-time shots of the real space. According to the world coordinate system of the real space and the camera parameters, the image coordinate system can be obtained. This application provides a method for calibrating the camera parameters of a real camera. There is also a virtual camera model in the virtual space. The virtual camera model takes real-time shots of the virtual space. Through the camera parameters of the virtual camera model and the world coordinate system of the virtual space, it can be mapped to the image coordinate system. By aligning the two image coordinate systems, virtual-real fusion can be achieved. Since the focal length and position of the camera in the real space may change at any time, real-time continuous calibration of the camera is required.

[0246] In one application mode, the camera calibration method provided by this application is applied to the application scenario of virtual production. Compared with the completely live-action shooting method, virtual-real production can conveniently replace the scene, greatly reducing the cost of scene layout. Virtual-real production only requires a green screen, while the cost of setting up a professional venue for live-action shooting is huge. Moreover, virtual production can provide very cool environmental effects, such as Figure 15 as shown Figure 15 shows implanting a real person into a virtual scene in the virtual production scenario. Virtual-real fusion also highly coincides with concepts such as Virtual Reality (VR), the metaverse, and the Omniverse, and can provide the very basic ability to implant real people into virtual scenes for them.

[0247] Figure 16 shows the flowchart of the camera calibration method provided by an exemplary embodiment of this application. This method mainly includes the following four stages:

[0248] Posting QR codes 1601: Post at least one QR code on the ground and wall of the three-dimensional space respectively;

[0249] 3D reconstruction of QR codes 1602: This stage is used to obtain the world coordinates of multiple corner points of the QR code;

[0250] QR code detection 1603: For each frame of each captured video, perform QR code detection to obtain the image coordinates of multiple corner points of the QR code;

[0251] Iterative continuous calibration algorithm 1604: For each frame, based on the <image coordinates of QR code corner points> obtained from QR code detection and the <world coordinates of QR code corner points> obtained from 3D reconstruction of QR codes, calculate the real-time calibration result through the iterative continuous calibration algorithm.

[0252] For the stage of posting QR codes 1601:

[0253] Generate multiple QR code panels through a QR code automatic generation algorithm. Each QR code panel includes 3 * 4 QR code patterns, and all the QR codes are different from each other. Print multiple QR code panels and paste them on the flat walls and floors in a three-dimensional space. Paste more than one QR code panel on the floor or wall.

[0254] For the 3D reconstruction stage of QR codes 1602:

[0255] Combined reference Figure 17 , Figure 17 shows a flowchart of 3D reconstruction.

[0256] Calibrate the internal parameters 1701: Calibrate the lens through the Zhang-Zhengyou calibration algorithm.

[0257] Take a photo 1702: Take an image that can simultaneously capture all the QR code panels in the three-dimensional space.

[0258] QR code detection 1703: Perform QR code detection on the captured image.

[0259] SolvePNP algorithm 1704: For each QR code panel i, use the SolvePNP algorithm to calculate its panel pose. The specific method is:

[0260] From the results of QR code detection, extract all the <image coordinates of QR code corner points> that belong to QR code panel i. Taking QR code panel i as the center, establish a local coordinate system, and construct the 3D coordinates of the detectable QR code corner points in this coordinate system. Use the SolvePNP algorithm, substitute the 3D coordinates and the image coordinates of the QR code corner points, and calculate the transformation matrix Mi (i.e., the panel pose) of the camera relative to QR code panel i.

[0261] Calculate the world coordinates of the QR code corner points:

[0262] Select a panel on the ground as the reference, establish a world coordinate system, and the 3D coordinates of the QR codes in the local coordinate system of the ground panel are the world coordinates. Denote the transformation matrix of this ground panel as Mf (ground pose);

[0263] For other non-reference QR code panels, calculate the transformation matrix Mti that transforms their respective panel local coordinate systems to the world coordinate system.

[0264] M ti = M f -1 · M i ;

[0265] The 3D coordinates of all the QR codes in their respective local coordinate systems of the panels are transformed to the world coordinate system using the transformation matrix Mti. Denote the homogeneous 3D coordinates of the QR code panel i in the local coordinate system as P, and the homogeneous world coordinates in the world coordinate system as Q. Then:

[0266] Q = M ti P;

[0267] Therefore, the world coordinates of all the QR code panels can be obtained.

[0268] For the QR code detection stage 1603:

[0269] For each video stream that needs to be calibrated continuously in real time, for each frame of the image, QR code detection is performed.

[0270] For the iterative continuous calibration algorithm stage 1604:

[0271] The overall steps are as Figure 18 shown. Specifically, its processing steps include:

[0272] Initialize the internal parameters 1801: An initial value can be given, or the internal camera parameters obtained in the aforementioned 3D reconstruction process can be used, or the initial camera internal parameters can be obtained directly by conversion according to the readings of the lens. It should be noted that the internal camera parameters used for initialization at this time do not need to be very accurate, and it is sufficient to be approximately the same as the standard value (such as Zhang Zhengyou).

[0273] For each frame of the image, the following processing is performed frame by frame to calculate its internal / external parameters for each frame:

[0274] Use the SolvePNP algorithm 1802: Perform QR code detection on the image to obtain the image coordinates Pi of the QR code corners. Initialize the iteration count n to 0. For the QR codes that can be detected on the image, extract the world coordinates Pw of the QR code corners, combine with the already initialized camera internal parameters I, and use the SolvePNP algorithm to calculate the external parameters of the camera, that is, the external parameter matrix, denoted as M.

[0275] Calculate the camera coordinates of the QR code corners 1803: Use the external parameter matrix M to transform the world coordinates of the QR code corners to the camera coordinate system to obtain the camera coordinates Pc of the QR code corners.

[0276] Calculate the reprojection coordinates of the QR code corners 1804: Project the camera coordinates Pc of the QR code corners onto the imaging plane to obtain the reprojection coordinates Pp of the QR code corners. Then calculate the mean square error between the reprojection coordinates Pp of the QR code corners and the image coordinates Pi of the QR code corners, that is, the reprojection error ε p , where N is the number of detected QR code corners, and the iteration count n is incremented by 1.

[0277]

[0278] If the number of iterations is greater than a given threshold Tn and / or the reprojection error ε p is less than a given threshold Te, stop the iteration; otherwise, proceed as follows:

[0279] Fitting the internal camera parameters 1805: For the points that are not on the imaging plane after reprojection, they are not used, that is, the corner points where the x-axis is not in the range of 0 to the image width and the y-axis is not in the range of 0 to the image height are discarded and not used. Denote the camera coordinates Pc of the QR code corner points as (x y z), the QR code image coordinates as (u v), and the four internal camera parameters to be calculated as fx, fy, cx, cy. Then there are:

[0280]

[0281] According to the above formula, the least squares method can be used to fit the values of fx, fy, cx, and cy.

[0282] It should be noted that, to ensure a solution, the fx, fy, cx, and cy before fitting can be used as constraint terms to ensure that normal fitting can still be performed when there are infinitely many solutions in the least squares method. The specific method is to denote the fx, fy, cx, and cy before fitting as where λ is a proportionality coefficient hyperparameter, and the recommended value is 0.001, E is the identity matrix, and f is calculated using the following formula x 、f y 、c x 、c y :

[0283]

[0284] After obtaining the internal parameters, update the values of the internal parameters and jump to continue processing using the SolvePNP algorithm 1802.

[0285] Figure 19 Fig. shows a calibration device for a camera provided by an exemplary embodiment of the present application. The device includes:

[0286] An acquisition module 1901, configured to acquire a first image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0287] The acquisition module 1901 is further configured to acquire the world coordinates corresponding to multiple feature points in the first image in the world coordinate system in the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively;

[0288] A processing module 1902, configured to, for the j-th iteration in a multi-round iteration process, obtain the j-th camera external parameter of the camera through a first fitting algorithm according to the j-th camera internal parameter of the camera, the world coordinates corresponding to multiple feature points, and the image coordinates corresponding to the multiple feature points respectively; when j is equal to 1, the j-th camera internal parameter is the initial camera internal parameter, and j is a positive integer;

[0289] The processing module 1902 is further configured to calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system respectively according to the j-th camera external parameter and the world coordinates corresponding to the multiple feature points respectively;

[0290] The processing module 1902 is further configured to, when the j-th iteration does not meet the iteration end condition, obtain the (j + 1)-th camera internal parameter through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively, and the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th iteration in the multi-round iteration process;

[0291] An output module 1903, configured to, when the j-th iteration meets the iteration end condition, determine the j-th camera internal parameter and the j-th camera external parameter as the calibration result of the camera;

[0292] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

[0293] In an optional embodiment, the processing module 1902 is further configured to, for any one of the multiple feature points, construct a fitting equation corresponding to the feature point based on the camera coordinate and the image coordinate of the feature point; and use the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the (j + 1)-th camera internal parameter.

[0294] In an optional embodiment, the processing module 1902 is further configured to use the least squares method, take the j-th camera internal parameter as a constraint term, and combine the fitting equations corresponding to the multiple feature points respectively to solve for the (j + 1)-th camera internal parameter; wherein, the j-th camera internal parameter is used to constrain to obtain a unique solution of the (j + 1)-th camera internal parameter by fitting in the case of obtaining an infinite number of solutions through the least squares method.

[0295] In an optional embodiment, an acquisition module 1901 is further configured to acquire a second image obtained by the camera photographing a feature pattern in a three-dimensional space, and the second image is a subsequent frame image of the first image;

[0296] In an optional embodiment, the processing module 1902 is further configured to use the camera internal parameter of the camera calibrated under the first image as the initial camera internal parameter for the second image to perform the multi-round iteration process;

[0297] In an alternative embodiment, the processing module 1902 is further configured to obtain the calibration result of the camera under the second image by performing multiple rounds of iterative processes under the second image based on the initial camera internal parameters of the second image.

[0298] In an alternative embodiment, the iteration end condition is that the reprojection errors of multiple feature points are less than a threshold. The processing module 1902 is further configured to project the camera coordinates corresponding to the multiple feature points onto the imaging plane of the camera to obtain the reprojection coordinates corresponding to the multiple feature points respectively; calculate the reprojection errors of the multiple feature points based on the reprojection coordinates and the image coordinates corresponding to the multiple feature points respectively; and determine whether the reprojection errors are less than the threshold.

[0299] In an alternative embodiment, the acquisition module 1901 is further configured to acquire a first image, which is obtained by the camera photographing a first plane and a second plane in a three-dimensional space. The first image shows all or part of the pictures of the first plane and the second plane. The first plane and the second plane intersect and each has at least one feature pattern, and one feature pattern includes multiple feature points.

[0300] In an alternative embodiment, the acquisition module 1901 is further configured to establish a world coordinate system of the three-dimensional space based on multiple first feature points of the first plane and multiple second feature points of the second plane in the three-dimensional space; and determine the world coordinates corresponding to the multiple feature points of the first image in the world coordinate system respectively.

[0301] In an alternative embodiment, the acquisition module 1901 is further configured to acquire a third image obtained by the camera photographing a first plane and a second plane in a three-dimensional space. The third image shows all the pictures of the first plane and the second plane;

[0302] For the first plane, detect the first image coordinates corresponding to the multiple first feature points of the first plane on the third image; establish a first local coordinate system based on the first plane; determine the first three-dimensional coordinates corresponding to the multiple first feature points in the first local coordinate system respectively; and obtain a first transformation matrix based on the first three-dimensional coordinates and the first image coordinates corresponding to the multiple first feature points respectively. The first transformation matrix is used to represent the plane attitude of the first plane;

[0303] For the second plane, detect the second image coordinates corresponding to the multiple second feature points of the second plane on the third image; establish a second local coordinate system based on the second plane; determine the second three-dimensional coordinates corresponding to the multiple second feature points in the second local coordinate system respectively; and obtain a second transformation matrix based on the second three-dimensional coordinates and the third image coordinates corresponding to the multiple second feature points respectively. The second transformation matrix is used to represent the plane attitude of the second plane;

[0304] Taking the first local coordinate system as the world coordinate system and the first three-dimensional coordinate as the world coordinate, the world coordinates corresponding to multiple first feature points are obtained;

[0305] Dividing the second transformation matrix by the first transformation matrix to obtain a third transformation matrix, where the third transformation matrix is used to represent the transformation matrix for transforming the second local coordinate system into the first local coordinate system; multiplying the second three-dimensional coordinates corresponding to multiple second feature points by the third transformation matrix to obtain the world coordinates corresponding to multiple second feature points respectively.

[0306] In an optional embodiment, the feature pattern includes a QR code pattern, the vertices of the QR code pattern correspond to the feature points, multiple QR code patterns on the first plane in the three-dimensional space are in a non-overlapping state, and multiple QR code patterns on the second plane in the three-dimensional space are in a non-overlapping state.

[0307] In summary, in the j-th iteration process, according to the world coordinates and image coordinates corresponding to multiple feature points respectively, combined with the j-th camera internal parameter, the j-th camera external parameter is obtained through the first fitting algorithm; according to the world coordinates corresponding to multiple feature points respectively and the j-th camera external parameter, the (j + 1)-th camera internal parameter is obtained through the second fitting algorithm, and the (j + 1)-th camera internal parameter is used for the (j + 1)-th iteration. Among them, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of multiple feature points, that is, the first fitting algorithm supports optimizing the camera internal parameter, and / or, the second fitting algorithm supports optimizing the camera external parameter.

[0308] Through multiple iteration processes, the camera internal parameter and the camera external parameter continuously approach the true values. Using the above device overcomes the disadvantages of the related technology that it can only be applicable to wide-angle lenses, two or three groups of parallel lines in the three-dimensional space need to be strictly parallel, and it can only calibrate the horizontal focal length f x and the vertical focal length f y of the same camera lens. The camera calibration device provided by the present application has a wider usage scenario, that is, it has stronger universality.

[0309] Figure 20 The calibration device of a camera provided by an exemplary embodiment of the present application is shown, and the device includes:

[0310] An acquisition module 2001, configured to acquire a fourth image obtained by the camera photographing a feature pattern in a three-dimensional space;

[0311] The acquisition module 2001 is further configured to acquire the world coordinates and the image coordinates in the image coordinate system corresponding to multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space respectively;

[0312] A processing module 2002 is configured to, for the k-th iteration in a multi-round iteration process, calculate the camera coordinates of multiple feature points in the camera coordinate system according to the k-th extrinsic camera parameters of the camera and the world coordinates corresponding to the multiple feature points respectively; k is greater than or equal to 2 and k is a positive integer.

[0313] The processing module 2002 is further configured to, when the iteration end condition is not satisfied in the k-th iteration, obtain the k-th intrinsic camera parameters through a second fitting algorithm according to the camera coordinates and the image coordinates corresponding to the multiple feature points respectively; obtain the (k + 1)-th extrinsic camera parameters of the camera through a first fitting algorithm according to the k-th intrinsic camera parameters, the world coordinates corresponding to the multiple feature points respectively, and the image coordinates corresponding to the multiple feature points respectively, and the (k + 1)-th extrinsic camera parameters are used to perform the (k + 1)-th iteration in the multi-round iteration process.

[0314] An output module 2003 is configured to, when the iteration end condition is satisfied in the k-th iteration, determine the k-th extrinsic camera parameters and the (k - 1)-th intrinsic camera parameters as the calibration result of the camera; when k is equal to 2, the (k - 1)-th extrinsic camera parameters used to fit the (k - 1)-th intrinsic camera parameters are the initial extrinsic camera parameters.

[0315] Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

[0316] In an optional embodiment, the processing module 2002 is further configured to, for any one of the multiple feature points, construct a fitting equation corresponding to the feature point based on the camera coordinates and the image coordinates of the feature point; use the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the k-th intrinsic camera parameters.

[0317] In an optional embodiment, the processing module 2002 is further configured to use the least squares method, take the (k - 1)-th intrinsic camera parameters as a constraint term, and combine the fitting equations corresponding to the multiple feature points respectively to solve for the k-th intrinsic camera parameters; wherein, the (k - 1)-th intrinsic camera parameters are used to constrain the unique solution of the k-th intrinsic camera parameters obtained by fitting in the case of obtaining an infinite number of solutions through the least squares method.

[0318] In an optional embodiment, an acquisition module 2001 is further configured to acquire a fifth image obtained by the camera photographing a feature pattern in a three-dimensional space, and the fifth image is a subsequent frame image of the fourth image.

[0319] In an optional embodiment, the processing module 2002 is further configured to use the extrinsic camera parameters of the camera calibrated under the fourth image as the initial extrinsic camera parameters for the fifth image to perform the multi-round iteration process.

[0320] In an optional embodiment, the processing module 2002 is further configured to obtain the calibration result of the camera under the fifth image by performing multiple rounds of iterative processes under the fifth image based on the initial extrinsic camera parameters of the fifth image.

[0321] In an optional embodiment, the iteration end condition is that the reprojection errors of multiple feature points are less than a threshold. In an optional embodiment, the processing module 2002 is further configured to project the camera coordinates corresponding to the multiple feature points onto the imaging plane of the camera to obtain the reprojection coordinates corresponding to the multiple feature points respectively; calculate the reprojection errors of the multiple feature points according to the reprojection coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively; and determine whether the reprojection errors are less than the threshold.

[0322] In summary, in the k-th round of iterative process, the k-th camera intrinsic parameters are obtained through the second fitting algorithm according to the world coordinates corresponding to the multiple feature points respectively and the k-th extrinsic camera parameters. According to the world coordinates and image coordinates corresponding to the multiple feature points respectively, and in combination with the k-th camera intrinsic parameters, the (k + 1)-th extrinsic camera parameters are obtained through the first fitting algorithm, and the (k + 1)-th extrinsic camera parameters are used to perform the (k + 1)-th round of iteration; wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection errors of the multiple feature points, that is, the first fitting algorithm supports optimizing the camera intrinsic parameters, and / or, the second fitting algorithm supports optimizing the camera extrinsic parameters.

[0323] Through multiple rounds of iterative processes, the camera intrinsic parameters and the camera extrinsic parameters continuously approach the true values. By using the above device, the disadvantages of the related technology that it can only be applied to wide-angle lenses, two or three groups of parallel lines in three-dimensional space need to be strictly parallel, and it can only calibrate the horizontal focal length f x and the vertical focal length f y of the same camera lens are overcome. The camera calibration device provided by the present application has a wider usage scenario, that is, it has stronger universality.

[0324] Figure 21It is a schematic structural diagram of a computer device shown according to an exemplary embodiment. The computer device 2100 includes a Central Processing Unit (CPU) 2101, a system memory 2104 including a Random Access Memory (RAM) 2102 and a Read-Only Memory (ROM) 2103, and a system bus 2105 connecting the system memory 2104 and the central processing unit 2101. The computer device 2100 may also include a basic Input / Output (I / O) system 2106 for facilitating information transmission between various components within the computer device, and a mass storage device 2107 for storing an operating system 2113, application programs 2114, and other program modules 2115.

[0325] In some embodiments, the basic I / O system 2106 includes a display 2108 for displaying information and input devices 2109 such as a mouse, keyboard, etc. for user input of information. Both the display 2108 and the input devices 2109 are connected to the central processing unit 2101 through an input / output controller 2110 connected to the system bus 2105. The basic I / O system 2106 may also include an input / output controller 2110 for receiving and processing inputs from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 2110 also provides output to a display screen, printer, or other types of output devices.

[0326] The mass storage device 2107 is connected to the central processing unit 2101 through a mass storage controller (not shown) connected to the system bus 2105. The mass storage device 2107 and its associated computer-readable medium provide non-volatile storage for the computer device 2100. That is to say, the mass storage device 2107 may include computer-readable media (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0327] Without loss of generality, the computer device-readable medium may include a computer device storage medium and a communication medium. The computer device storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device-readable instructions, data structures, program modules, or other data. The computer device storage medium includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD), or other optical storage, magnetic tape cartridges, magnetic tapes, disk storage, or other magnetic storage devices. Of course, those skilled in the art will know that the computer device storage medium is not limited to the above several types. The above-mentioned system memory 2104 and mass storage device 2107 can be collectively referred to as memory.

[0328] According to various embodiments of the present disclosure, the computer device 2100 may also operate by connecting to a remote computer device on a network such as the Internet. That is, the computer device 2100 may be connected to the network 2111 through the network interface unit 2112 connected to the system bus 2105. Or rather, the network interface unit 2112 may also be used to connect to other types of networks or remote computer device systems (not shown).

[0329] The memory further includes one or more programs. The one or more programs are stored in the memory, and the central processing unit 2101 implements all or part of the steps of the above-mentioned camera calibration method by executing the one or more programs.

[0330] This application also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the camera calibration method provided in the above method embodiments. This application provides a computer program product. The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the camera calibration method provided in the above method embodiments.

[0331] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0332] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a disk, an optical disc, etc.

[0333] The above are only alternative embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A calibration method for a camera, characterized in that, The method includes: Obtaining a first image captured by the camera for a feature pattern in a three-dimensional space; Obtaining the world coordinates corresponding to multiple feature points in the first image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively; For the j-th iteration in multiple rounds of iteration, according to the j-th camera internal parameter of the camera, the world coordinates corresponding to the multiple feature points respectively, and the image coordinates corresponding to the multiple feature points respectively, obtaining the j-th camera external parameter of the camera through a first fitting algorithm; when j is equal to 1, the j-th camera internal parameter is the initial camera internal parameter, and j is a positive integer; Calculating the camera coordinates corresponding to the multiple feature points in the camera coordinate system respectively according to the j-th camera external parameter and the world coordinates corresponding to the multiple feature points respectively; In the case where the j-th iteration does not meet the iteration end condition, obtaining the (j + 1)-th camera internal parameter through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively, and the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th iteration in the multiple rounds of iteration; In the case where the j-th iteration meets the iteration end condition, determining the j-th camera internal parameter and the j-th camera external parameter as the calibration result of the camera; Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

2. The method according to claim 1, characterized in that, The obtaining the (j + 1)-th camera internal parameter through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively includes: For any one of the multiple feature points, constructing a fitting equation corresponding to the feature point based on the camera coordinate of the feature point and the image coordinate of the feature point; Using the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the (j + 1)-th camera internal parameter.

3. The method according to claim 2, characterized in that, The using the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the (j + 1)-th camera internal parameter includes: Using the least squares method, taking the j-th camera internal parameter as a constraint term, and combining the fitting equations corresponding to the multiple feature points respectively to solve and obtain the (j + 1)-th camera internal parameter; Wherein, the j-th camera internal parameter is used to constrain to fit a unique solution of the (j + 1)-th camera internal parameter in the case of obtaining an infinite set of solutions through the least squares method.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtaining a second image captured by the camera for the feature pattern in the three-dimensional space, and the second image is a subsequent frame image of the first image; Taking the camera internal parameter of the camera calibrated under the first image as the initial camera internal parameter for the second image to perform multiple rounds of iteration; Based on the initial camera internal parameter of the second image, obtaining the calibration result of the camera under the second image by performing multiple rounds of iteration under the second image.

5. The method according to any one of claims 1 to 3, characterized in that The iteration end condition is that the reprojection error of the multiple feature points is less than a threshold, and the method further includes: Project the camera coordinates corresponding to the multiple feature points onto the imaging plane of the camera to obtain the reprojection coordinates corresponding to the multiple feature points respectively; Calculate the reprojection errors of the multiple feature points based on the reprojection coordinates corresponding to the multiple feature points respectively and the image coordinates corresponding to the multiple feature points respectively.

6. The method according to any one of claims 1 to 3, characterized in that The obtaining the first image captured by the camera for a feature pattern in a three-dimensional space includes: Obtain the first image, which is captured by the camera for a first plane and a second plane in the three-dimensional space. The first image shows all or part of the images of the first plane and the second plane. The first plane and the second plane intersect and each has at least one feature pattern, and one feature pattern includes multiple feature points; The obtaining the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system corresponding to the three-dimensional space respectively includes: Based on multiple first feature points on the first plane and multiple second feature points on the second plane in the three-dimensional space, establish the world coordinate system of the three-dimensional space; determine the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system respectively.

7. The method according to claim 6, characterized in that, Based on multiple first feature points on the first plane and multiple second feature points on the second plane in the three-dimensional space, establish the world coordinate system of the three-dimensional space; Determining the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system respectively includes: Obtain a third image captured by the camera for the first plane and the second plane in the three-dimensional space. The third image shows all the images of the first plane and the second plane; For the first plane, detect the first image coordinates corresponding to the multiple first feature points on the first plane in the third image; based on the first plane, establish a first local coordinate system; determine the first three-dimensional coordinates corresponding to the multiple first feature points in the first local coordinate system respectively; according to the first three-dimensional coordinates and the first image coordinates corresponding to the multiple first feature points respectively, obtain a first transformation matrix, and the first transformation matrix is used to represent the plane attitude of the first plane; For the second plane, detect the second image coordinates corresponding to the multiple second feature points on the second plane in the third image; based on the second plane, establish a second local coordinate system; determine the second three-dimensional coordinates corresponding to the multiple second feature points in the second local coordinate system respectively; according to the second three-dimensional coordinates and the third image coordinates corresponding to the multiple second feature points respectively, obtain a second transformation matrix, and the second transformation matrix is used to represent the plane attitude of the second plane; Take the first local coordinate system as the world coordinate system, and take the first three-dimensional coordinates as the world coordinates to obtain the world coordinates corresponding to the multiple first feature points respectively; Divide the second transformation matrix by the first transformation matrix to obtain a third transformation matrix, where the third transformation matrix is used to represent the transformation matrix for transforming from the second local coordinate system to the first local coordinate system; multiply the second three-dimensional coordinates corresponding to the multiple second feature points by the third transformation matrix to obtain the world coordinates corresponding to the multiple second feature points respectively.

8. The method according to claim 6, wherein The feature pattern includes a two-dimensional code pattern, the vertices of the two-dimensional code pattern correspond to the feature points, multiple two-dimensional code patterns on the first plane in the three-dimensional space are in a non-overlapping state, and multiple two-dimensional code patterns on the second plane in the three-dimensional space are in a non-overlapping state.

9. A calibration method for a camera, characterized in that, The method includes: Obtain a fourth image captured by the camera for a feature pattern in a three-dimensional space; Obtain the world coordinates corresponding to the multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system respectively; For the k-th iteration in multiple rounds of iteration, calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system according to the k-th extrinsic camera parameters of the camera and the world coordinates corresponding to the multiple feature points respectively; k is greater than or equal to 2 and k is a positive integer; When the k-th iteration does not meet the iteration end condition, obtain the k-th intrinsic camera parameters through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively; obtain the (k + 1)-th extrinsic camera parameters of the camera through a first fitting algorithm according to the k-th intrinsic camera parameters, the world coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively, and the (k + 1)-th extrinsic camera parameters are used to perform the (k + 1)-th iteration in the multiple rounds of iteration; When the k-th iteration meets the iteration end condition, determine the k-th extrinsic camera parameters and the (k - 1)-th intrinsic camera parameters as the calibration result of the camera; when k is equal to 2, the (k - 1)-th extrinsic camera parameters used to fit the (k - 1)-th intrinsic camera parameters are the initial extrinsic camera parameters; Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

10. The method according to claim 9, wherein The obtaining the k-th intrinsic camera parameters through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively includes: For any one of the multiple feature points, construct a fitting equation corresponding to the feature point based on the camera coordinate of the feature point and the image coordinate of the feature point; Use the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the k-th intrinsic camera parameters.

11. The method according to claim 10, wherein The using the least squares method to solve the fitting equations corresponding to the multiple feature points respectively to obtain the k-th intrinsic camera parameters includes: Use the least squares method, take the (k - 1)-th intrinsic camera parameters as a constraint term, and combine the fitting equations corresponding to the multiple feature points respectively to solve for the k-th intrinsic camera parameters; Among them, the (k - 1)-th camera internal parameter is used to constrain the unique solution of the k-th camera internal parameter obtained by fitting in the case of obtaining an infinite number of solutions through the least squares method.

12. The method according to any one of claims 9 to 11, characterized in that, The method further includes: Obtaining a fifth image obtained by the camera photographing a feature pattern in the three-dimensional space, where the fifth image is a subsequent frame image of the fourth image; Using the external camera parameter of the camera calibrated under the fourth image as the initial external camera parameter for the fifth image to perform multiple rounds of iteration processes; Based on the initial external camera parameter of the fifth image, obtaining the calibration result of the camera under the fifth image by performing multiple rounds of iteration processes under the fifth image.

13. The method according to any one of claims 9 to 11, characterized in that, The iteration end condition is that the reprojection error of the multiple feature points is less than a threshold, and the method further includes: Projecting the camera coordinates corresponding to the multiple feature points onto the imaging plane of the camera to obtain the reprojection coordinates corresponding to the multiple feature points; Calculating the reprojection error of the multiple feature points according to the reprojection coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points.

14. A calibration device for a camera, characterized in that, The device includes: An acquisition module, configured to acquire a first image obtained by the camera photographing a feature pattern in the three-dimensional space; The acquisition module is further configured to acquire the world coordinates corresponding to the multiple feature points in the first image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system; A processing module, configured to, for the j-th round of iteration in multiple rounds of iteration processes, obtain the j-th external camera parameter of the camera through a first fitting algorithm according to the j-th camera internal parameter of the camera, the world coordinates corresponding to the multiple feature points, and the image coordinates corresponding to the multiple feature points; when j is equal to 1, the j-th camera internal parameter is the initial camera internal parameter, and j is a positive integer; The processing module is further configured to calculate the camera coordinates corresponding to the multiple feature points in the camera coordinate system according to the j-th external camera parameter and the world coordinates corresponding to the multiple feature points; The processing module is further configured to, in the case that the j-th round of iteration does not meet the iteration end condition, obtain the (j + 1)-th camera internal parameter through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points, where the (j + 1)-th camera internal parameter is used to perform the (j + 1)-th round of iteration in the multiple rounds of iteration processes; An output module, configured to, in the case that the j-th round of iteration meets the iteration end condition, determine the j-th camera internal parameter and the j-th external camera parameter as the calibration result of the camera; Among them, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

15. A calibration device for a camera, characterized in that, The device includes: An acquisition module, configured to acquire a fourth image obtained by the camera photographing a feature pattern in the three-dimensional space; The acquisition module is further configured to acquire the world coordinates corresponding to the multiple feature points in the fourth image in the world coordinate system corresponding to the three-dimensional space and the image coordinates corresponding to the multiple feature points in the image coordinate system; A processing module, for the k-th iteration in the multi-round iteration process, calculates the camera coordinates corresponding to the multiple feature points in the camera coordinate system according to the k-th extrinsic camera parameters of the camera and the world coordinates corresponding to the multiple feature points respectively; k is greater than or equal to 2 and k is a positive integer; The processing module is further configured to, when the k-th iteration does not meet the iteration end condition, obtain the k-th intrinsic camera parameters through a second fitting algorithm according to the camera coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively; obtain the (k + 1)-th extrinsic camera parameters of the camera through a first fitting algorithm according to the k-th intrinsic camera parameters, the world coordinates corresponding to the multiple feature points and the image coordinates corresponding to the multiple feature points respectively, and the (k + 1)-th extrinsic camera parameters are used to perform the (k + 1)-th iteration in the multi-round iteration process; An output module, for when the k-th iteration meets the iteration end condition, determining the k-th extrinsic camera parameters and the (k - 1)-th intrinsic camera parameters as the calibration result of the camera; when k is equal to 2, the (k - 1)-th extrinsic camera parameters used to fit the (k - 1)-th intrinsic camera parameters are the initial extrinsic camera parameters; Wherein, at least one of the first fitting algorithm and the second fitting algorithm supports reducing the reprojection error of the multiple feature points.

16. A computer device, characterized in that, The computer device includes: a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the camera calibration method according to any one of claims 1 to 8, or the camera calibration method according to any one of claims 9 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by the processor to implement the camera calibration method according to any one of claims 1 to 8, or the camera calibration method according to any one of claims 9 to 13.

18. A computer program product, characterized in that, The computer program product stores a computer program, and the computer program is loaded and executed by the processor to implement the camera calibration method according to any one of claims 1 to 8, or the camera calibration method according to any one of claims 9 to 13.

Images