A camera calibration method, device, computer equipment and storage medium

Abstract

The application discloses a camera calibration method, device, equipment and storage medium, the method comprises the following steps: determining N groups of homonymic points according to a three-dimensional model of a target area and a video image of the target area obtained by a camera to be calibrated; constructing a first equation according to a first relationship and a second relationship, substituting two-dimensional coordinates of a first target feature point and three-dimensional coordinates of a second target feature point in the N groups of homonymic points into the first equation, and determining a rotation matrix and first and second translation parameters of the camera to be calibrated; constructing a second equation according to the second relationship, substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N groups of homonymic points into the second equation, and determining a focal length, a radial distortion parameter and a third translation parameter of the camera to be calibrated; determining an attitude angle of the camera to be calibrated according to the rotation matrix, and determining position information of the camera to be calibrated according to a translation matrix. Camera calibration is realized without setting a hardware calibration facility.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a camera calibration method, apparatus, computer device and storage medium. Background Technology

[0002] Camera calibration is a fundamental step in machine vision applications such as visual measurement and 3D reconstruction. The accuracy and precision of the calibration results directly determine whether the vision system can function properly.

[0003] In the prior art, before the camera leaves the factory, the internal and external parameters of the camera can be calibrated at the calibration station on the production line; after the camera leaves the factory, the camera can be placed in the calibration space, and the external parameters of the camera can be calibrated by using the three-dimensional coordinates of known feature points in the calibration plate set in the calibration space and the corresponding image coordinates on the calibration image.

[0004] In the existing technology, after the camera leaves the factory, it is not possible to achieve common calibration of the camera's intrinsic parameters, and it is necessary to calibrate the camera's extrinsic parameters in the calibration space, which results in high calibration costs. Summary of the Invention

[0005] This invention provides a camera calibration method, apparatus, device, and storage medium that enables the calibration of camera intrinsic and extrinsic parameters without the need for hardware calibration facilities.

[0006] In a first aspect, embodiments of the present invention provide a camera calibration method, including:

[0007] N sets of corresponding points are determined based on the 3D model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the 3D model aligned with the video image, and N is an integer greater than or equal to 5;

[0008] A first equation is constructed based on the first and second relational expressions. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter. The first relational expression is used to determine the camera coordinates based on the world coordinates, and the second relational expression is used to determine the pixel coordinates based on the camera coordinates.

[0009] Based on the second relation, construct a second equation, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated.

[0010] The attitude angle of the camera to be calibrated is determined according to the rotation matrix, and the position information of the camera to be calibrated is determined according to the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

[0011] The technical solution of this invention provides a camera calibration method, comprising: determining N sets of corresponding points based on a three-dimensional model of a target region and a video image of the target region acquired by a camera to be calibrated; wherein the corresponding points include a first target feature point in the video image and a second target feature point in a three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5; constructing a first equation based on a first relation and a second relation; substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as a first translation parameter and a second translation parameter. Translation parameters; wherein, the first relation is used to determine camera coordinates based on world coordinates, and the second relation is used to determine pixel coordinates based on camera coordinates; a second equation is constructed based on the second relation, and the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the second equation to determine the focal length, radial distortion parameters, and third translation parameters of the camera to be calibrated; the attitude angle of the camera to be calibrated is determined based on the rotation matrix, and the position information of the camera to be calibrated is determined based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter, and the third translation parameter. The above technical solution obtains at least five sets of corresponding points from the video image of the target area acquired by the camera to be calibrated and the 3D model aligned with the video image. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the first equation constructed by the first relational expression for determining camera coordinates based on world coordinates and the second relational expression for determining pixel coordinates based on camera coordinates, the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated can be determined. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the second equation constructed by the second relational expression, the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated can be determined. By determining the focal length and radial distortion parameter of the camera to be calibrated, the intrinsic parameter calibration of the camera to be calibrated can be achieved. Furthermore, the position information and attitude angle of the camera to be calibrated can be determined based on the rotation matrix and translation matrix, thus achieving the extrinsic parameter calibration of the camera to be calibrated. Therefore, camera calibration can be achieved without setting up hardware calibration facilities.

[0012] Furthermore, the first relation is: in, Let (X,Y,Z) represent the rotation matrix, (x,y,z) represent the world coordinates, (x,y,z) represent the camera coordinates, and tx,ty represent the first and second translation parameters contained in the translation matrix.

[0013] Furthermore, the second relation is: Where (u,v) represents pixel coordinates, tz represents the third translation parameter contained in the translation matrix, f represents focal length, and k1,k2…kn represent radial distortion parameters.

[0014] Furthermore, the first equation is -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0.

[0015] Furthermore, the second equation is

[0016] Furthermore, before determining N sets of corresponding points based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated, the process further includes:

[0017] Image data of the target area is acquired using the camera sensor mounted on the UAV, and a three-dimensional model of the target area is generated based on the image data; image data of the target area is acquired using the camera to be calibrated, and a video image of the target area is determined based on the number of images; the three-dimensional model is adjusted to be aligned with the video image.

[0018] Further, based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated, N sets of corresponding points are determined, including:

[0019] After determining the image cache based on the 3D model aligned with the size and viewpoint of the video image, a first set of feature points is extracted from the video image, and a second set of feature points is extracted from the image cache. The first set of feature points and the second set of feature points are matched to obtain initial point pairs. Noise is removed from the initial point pairs to obtain N sets of target point pairs, wherein each target point pair includes a first target feature point and a second target feature point that correspond to each other. The 2D coordinates of the first target feature point in the video image are determined, and the 3D coordinates of the second target feature point in the 3D model are determined. Based on the 2D coordinates of the first target feature point and the 3D coordinates of the second target feature point, N sets of corresponding points are determined.

[0020] Secondly, embodiments of the present invention also provide a camera calibration device, comprising:

[0021] The corresponding point determination module is used to determine N sets of corresponding points based on the three-dimensional model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include a first target feature point in the video image and a second target feature point in the three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5;

[0022] The first parameter determination module is used to construct a first equation based on a first relation and a second relation, and to substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter; wherein, the first relation is used to determine the camera coordinates based on the world coordinates, and the second relation is used to determine the pixel coordinates based on the camera coordinates;

[0023] The second parameter determination module is used to construct a second equation based on the second relational expression, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameter and third translation parameter of the camera to be calibrated.

[0024] The third parameter determination module is used to determine the attitude angle of the camera to be calibrated based on the rotation matrix and to determine the position information of the camera to be calibrated based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

[0025] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the camera calibration method as described in any of the first aspects.

[0026] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the camera calibration method as described in any of the first aspects.

[0027] Fifthly, this application provides a computer program product including computer instructions that, when executed on a computer, cause the computer to perform the camera calibration method as provided in the first aspect.

[0028] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the camera calibration device, or it may be packaged separately from the processor of the camera calibration device; this application does not impose any limitations on this.

[0029] The descriptions of the second, third, fourth, and fifth aspects in this application can be referred to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referred to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0030] In this application, the name of the aforementioned camera calibration device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this application, it falls within the scope of the claims of this application and its equivalents.

[0031] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart of a camera calibration method provided in an embodiment of the present invention;

[0034] Figure 2 A flowchart illustrating another camera calibration method provided in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a camera calibration device provided in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0038] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0039] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0040] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0041] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc. Moreover, embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0042] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0044] In existing technologies, camera calibration methods may include: 1) pre-calibrating the camera based on a calibration board, calibration field, or calibration frame before camera installation to determine the intrinsic parameter matrix, and then interpolating the camera after installation to update the intrinsic parameter matrix; 2) updating the focal length, intrinsic parameter matrix, and extrinsic parameters based on the motion model. Among the existing camera calibration methods, 1) the process of fabricating a calibration board, calibration field, or calibration frame is cumbersome and costly, applicable only to cameras that are not yet installed, and cannot calibrate installed cameras; furthermore, it can only calibrate the camera's intrinsic parameters; 2) it can only calibrate the positional parameters among the camera's intrinsic and extrinsic parameters.

[0045] Therefore, this application proposes a camera calibration method to calibrate the camera's intrinsic and extrinsic parameters without the need for hardware calibration facilities.

[0046] Figure 1 This is a flowchart illustrating a camera calibration method provided in an embodiment of the present invention. This embodiment is applicable to situations where camera calibration is not required to utilize hardware calibration facilities. The method can be executed by a camera calibration device, such as... Figure 1 As shown, the specific steps include the following:

[0047] Step 110: Determine N sets of corresponding points based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated.

[0048] Wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5.

[0049] The drone uses its camera sensors to capture images of the target area, obtaining photographic data. This data is then processed to obtain model data. This model data is loaded into a 3D rendering engine and displayed to create a 3D model of the target area. Alternatively, the drone can use a camera to capture images of the target area, obtaining corresponding image data. This image data is then loaded into a 3D rendering engine and displayed to create a video image of the target area. After determining the video image and 3D model of the target area, the viewpoint and / or size of the 3D model can be adjusted until they align with the size and viewpoint of the video image.

[0050] Specifically, N sets of corresponding points can be determined in the video image and the 3D model aligned with the size and viewpoint of the video image. That is, N first target feature points are determined in the video image, and N second target feature points are determined in the 3D model. The first target feature points and the second target feature points match each other.

[0051] In practical applications, the first and second target feature points can be easily identifiable locations, such as house corners or roadside edges within the target area. Of course, to accelerate the calibration process while maintaining accuracy, at least five sets of corresponding points can be determined in the video image and the 3D model aligned with it.

[0052] In this embodiment of the invention, N sets of first target feature points and second target feature points, i.e., N sets of corresponding points, can be determined in the video image and the three-dimensional model aligned with the size and viewpoint of the video image.

[0053] Step 120: Construct a first equation based on the first and second relational expressions, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first and second translation parameters.

[0054] The first relation is used to determine camera coordinates based on world coordinates, and the second relation is used to determine pixel coordinates based on camera coordinates. The first relation can be: The second relation can be... Let (x, y, z) represent the rotation matrix, (x, y, z) represent the world coordinates, (x, y, z) represent the camera coordinates, (tx, ty, tz) represent the translation matrix, where tx represents the first translation parameter, ty represents the second translation parameter, tz represents the third translation parameter, (u, v) represent the pixel coordinates, f represents the focal length, k1, k2…kn represent the radial distortion parameters, and r represents the distance from (u, v) to the image center. Additionally, the camera intrinsic parameters include the focal length and radial distortion parameters, while the camera extrinsic parameters include the camera's position information and attitude angles.

[0055] Specifically, by performing a cross product on the first two rows of the first and second relations, r7, r8, and r9 can be determined. Combining the first two rows of the first and second relations yields the first equation: -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0. It is known that the first equation has eight unknowns. Therefore, the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the eight sets of corresponding points can be substituted into the first equation. The two-dimensional coordinates can be pixel coordinates (u,v), and the three-dimensional coordinates can be world coordinates (X,Y,Z). Therefore, by substituting the eight sets of (u,v) and (X,Y,Z) into the first equation, r1, r2, r3, r4, r5, r6, tx, and ty can be determined. This allows the determination of the rotation matrix of the camera to be calibrated. The first translation parameter tx and the second translation parameter ty in the translation matrix of the camera to be calibrated can be determined.

[0056] In this embodiment of the invention, a first equation is constructed by using a first relational expression for determining camera coordinates based on world coordinates and a second relational expression for determining pixel coordinates based on camera coordinates. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point among the previously determined corresponding points are substituted into the first equation to determine the first translation parameter and the second translation parameter in the translation matrix of the camera to be calibrated.

[0057] Step 130: Construct a second equation based on the second relational expression, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated.

[0058] The second equation is:

[0059] Specifically, regarding the second relation... By transforming the equation, we can obtain the second equation. It is known that the second equation has n+2 unknowns. Therefore, by substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the n+2 sets of corresponding points into the second equation, similarly, the two-dimensional coordinates can be pixel coordinates (u,v), and the three-dimensional coordinates can be camera coordinates (x,y,z). Therefore, by substituting the n+2 sets of (u,v) and (X,Y,Z) into the second equation, k1,k2,...,kn, f, and tz can be determined. Furthermore, the focal length f of the camera to be calibrated, the radial distortion parameters k1,k2,...,kn, and the third translation parameter tz in the translation matrix can be determined, thus determining the translation matrix (tx,ty,tz) of the calibrated camera.

[0060] In practical applications, the radial distortion parameters can include k1, k2, and k3; therefore, the second equation has five unknowns. Substituting the two-dimensional coordinates (u, v) of the first target feature point and the three-dimensional coordinates (x, y, z) of the second target feature point from the five sets of corresponding points into the second equation determines k1, k2, k3, f, and tz. This allows for the determination of the focal length f of the camera to be calibrated, the radial distortion parameters k1, k2, ..., kn, and the third translation parameter tz in the translation matrix, thereby determining the translation matrix (tx, ty, tz) of the calibration camera.

[0061] In this embodiment of the invention, the second relation can be transformed to construct a second equation. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point among the previously determined corresponding points are substituted into the second equation to determine the focal length, radial distortion parameter, and third translation parameter in the translation matrix of the camera to be calibrated.

[0062] Step 140: Determine the attitude angle of the camera to be calibrated based on the rotation matrix, and determine the position information of the camera to be calibrated based on the translation matrix.

[0063] The translation matrix includes a first translation parameter, a second translation parameter, and a third translation parameter.

[0064] Specifically, in determining the rotation matrix of the camera to be calibrated After obtaining the translation matrix T = (tx, ty, tz), the position of the camera to be calibrated, C = -R, can be calculated. T* T,R T Let T denote the transpose of the rotation matrix and T denote the translation matrix. The position of the camera to be calibrated is thus determined. The attitude angles of the camera to be calibrated are determined based on the rotation matrix, using currently common algorithms, which will not be elaborated upon here.

[0065] In this embodiment of the invention, after determining the rotation matrix and translation matrix of the camera to be calibrated, the position information and attitude angle of the camera to be calibrated can be determined based on the rotation matrix and translation matrix, thereby determining the extrinsic parameters of the camera to be calibrated.

[0066] The camera calibration method provided in this embodiment of the invention includes: determining N sets of corresponding points based on a three-dimensional model of a target region and a video image of the target region acquired by the camera to be calibrated; wherein the corresponding points include a first target feature point in the video image and a second target feature point in a three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5; constructing a first equation based on a first relational expression and a second relational expression; substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as a first translation parameter and a second translation parameter. The first relation is used to determine camera coordinates based on world coordinates, and the second relation is used to determine pixel coordinates based on camera coordinates. A second equation is constructed based on the second relation, and the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the second equation to determine the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated. The attitude angle of the camera to be calibrated is determined based on the rotation matrix, and the position information of the camera to be calibrated is determined based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter, and the third translation parameter. The above technical solution obtains at least five sets of corresponding points from the video image of the target area acquired by the camera to be calibrated and the 3D model aligned with the video image. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the first equation constructed by the first relational expression for determining camera coordinates based on world coordinates and the second relational expression for determining pixel coordinates based on camera coordinates, the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated can be determined. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the second equation constructed by the second relational expression, the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated can be determined. By determining the focal length and radial distortion parameter of the camera to be calibrated, the intrinsic parameter calibration of the camera to be calibrated can be achieved. Furthermore, the position information and attitude angle of the camera to be calibrated can be determined based on the rotation matrix and translation matrix, thus achieving the extrinsic parameter calibration of the camera to be calibrated. Therefore, camera calibration can be achieved without setting up hardware calibration facilities.

[0067] Figure 2 A flowchart of another camera calibration method provided by an embodiment of the present invention is shown. This embodiment is a specific embodiment based on the above embodiments. In this embodiment, the method may further include:

[0068] Step 210: Acquire image data of the target area based on the camera sensor mounted on the UAV, and generate a three-dimensional model of the target area based on the image data.

[0069] Specifically, after processing the image data corresponding to the target area acquired by the camera sensors contained in the drone, model data can be obtained. The model data can then be loaded into the 3D rendering engine and displayed on the first interface of the 3D rendering engine to obtain the three-dimensional model corresponding to the target area.

[0070] Step 220: Acquire image data of the target area based on the camera to be calibrated, and determine the video image of the target area based on the image data.

[0071] Image data can be video stream data.

[0072] Specifically, the image data corresponding to the target area acquired by the camera to be calibrated is loaded into the 3D rendering engine, and the image data is displayed in the second interface parallel to the first interface in the 3D rendering engine to obtain the video image corresponding to the target area.

[0073] Step 230: Adjust the 3D model to align with the video image.

[0074] In one implementation, step 230 may specifically include:

[0075] Align 3D models and video images based on adjustment commands triggered by the user in the rendering engine.

[0076] The adjustment command can be used to adjust the size and viewpoint of a 3D model.

[0077] Specifically, the viewpoint and size of the video image displayed in the second interface of the 3D rendering engine are fixed, while the viewpoint and size of the 3D model displayed in the first interface are adjustable. After receiving an adjustment command triggered by the user, the 3D rendering engine places the first interface displaying the 3D model on top of the second interface displaying the video image, and adjusts the viewpoint and / or size of the 3D model according to the adjustment command until the size and viewpoint of the 3D model are nearly identical to the size and viewpoint of the video image, thus aligning the 3D model and the video image.

[0078] In this embodiment of the invention, in the 3D rendering engine, based on the adjustment command triggered by the user, the viewpoint and / or size of the three-dimensional model displayed in the first interface of the 3D rendering engine can be adjusted so that the viewpoint and size of the three-dimensional model are approximately consistent with the viewpoint and size of the video image, and when the viewpoint and size of the three-dimensional model are approximately consistent with the viewpoint and size of the video image, the alignment of the three-dimensional model and the video image is determined.

[0079] Step 240: Determine N sets of corresponding points based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated.

[0080] Wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5.

[0081] In one implementation, step 240 may specifically include:

[0082] After determining the image cache based on the 3D model aligned with the size and viewpoint of the video image, a first set of feature points is extracted from the video image, and a second set of feature points is extracted from the image cache. The first set of feature points and the second set of feature points are matched to obtain initial point pairs. Noise is removed from the initial point pairs to obtain N sets of target point pairs, wherein each target point pair includes a first target feature point and a second target feature point that correspond to each other. The 2D coordinates of the first target feature point in the video image are determined, and the 3D coordinates of the second target feature point in the 3D model are determined. Based on the 2D coordinates of the first target feature point and the 3D coordinates of the second target feature point, N sets of corresponding points are determined.

[0083] Specifically, a first set of feature points is extracted from the video image based on a feature extraction algorithm, and a second set of feature points is extracted from the image cache. A feature matching algorithm is then used to match the first and second set of feature points to obtain an initial set of point pairs. For example, the first set of feature points can be extracted from the video image based on the SIFT feature extraction operator, and the second set of feature points can be extracted from the image cache. Alternatively, the first and second set of feature points can be matched using the SIFT feature matching operator to obtain the initial set of point pairs.

[0084] In the rendering engine, the connecting lines between the first and second initial feature points in each set of initial point pairs are determined, and the angle between the connecting lines and a preset standard line is determined, where the preset standard line is the bottom edge of the first and second interfaces. Each angle is placed in a histogram segmented by angle. Based on the first and second initial feature points corresponding to each angle in the rectangle containing the most angles, the target point pair set is determined. The histogram includes at least one rectangle segmented by angle. For example, the 3D rendering engine can display video images and image caches corresponding to 3D models in two parallel interfaces. To determine whether the first and second initial feature points contained in each initial point pair in the initial point pair set correspond, the connecting lines between the first and second initial feature points in each initial point pair are first determined, and the angle between the connecting lines and the bottom edge of the display interface is determined. The angle distribution is statistically analyzed, and based on the statistical information, initial point pairs consisting of first and second initial feature points with low matching degrees are deleted from the initial point pair set, achieving noise removal of the initial point pair set and obtaining the target point pair set. Each angle can be placed into a histogram interval divided by angle. If the previously determined angle is not in the histogram interval with the most points, it is determined that the first initial feature point and the second initial feature point corresponding to that angle do not match. Then, the initial point pair formed by the first initial feature point and the second initial feature point corresponding to that angle can be deleted to remove noise from the set of initial point pairs.

[0085] In practical applications, firstly, the 360° is divided into 72 equal parts, with each 5° portion serving as a rectangular frame for a histogram, thus establishing the histogram. Secondly, the rectangular frames to which the previously determined angles belong are identified, and the target point pair set is determined based on the first and second initial feature points corresponding to each angle in the rectangular frame containing the most angles. Simultaneously, the first and second initial feature points corresponding to each angle in other rectangular frames are removed, achieving noise removal from the initial point pair set.

[0086] After aligning the 3D model and the video image, the depth cache of each point in the image cache can be determined. The depth cache indicates the distance of each point in the image cache from the camera. The parameters of the camera acquiring the image cache can also be determined, such as the camera's pose angle and position. Based on the depth cache and camera parameters, the 3D coordinates of the second target feature point in the 3D model can be determined. As described in Embodiment 1, when determining the first feature point set based on the video image, the 2D coordinates of each first feature point contained in the first feature point set in the video image can be determined. Therefore, the 2D coordinates of the first target feature point in the video image can be determined. When determining the second feature point set based on the image cache, the pixel coordinates of each second feature point contained in the second feature point set in the image cache can be determined. Therefore, the pixel coordinates of the second target feature point in the image cache can be determined, and then the pixel coordinates of the second target feature point in the image cache can be transformed according to the depth cache and camera parameters to obtain the 3D coordinates of the second target feature point in the 3D model.

[0087] In this embodiment of the invention, the automatic acquisition of the set of corresponding points is realized, which improves the efficiency of acquiring the set of corresponding points. Moreover, since the initial point pairs consisting of mismatched first and second initial feature points are removed from the target point pair set, the accuracy of the corresponding points determined based on the target point pair set is higher.

[0088] Step 250: Construct a first equation based on the first and second relational expressions, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first and second translation parameters.

[0089] The first relation is used to determine camera coordinates based on world coordinates, and the second relation is used to determine pixel coordinates based on camera coordinates. The first relation is... in, Let (X,Y,Z) represent the rotation matrix, (x,y,z) represent the world coordinates, (x,y,z) represent the camera coordinates, and tx,ty represent the first and second translation parameters included in the translation matrix. The second relation is: Where (u,v) represents pixel coordinates, tz represents the third translation parameter contained in the translation matrix, f represents the focal length, and k1,k2…kn represent radial distortion parameters. The first equation is -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0.

[0090] Specifically, by performing a cross product on the first two rows of the first and second relations, r7, r8, and r9 can be determined. Combining the first two rows of the first and second relations yields the first equation: -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0. Transforming the first equation, it can be converted into a matrix form M*s=0, where... M is a factor matrix, and r1, r2, r3, r4, r5, r6, tx, and ty are unknowns to be solved. M*s = 0 is solved by QR decomposition.

[0091] Specifically, M can be determined T = Q*K, where Q represents the orthogonal matrix of the decomposition, and K represents the upper triangular matrix of the decomposition. Since the rank of matrix M is 5, the three right columns of matrix Q form the null space of M (M... T Given a matrix of dimension m*n, m=8, n=5, and the mn columns on the right side of matrix Q are the null space of matrix M), then s can be represented by a linear combination of the null space as: s=n1*a+n2*b+n3(1). n1n2n3 represent the three columns of the null space respectively, and at this time there are only two unknowns, a and b. All elements in s can be represented by ab. Since matrix R is an orthogonal matrix, the matrix row vectors have a modulus of 1 and the row vectors are perpendicular to each other, we can know that r1*r4+r2*r5+r3*r6=0(2), r1 2 +r2 2 +r3 2 =r4 2 +r5 2 +r6 2 (3). Substituting equation (1) into equations (2) and (3), the two equations can solve for two unknowns. Therefore, based on the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point among the five sets of points with the same name, r1, r2, r3, r4, r5, r6, tx and ty can be determined.

[0092] Step 260: Construct a second equation based on the second relational expression, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated.

[0093] The second equation is:

[0094] Specifically, transforming the second equation yields: z + tz = λ(f + k1*r 2 +k2*r 4 +...kn*r 2n )and There are n+2 unknowns. Therefore, the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the n+2 sets of corresponding points are substituted into the second equation. Similarly, the two-dimensional coordinates can be pixel coordinates (u,v), and the three-dimensional coordinates can be camera coordinates (x,y,z). Therefore, the n+2 sets of (u,v) and (x,y,z) can be substituted into the second equation to determine k1,k2,...,kn, f, and tz. This allows us to determine the focal length f of the camera to be calibrated, the radial distortion parameters k1,k2,...,kn, and the third translation parameter tz in the translation matrix, thus determining the translation matrix (tx,ty,tz) of the calibrated camera.

[0095] The radial distortion parameters can include k1, k2, and k3; therefore, the second equation has five unknowns. Substituting the two-dimensional coordinates (u, v) of the first target feature point and the three-dimensional coordinates (x, y, z) of the second target feature point from the five sets of corresponding points into the second equation determines k1, k2, k3, f, and tz. This allows us to determine the focal length, radial distortion parameters, and the third translation parameter in the translation matrix of the camera to be calibrated, thereby determining the translation matrix (tx, ty, tz) of the calibrated camera.

[0096] Step 270: Determine the attitude angle of the camera to be calibrated based on the rotation matrix, and determine the position information of the camera to be calibrated based on the translation matrix.

[0097] The translation matrix includes a first translation parameter, a second translation parameter, and a third translation parameter.

[0098] Step 270 has been described in detail in the aforementioned Embodiment 1, and will not be repeated here.

[0099] The camera calibration method provided in this invention includes: acquiring image data of a target region based on a camera sensor; generating a three-dimensional model of the target region based on the image data; acquiring image data of the target region based on the camera to be calibrated; determining video images of the target region based on the number of images; adjusting the three-dimensional model to align with the video images; determining depth buffers and camera parameters based on the three-dimensional model aligned with the size and viewpoint of the video images; determining N sets of corresponding points based on the three-dimensional model of the target region and the video images of the target region acquired by the camera to be calibrated; constructing a first equation based on a first relation and a second relation, and determining the first target region among the N sets of corresponding points. Substituting the two-dimensional coordinates of the feature point and the three-dimensional coordinates of the second target feature point into the first equation, the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated are determined. Based on the second relational expression, a second equation is constructed. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the N sets of corresponding points into the second equation, the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated are determined. The attitude angle of the camera to be calibrated is determined based on the rotation matrix, and the position information of the camera to be calibrated is determined based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter, and the third translation parameter.The above technical solution, after acquiring the image data and video data corresponding to the target area, can further determine the video image corresponding to the image data and the 3D model corresponding to the video data. Then, the 3D model can be adjusted to align with the video image, and the image cache when the 3D model is aligned with the video image can be determined. Furthermore, the first feature point set extracted from the video image and the second feature point set extracted from the image cache can be matched to determine an initial point pair set. The initial point pair set includes at least one initial point pair, each consisting of corresponding first and second initial feature points. Due to extraction or matching errors, the initial point pair set may contain mismatched initial point pairs consisting of first and second initial feature points. Therefore, noise removal can be performed on the initial point pair set to delete the mismatched initial point pairs, resulting in a target point pair set. The target point pair set includes at least one target point pair, each consisting of corresponding first and second target feature points. This allows for the determination of the first target... By determining the two-dimensional coordinates of the target feature points in the video image and the three-dimensional coordinates of the second target feature points in the three-dimensional model, a set of corresponding points is obtained, which realizes automatic acquisition of the corresponding point set and improves the efficiency of acquiring corresponding points. Moreover, since the initial point pairs formed by the first and second initial feature points that do not match are removed from the target point pair set, the accuracy of the acquired corresponding points is higher. Based on the video image of the target area acquired from the camera to be calibrated and the five sets of corresponding points obtained from the three-dimensional model aligned with the video image, the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated can be determined. Based on the video image of the target area acquired from the camera to be calibrated and the three sets of corresponding points obtained from the three-dimensional model aligned with the video image, the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated can be determined. By determining the focal length and radial distortion parameter of the camera to be calibrated, the intrinsic parameter calibration of the camera to be calibrated can be achieved. Then, the position information and attitude angle of the camera to be calibrated can be determined according to the rotation matrix and translation matrix, thus achieving the extrinsic parameter calibration of the camera to be calibrated. Therefore, camera calibration can be achieved without setting up hardware calibration facilities.

[0100] Figure 3 This is a schematic diagram of a camera calibration device provided in an embodiment of the present invention. This device is applicable to situations where camera calibration is performed without the need for hardware calibration facilities. The device can be implemented through software and / or hardware and is generally integrated into a computer device.

[0101] like Figure 3 As shown, the device includes:

[0102] The corresponding point determination module 310 is used to determine N sets of corresponding points based on the three-dimensional model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include a first target feature point in the video image and a second target feature point in the three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5;

[0103] The first parameter determination module 320 is used to construct a first equation based on a first relation and a second relation, and to substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter; wherein, the first relation is used to determine the camera coordinates based on the world coordinates, and the second relation is used to determine the pixel coordinates based on the camera coordinates;

[0104] The second parameter determination module 330 is used to construct a second equation based on the second relational expression, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameter and third translation parameter of the camera to be calibrated.

[0105] The third parameter determination module 340 is used to determine the attitude angle of the camera to be calibrated according to the rotation matrix and to determine the position information of the camera to be calibrated according to the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

[0106] The camera calibration device provided in this embodiment determines N sets of corresponding points based on a three-dimensional model of the target area and video images of the target area acquired by the camera to be calibrated. The corresponding points include a first target feature point in the video image and a second target feature point in the three-dimensional model aligned with the video image, where N is an integer greater than or equal to 5. A first equation is constructed based on a first relational expression and a second relational expression. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the N sets of corresponding points are substituted into the first equation to determine the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated. In this process, the first relation is used to determine camera coordinates based on world coordinates, and the second relation is used to determine pixel coordinates based on camera coordinates. A second equation is constructed based on the second relation, and the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the second equation to determine the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated. The attitude angle of the camera to be calibrated is determined based on the rotation matrix, and the position information of the camera to be calibrated is determined based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter, and the third translation parameter. The above technical solution obtains at least five sets of corresponding points from the video image of the target area acquired by the camera to be calibrated and the 3D model aligned with the video image. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the first equation constructed by the first relational expression for determining camera coordinates based on world coordinates and the second relational expression for determining pixel coordinates based on camera coordinates, the rotation matrix, first translation parameter, and second translation parameter of the camera to be calibrated can be determined. Substituting the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point from the at least five sets of corresponding points into the second equation constructed by the second relational expression, the focal length, radial distortion parameter, and third translation parameter of the camera to be calibrated can be determined. By determining the focal length and radial distortion parameter of the camera to be calibrated, the intrinsic parameter calibration of the camera to be calibrated can be achieved. Furthermore, the position information and attitude angle of the camera to be calibrated can be determined based on the rotation matrix and translation matrix, thus achieving the extrinsic parameter calibration of the camera to be calibrated. Therefore, camera calibration can be achieved without setting up hardware calibration facilities.

[0107] Preferably, the first relation is in, Let (X,Y,Z) represent the rotation matrix, (x,y,z) represent the world coordinates, (x,y,z) represent the camera coordinates, and tx,ty represent the first and second translation parameters contained in the translation matrix.

[0108] Preferably, the second relation is Where (u,v) represents pixel coordinates, tz represents the third translation parameter contained in the translation matrix, f represents focal length, and k1,k2…kn represent radial distortion parameters.

[0109] Preferably, the first equation is -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0.

[0110] Preferably, the second equation is:

[0111] Based on the above embodiments, the device further includes: a first execution module, used to determine depth buffer and camera parameters based on a three-dimensional model aligned with the size and viewpoint of the video image.

[0112] Based on the above embodiments, the device further includes: a second execution module, configured to acquire image data of the target area based on the camera sensor mounted on the UAV, generate a three-dimensional model of the target area based on the image data; acquire image data of the target area based on the camera to be calibrated, determine a video image of the target area based on the number of images; and adjust the three-dimensional model to be aligned with the video image.

[0113] Based on the above embodiments, the corresponding point determination module 310 is specifically used for: determining an image cache based on a three-dimensional model aligned with the size and viewpoint of the video image; extracting a first set of feature points from the video image; extracting a second set of feature points from the image cache; matching the first set of feature points and the second set of feature points to obtain initial point pairs; removing noise from the initial point pairs to obtain N sets of target point pairs, wherein the target point pairs include corresponding first target feature points and second target feature points; determining the two-dimensional coordinates of the first target feature points in the video image; determining the three-dimensional coordinates of the second target feature points in the three-dimensional model; and determining N sets of corresponding points based on the two-dimensional coordinates of the first target feature points and the three-dimensional coordinates of the second target feature points.

[0114] In one embodiment, determining the three-dimensional coordinates of the second target feature point in the three-dimensional model includes: determining the three-dimensional coordinates of the second target feature point in the three-dimensional model based on the depth cache and the camera parameters.

[0115] The camera calibration device provided in the embodiments of the present invention can execute the camera calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0116] It is worth noting that in the above-described embodiments of the camera calibration device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0117] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary computer device 4 suitable for implementing embodiments of the present invention is shown. Figure 4 The computer device 4 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0118] like Figure 4 As shown, the computer device 4 is represented in the form of a general-purpose computing electronic device. The components of the computer device 4 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0119] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0120] Computer device 4 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 4, including volatile and non-volatile media, removable and non-removable media.

[0121] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 4 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0122] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0123] Computer device 4 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 4, and / or with any device that enables computer device 4 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 4 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 20. Figure 4 As shown, network adapter 20 communicates with other modules of computer device 4 via bus 18. It should be understood that, although... Figure 4 As not shown in the diagram, it can be used in conjunction with computer device 4 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0124] Processing unit 16 executes various functional applications and page displays by running programs stored in system memory 28, such as implementing the camera calibration method provided in this embodiment, which includes:

[0125] N sets of corresponding points are determined based on the 3D model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the 3D model aligned with the video image, and N is an integer greater than or equal to 5;

[0126] A first equation is constructed based on the first and second relational expressions. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter. The first relational expression is used to determine the camera coordinates based on the world coordinates, and the second relational expression is used to determine the pixel coordinates based on the camera coordinates.

[0127] Based on the second relation, construct a second equation, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated.

[0128] The attitude angle of the camera to be calibrated is determined according to the rotation matrix, and the position information of the camera to be calibrated is determined according to the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

[0129] Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the camera calibration method provided in any embodiment of the present invention.

[0130] This invention provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements, for example, the camera calibration method provided in this invention, which includes:

[0131] N sets of corresponding points are determined based on the 3D model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the 3D model aligned with the video image, and N is an integer greater than or equal to 5;

[0132] A first equation is constructed based on the first and second relational expressions. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter. The first relational expression is used to determine the camera coordinates based on the world coordinates, and the second relational expression is used to determine the pixel coordinates based on the camera coordinates.

[0133] Based on the second relation, construct a second equation, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated.

[0134] The attitude angle of the camera to be calibrated is determined according to the rotation matrix, and the position information of the camera to be calibrated is determined according to the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

[0135] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0137] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0138] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0140] Furthermore, the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0141] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A camera calibration method, characterized in that, include: N sets of corresponding points are determined based on the 3D model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include the first target feature point in the video image and the second target feature point in the 3D model aligned with the video image, and N is an integer greater than or equal to 5; A first equation is constructed based on the first and second relational expressions. The two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points are substituted into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter. The first relational expression is used to determine the camera coordinates based on the world coordinates, and the second relational expression is used to determine the pixel coordinates based on the camera coordinates. Based on the second relation, construct a second equation, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameters and third translation parameters of the camera to be calibrated. The attitude angle of the camera to be calibrated is determined according to the rotation matrix, and the position information of the camera to be calibrated is determined according to the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

2. The camera calibration method according to claim 1, characterized in that, The first relation is ,in, Let (X,Y,Z) represent the rotation matrix, (x,y,z) represent the world coordinates, (x,y,z) represent the camera coordinates, and tx,ty represent the first and second translation parameters contained in the translation matrix.

3. The camera calibration method according to claim 2, characterized in that, The second relation is , where (u,v) represents pixel coordinates, tz represents the third translation parameter contained in the translation matrix, f represents focal length, k1,k2…kn represents radial distortion parameters, and r represents the distance from (u,v) to the image center.

4. The camera calibration method according to claim 3, characterized in that, The first equation is -v(r1*X+r2*Y+r3*Z+tx)+u(r4*X+r5*Y+r6*Z+ty)=0. By performing a cross product on the first two rows of the first and second equations, r7, r8, and r9 are determined.

5. The camera calibration method according to claim 3, characterized in that, The second equation is .

6. The camera calibration method according to claim 1, characterized in that, Before determining N sets of corresponding points based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated, the process also includes: Image data of the target area is acquired using the camera sensor mounted on the drone, and a three-dimensional model of the target area is generated based on the image data; Based on the image data of the target area acquired by the camera to be calibrated, a video image of the target area is determined according to the image data; Adjust the 3D model to align with the video image.

7. The camera calibration method according to claim 6, characterized in that, Based on the 3D model of the target area and the video images of the target area acquired by the camera to be calibrated, N sets of corresponding points are determined, including: After determining the image cache based on the 3D model aligned with the size and viewpoint of the video image, a first feature point set is extracted from the video image, a second feature point set is extracted from the image cache, and the first feature point set and the second feature point set are matched to obtain an initial point pair; The initial point pairs are subjected to noise removal to obtain N sets of target point pairs, wherein the target point pairs include corresponding first target feature points and second target feature points; Determine the two-dimensional coordinates of the first target feature point in the video image, determine the three-dimensional coordinates of the second target feature point in the three-dimensional model, and determine N sets of corresponding points based on the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point.

8. A camera calibration device, characterized in that, include: The corresponding point determination module is used to determine N sets of corresponding points based on the three-dimensional model of the target area and the video image of the target area acquired by the camera to be calibrated; wherein, the corresponding points include a first target feature point in the video image and a second target feature point in the three-dimensional model aligned with the video image, and N is an integer greater than or equal to 5; The first parameter determination module is used to construct a first equation based on a first relation and a second relation, and to substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the first equation to determine the rotation matrix of the camera to be calibrated, as well as the first translation parameter and the second translation parameter; wherein, the first relation is used to determine the camera coordinates based on the world coordinates, and the second relation is used to determine the pixel coordinates based on the camera coordinates; The second parameter determination module is used to construct a second equation based on the second relational expression, and substitute the two-dimensional coordinates of the first target feature point and the three-dimensional coordinates of the second target feature point in the N sets of corresponding points into the second equation to determine the focal length, radial distortion parameter and third translation parameter of the camera to be calibrated. The third parameter determination module is used to determine the attitude angle of the camera to be calibrated based on the rotation matrix and to determine the position information of the camera to be calibrated based on the translation matrix, wherein the translation matrix includes the first translation parameter, the second translation parameter and the third translation parameter.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the camera calibration method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the camera calibration method as described in any one of claims 1-7.

Images