Camera calibration system and method

Abstract

The present application discloses a camera calibration system and method. The method is applied to electronic equipment, and the method includes: obtaining multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each target image, superimposing the multiple target images to obtain a target composite image, determining the image coordinates of the marking point in the target composite image, and determining the world coordinates of the marking point in the virtual target; fitting the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera; optimizing the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera, and then when calibrating the camera, it is not necessary to move the target, but only to collect images during the rotation of the rotating mechanism, so as to reduce the difficulty of camera calibration and complete the fast and high-precision calibration of the camera in a smaller calibration space.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and more specifically, to a camera calibration system and method. Background Art

[0002] Camera calibration is one of the key technologies in machine vision, photogrammetry, 3D imaging, and image geometry correction. Its main function is to estimate the internal and external parameters of the camera. The accuracy of the calibration results and the stability of the calibration algorithm directly affect the accuracy of subsequent work. In general calibration methods, since multiple images in various postures need to be collected, it is necessary to manually move the calibration plate or camera, and the distance between the target and the camera during calibration must also meet the working distance of the camera. In practical applications, this is not only time-consuming and labor-intensive, but also increases the production cost of the target. Summary of the invention

[0003] In view of this, the present application proposes a camera calibration system and method to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a camera calibration system, the system comprising a target carrier, a teleconverter, a camera, a rotating mechanism, a servo driver and an electronic device. The teleconverter is disposed between the camera and the target carrier, and the camera and the servo driver are electrically connected to the electronic device. The target carrier comprises a plane, the plane is used to form a target, and the target comprises at least one marking point. The camera is mounted on the rotating mechanism, the camera is used to photograph a virtual target formed after the target is magnified by the teleconverter, and obtain a target image, and the target image comprises the marking point in the virtual target. The rotating mechanism comprises two rotating axes, and the optical axis of the camera is coaxial with one of the rotating axes. The servo driver is used to drive the rotating mechanism to rotate, and send the position information of the rotating mechanism to the electronic device. The electronic device is used to obtain multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each of the target images, superimpose the multiple target images to obtain a target composite image, determine the image coordinates of the marking point in the target composite image according to the target composite image, determine the world coordinates of the marking point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter, fit the image coordinates and the world coordinates according to the perspective imaging model of the camera, obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera, and optimize the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera.

[0005] In a second aspect, an embodiment of the present application provides a camera calibration method, the method comprising: obtaining a plurality of target images captured by the camera during the rotation of the rotating mechanism and position information of the rotating mechanism corresponding to each of the target images, the target image being a virtual target formed by capturing the target and magnifying it through the teleconverter, the target image comprising a marking point in the virtual target; superimposing the plurality of target images to obtain a target composite image, and determining the image coordinates of the marking point in the target composite image according to the target composite image; determining the world coordinates of the marking point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter; fitting the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain initial values ​​of the focal length, principal point and distortion parameters of the camera; and optimizing the initial values ​​of the focal length, principal point and distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera.

[0006] The camera calibration system and method provided in the embodiments of the present application obtain multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each target image, superimpose the multiple target images to obtain a target composite image, determine the image coordinates of the marking point in the target composite image, and determine the world coordinates of the marking point in the virtual target; fit the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera; optimize the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera, so that when calibrating the camera, there is no need to move the target, only to collect images during the rotation of the rotating mechanism, thereby reducing the difficulty of camera calibration and completing the fast and high-precision calibration of the camera in a smaller calibration space. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0008] Figure 1 One of the schematic diagrams of the camera calibration system provided in an embodiment of the present application is shown;

[0009] Figure 2 A second schematic diagram of a camera calibration system provided in an embodiment of the present application is shown;

[0010] Figure 3 A schematic diagram of the structure of the rotating structure provided in an embodiment of the present application is shown;

[0011] Figure 4 A schematic diagram of a target provided by an embodiment of the present application is shown;

[0012] Figure 5 A schematic diagram of a target provided by another embodiment of the present application is shown;

[0013] Figure 6 A schematic diagram of a target provided in yet another embodiment of the present application is shown;

[0014] Figure 7-Figure 8 A schematic diagram of a target provided in other embodiments of the present application is shown;

[0015] Fig. 9 A schematic diagram of a virtual sphere provided by an embodiment of the present application is shown;

[0016] Fig.10 A schematic diagram of a virtual sphere provided by another embodiment of the present application is shown;

[0017] Fig.11 A schematic diagram of the projection of a virtual image of a marking point provided in an embodiment of the present application is shown;

[0018] Fig.12 A schematic diagram of a camera calibration method according to an embodiment of the present application is shown;

[0019] Fig.13 A schematic diagram showing coordinates provided by an embodiment of the present application is shown;

[0020] Fig.14 A structural block diagram of a camera calibration device provided by an embodiment of the present application is shown;

[0021] Fig.15 A structural block diagram of an electronic device for executing a camera calibration method according to an embodiment of the present application is shown;

[0022] Fig.16 A storage medium provided by an embodiment of the present application for storing or carrying a program code for implementing a camera calibration method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0024] Camera calibration is one of the key technologies in machine vision, photogrammetry, 3D imaging, and image geometry correction. Its main function is to estimate the internal and external parameters of the camera. The accuracy of the calibration result and the stability of the calibration algorithm directly affect the accuracy of subsequent work. Generally, the perspective camera can be represented by a pinhole model, and can be calibrated using perspective projection mapping and affine transformation.

[0025] The method that has been studied maturely is to use a plane target. There are tool software based on plane targets, such as Matlab toolbox and Opencv tool software. In these methods, a plane calibration plate is placed in different positions in front of the camera to collect multiple target images to obtain calibration raw data with a large distribution range. This method requires placing the calibration plate in different positions multiple times and collecting target images, or rotating the camera multiple times in different positions and collecting target images, so as to achieve camera calibration.

[0026] At present, in the process of calibrating a camera, it is necessary to consider the working distance of the calibrated camera during normal use. When calibrating a camera, the distance between the target and the camera should be consistent with the working distance of the camera during normal use, so as to ensure that the calibration result is accurate and reliable. The inventor found in the study that some cameras have a longer working distance during normal use, such as vehicle-mounted cameras. When calibrating a camera with a longer working distance, it is necessary to control the distance between the camera and the target to the working distance of the camera during normal use. Due to the longer working distance, the size of the target is larger, which not only increases the production cost of the target, but also increases the difficulty of calibration.

[0027] In view of the above problems, the inventors have discovered after long-term research and proposed the camera calibration system and method provided in the embodiments of the present application, so that when calibrating the camera, there is no need to move the target, and only needs to collect images during the rotation of the rotating mechanism, which reduces the difficulty of camera calibration and completes the rapid and high-precision calibration of the camera in a smaller calibration space. The specific camera calibration method is described in detail in the subsequent embodiments.

[0028] The following describes a camera calibration method that can be used in an embodiment of the present application.

[0029] See also Figure 1 and Figure 2, an embodiment of the present application provides a camera calibration system. Specifically, the system may include a target carrier 100, a teleconverter 200, a camera 300, a rotating mechanism 400, a servo driver 500 and an electronic device 600. The teleconverter 200 is disposed between the camera 300 and the target carrier 100, and the camera 300 and the servo driver 500 are electrically connected to the electronic device 600, respectively. The camera 300 may include a camera with a longer working distance, such as a vehicle-mounted camera, which is not limited here.

[0030] The target carrier 100 includes a plane, which is used to form a target, and the target includes at least one marking point.

[0031] The camera 300 is mounted on the rotating mechanism 400 through the camera fixing bracket (not shown), and the camera 300 is used to shoot the virtual target formed after the target is magnified by the teleconverter to obtain a target image, and the target image includes the marking points in the virtual target. The rotating mechanism 400 includes two rotating axes, and the optical axis of the camera 300 is coaxial with one of the rotating axes. The servo driver 500 is used to drive the rotating mechanism 400 to rotate and send the position information of the rotating mechanism 400 to the electronic device 600. The electronic device 600 analyzes and processes the acquired target image and the position information of the rotating mechanism 400 corresponding to the target image, determines the imaging model parameters of the camera 300, and thereby realizes the calibration of the camera 300.

[0032] See also Figure 3 , showing a schematic diagram of a rotating structure provided in an embodiment of the present application. The rotating mechanism 400 includes a carrying platform 410, a first rotating mechanism 420, a second rotating mechanism 430 and a camera fixing frame. The first rotating mechanism 420 has a first rotating axis 421, and the second rotating mechanism 430 has a second rotating axis 431. Specifically, the second rotating mechanism 430 is installed on the carrying platform 410, and the second rotating axis 431 is perpendicular to the carrying platform 410. The first rotating mechanism 420 is fixed to the second rotating mechanism 430, and the first rotating axis 421 is perpendicular to the second rotating axis 431 and parallel to the carrying platform 410. The optical center of the camera 300 coincides with the intersection of the first rotating axis 421 and the second rotating axis 431, and the optical axis of the camera 300 is coaxial with the first rotating axis 421.

[0033] As a method, the first rotating mechanism 420 can be directly fixed to the second rotating mechanism 430, wherein the first rotating mechanism 420 can be fixed to the second rotating mechanism by in-mold injection molding. The first rotating mechanism 420 can also be fixed to the second rotating mechanism 430 by screws, and the specific fixing method is not limited here.

[0034] As another embodiment, the rotating mechanism 400 may further include a connecting bracket 450, and the first rotating mechanism 420 and the second rotating mechanism 430 are fixed to the connecting bracket 450. Specifically, the connecting bracket 450 includes a horizontal bracket 452 and a vertical bracket 451 that are perpendicular to each other, the horizontal bracket 452 and the vertical bracket 451 are fixedly connected, the first rotating mechanism 420 is fixed to the vertical bracket 451, and the second rotating mechanism 430 is connected between the horizontal bracket 452 and the carrying platform 410.

[0035] Specifically, the horizontal bracket 452 and the vertical bracket 451 can be combined into the connecting bracket 450 by means of in-mold injection molding. The horizontal bracket 452 and the vertical bracket 451 can also be connected and combined into the connecting bracket 450 by means of screws.

[0036] The vertical bracket 451 may have a groove matching the first rotating mechanism 420, and the first rotating mechanism 420 is fixed to the vertical bracket 451 through the groove, and the horizontal bracket 452 may have a groove matching the second rotating mechanism 430, and the second rotating mechanism 430 is fixed to the horizontal bracket 452 through the groove. As another way, the vertical bracket 451 may have a spiral groove having the same size as the first rotating mechanism 420, and the first rotating mechanism 420 may be fixed to the vertical bracket 451 by screws, and the horizontal bracket 452 may have a spiral groove having the same size as the second rotating mechanism 430, and the second rotating mechanism 430 may be fixed to the horizontal bracket 452 by screws.

[0037] The camera mounting bracket is fixed to the first rotating mechanism 420 and is used to mount the camera 300. The camera mounting bracket is configured so that when the camera 300 is mounted on the camera mounting bracket, the intersection of the center lines of the first rotating axis 421 and the second rotating axis 431 coincides with the optical center of the camera 300, and the optical axis of the camera 300 is coaxial with the first rotating axis 421.

[0038] Specifically, the camera fixing frame is fixed to the first rotating mechanism 420. In some embodiments, the first rotating mechanism 420 may have a groove adapted to the camera fixing frame, and the groove is used to fix the camera fixing frame to the first rotating mechanism 120. In some embodiments, the camera fixing frame and the first rotating mechanism 420 both have spiral grooves of the same size, and the camera fixing frame can be fixed to the first rotating mechanism 420 by screws. In some embodiments, the camera fixing frame and the first rotating mechanism 420 can also be fixed by in-mold injection.

[0039] The camera fixing frame may be a camera clip, through which the camera is fixed on the camera fixing frame. The camera fixing frame may also be a groove, and the groove is provided with a knob for adjusting the size of the groove, and the size of the groove is adjusted by the knob so that the groove is adapted to the size of the camera, and the camera is mounted on the camera fixing frame by snapping the camera into the groove.

[0040] Please attend Figure 2 In the application embodiment, the servo driver 500 may be one or two, and the servo driver 500 may be driven to rotate the second rotating mechanism 430 within a preset range while driving the first rotating mechanism 420 to rotate one circle, or the first rotating mechanism 420 and the second rotating mechanism 430 may be driven to rotate separately. For example, after driving the first rotating mechanism 420 to rotate one circle, the second rotating mechanism 430 is driven to rotate, or the second rotating mechanism 430 is driven to rotate first, and then the first rotating mechanism 420 is driven to rotate.

[0041] The servo driver 500 can send the rotation angle φ of the first rotating mechanism 420 and the rotation angle θ of the second rotating mechanism 430 as position information to the electronic device 600. The preset amplitude is determined by the field of view of the camera under test, for example, it can be ± half of the field of view angle with the second rotating axis 431 as the central axis direction.

[0042] Furthermore, the rotating mechanism 400 also includes a first rotary encoder installed on the first rotating mechanism 420 and a second rotary encoder installed on the second rotating mechanism 430, the first rotary encoder and the second rotary encoder are both electrically connected to the servo driver 500, the first rotary encoder is used to obtain the rotational angle position of the first rotating mechanism 420, and the second rotary encoder is used to obtain the rotational angle position of the second rotating mechanism 430.

[0043] Furthermore, the plane of the target carrier 100 can be used to carry a physical target, or to display a target image, or to carry an electronic chart card displaying a target pattern.

[0044] Furthermore, the target carrier 100 includes a plane, and the plane is used to form a target 110. The target includes at least one marking point, and the marking point may include but is not limited to a circle, a grid, a BMW pattern, a chessboard, and a ChArUco, or a combination of several of them. Figure 4 , Figure 4 A schematic diagram of a target provided by an embodiment of the present application is shown, wherein the target 110 displays circular spot marking points 111A, and the circular spot marking points 111A may be one or more, and may be arranged in rows, columns, or a matrix, without limitation. Figure 5 , Figure 5 Schematic diagram of another target provided by an embodiment of the present application is shown, wherein the target 110 displays the marking points 111B in the shape of a square, which can also be arranged in rows, columns, or a matrix, without limitation. Figure 6 , Figure 6 A schematic diagram of another target provided in an embodiment of the present application is shown, wherein the target 110 displays a field-shaped marking point 111C and a circular spot 112C.

[0045] It is understood that the shape and number of the marking points displayed by the target 110 can also be Figure 7 Any one or a combination of the figures shown in a, b, c, d, e, and f, or Figure 8 The graphics of the chessboard shown are not limited here.

[0046] In the embodiment of the present application, the center of the target 110 and the center of the teleconverter 200 are on the same reference line, and the optical axis of the teleconverter 200 is perpendicular to the plane of the target carrier 100. The distance between the teleconverter 200 and the target carrier 100 is less than the focal length of the teleconverter 200, so the target 110 can be magnified by the teleconverter 200 to form a virtual target that is magnified relative to the target. And the distance of the virtual target relative to the teleconverter 200 is greater than the focal length of the teleconverter 200, wherein the size of the virtual target is larger than the target, so that a clear target image of the virtual target can be obtained.

[0047] The camera 300 rotates with the first rotating mechanism 420 and the second rotating mechanism 430, which is equivalent to the camera 300 not rotating. The target image rotates around the camera 300 in the direction of rotation of the first rotating mechanism 420 and the direction of rotation of the second rotating mechanism 430. The motion trajectory of the marking point forms a virtual spherical surface with the optical center of the camera 300 as the sphere center. The camera 300 captures the virtual image of the marking point through the teleconverter 200, so the motion trajectory of the marking point forms a virtual spherical surface with the optical center of the camera 300 as the sphere center. When multiple target images collected by the camera are superimposed, a target composite image containing multiple virtual images of the marking point can be obtained, and the image coordinates of the marking point in the target composite image can be determined according to the target composite image. The distance between each marking point and the optical center of the camera 300 is different, so the radius of the virtual spherical surface formed by each marking point is also different. If there are multiple marking points, multiple virtual spherical surfaces will be formed, and the radius of each virtual spherical surface is the distance from the corresponding marking point to the optical center of the camera 300. It can be understood that the number of marking points included in the target image corresponds to the number of virtual spherical surfaces.

[0048] See also Fig. 9 , Fig. 9 It shows that when the number of marking points 111 is one, the virtual sphere ( Fig. 9 Left) and the projection diagram of the virtual sphere ( Fig. 9 right). Fig.10 FIG. 1 shows a virtual sphere formed when there are two marking points 111, where each marking point corresponds to a virtual sphere. Fig.10 The virtual spherical surface located at the upper part corresponds to the marking point which is farther from the optical center of the camera, and the virtual spherical surface located at the lower part corresponds to the marking point which is closer to the optical center of the camera.

[0049] It can be understood that when the first rotating mechanism 420 rotates, the movement trajectory of the virtual image of the marking point relative to the camera 300 coincides with the latitude line of the spherical coordinate system, and when the second rotating mechanism 430 rotates, the movement trajectory of the virtual image of the marking point relative to the camera 300 coincides with the longitude line of the spherical coordinate system. The longitude and latitude lines of the virtual spherical coordinate system formed by the rotation of the first rotating mechanism 420 and the second rotating mechanism 430 can cover the entire spherical surface in the field of view of the camera 300, thereby generating enough marking points to provide enough information for camera calibration. Fig.11 As shown, assuming that 121 is a virtual image of the marking point 111 , the virtual image 121 of the marking point will be formed on the latitude line 122 and the longitude line 123 of the virtual spherical coordinate system formed by the rotation of the first rotating mechanism 420 and the second rotating mechanism 430 .

[0050] See also Fig.12 , Fig.12FIG. 1 is a flow chart of a camera calibration method provided by an embodiment of the present application. In a specific embodiment, the camera calibration method is applied to Fig.14 The camera calibration device 700 and the electronic device 600 ( Fig.15 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment can be a terminal device with computing capabilities, such as a smart phone, a tablet computer, a desktop computer, a wearable electronic device, etc., which is not limited here. Fig.12 The process shown is described in detail, and the camera calibration method may specifically include the following steps:

[0051] Step S110: acquiring a plurality of target images captured by the camera during the rotation of the rotating mechanism and position information of the rotating mechanism corresponding to each of the target images.

[0052] In some embodiments, the electronic device may be connected to the camera and the servo driver so that data can be exchanged between the electronic device and the camera and the servo driver, wherein the data may include multiple target images captured by the camera and position information of the rotating mechanism corresponding to each target image. The camera and the servo driver may be connected to the electronic device via a wired or wireless method such as a ZigBee network, WiFi, and Bluetooth, and the specific connection method is not limited here.

[0053] As one way, the electronic device can be set to obtain the camera taking multiple target images during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each target image at regular intervals. As another way, the electronic device can also be set to obtain the camera taking multiple target images during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each target image when the rotating mechanism rotates a certain angle.

[0054] Step S120: acquiring a target image, wherein the target image is obtained by photographing a virtual target formed after the target is magnified by the teleconverter, and the target image includes marking points in the virtual target.

[0055] In this embodiment, the electronic device acquires a target image, which is obtained by a virtual target formed by a camera shooting a target through a teleconverter, and the target image includes a marking point in the virtual target. The size of the virtual target is larger than the target, and the structure is completely consistent with the target. Because the target image includes each marking point in the virtual target, the marking point image is an image on the target image that is easy to extract the center of the marking point.

[0056] In some embodiments, see Figure 4-Figure 8The marking points may include but are not limited to one or a combination of circular spots, field grids, BMW patterns, chessboards, and ChArUco, which are not limited here.

[0057] Step S130: superimposing the multiple target images to obtain a target composite image, and determining the image coordinates of the marking point in the target composite image according to the target composite image.

[0058] In this embodiment, the image coordinates of each marker point are determined according to the position of the image of the marker point in the target composite image, wherein the target image includes a plurality of marker points. To determine the image coordinates of the marker point, it is necessary to extract the center point of each marker point. Extracting the center point of the marker point may be to perform ellipse fitting on the marker point in the target image; obtain the ellipse equation of the marker point according to the boundary of the marker point and the ellipse fitting equation, and determine the center coordinates of the marker point according to the ellipse equation of the marker point as the image coordinates of the corresponding marker point.

[0059] In some embodiments, the longitude and latitude coordinates of the marking point in the target composite image are determined based on the position information of the rotation mechanism when each target image is collected: wherein the rotation angle φ of the first rotation mechanism is used as the longitude coordinate of the marking point, and the rotation angle θ of the second rotation mechanism is used as the latitude coordinate of the center point of the target composite image.

[0060] In some embodiments, the marking points included in the target image displayed by the target may be circular, and the target image obtained by photographing the target and forming a virtual target by magnifying it through a teleconverter, due to the projection relationship, the circular marking points appear as elliptical in the target image, and thus the ellipse fitting method can be used to calculate the center point of each ellipse in each target composite image, thereby obtaining the image coordinates of the marking points in the target composite image.

[0061] Specifically, we need to extract the boundary of each ellipse to know the length a of the major axis and the length b of the minor axis of each ellipse. For each ellipse in the target composite image, the ellipse fitting equation is: Where a is the length of the major axis of the ellipse, and b is the length of the minor axis of the ellipse. The values ​​of a and b can be obtained by measuring each ellipse in the target synthetic image, so that the center point (u 0 , v 0 ) value. Thus, the equation of the marked point ellipse can be expressed as Among them, ai and bi are the length of the major axis and the minor axis of the i-th ellipse respectively, (v i0 ,u i0 ) is the image coordinate of the center of the i-th ellipse. Thus, by fitting the ellipse according to the ellipse equation of the marked points, the parameters ai, bi, (v i0,u i0 ), (v i0 ,u i0 ) is the image coordinate of each marking point.

[0062] Step S140: determining the world coordinates of the marked point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter.

[0063] In this embodiment, the target carrier includes a plane, which is used to form a target. The target image is obtained by forming a virtual target after the target photographed by the camera is magnified by a teleconverter, so that the world coordinates of the mark point in the virtual target are determined according to the position information of the rotating mechanism corresponding to the target and the parameters of the teleconverter. It can be understood that since the virtual target is a virtual image formed by the actual target after being magnified by the teleconverter, the structure of the virtual target is completely consistent with the target, and the size of the virtual target is larger than the target.

[0064] In some embodiments, the world coordinates of the marked point in the virtual target are determined according to the position information of the rotating mechanism corresponding to the target and the parameters of the teleconverter. The size of the virtual target formed by the target through the teleconverter in the world coordinate system can be obtained according to the focal length of the teleconverter and the distance between the teleconverter and the target carrier, and then the position of the marked point in the virtual target is determined as the world coordinate according to the position information of the rotating mechanism corresponding to the target and the size of the virtual target in the world coordinate system. Since the virtual target is a virtual image after the target is enlarged, the structure of the virtual target is completely consistent with the structure of the target carrier. After determining the size of the virtual target in the world coordinate system, the position of the marked point in the virtual target can be determined as the world coordinate according to the position information of the rotating mechanism corresponding to the target.

[0065] In other embodiments, after obtaining the spherical radius corresponding to the center point of each marked point in the virtual target based on the focal length of the teleconverter and the distance between the teleconverter and the target carrier, the world coordinates of each marked point in the virtual target are determined based on the longitude and latitude coordinates of each marked point in the target composite image and the spherical radius corresponding to the center point of each marked point.

[0066] In some embodiments, the world coordinates of the marking point in the virtual target are determined according to the parameters of the target and the teleconverter. The world coordinates of each marking point on the target can be determined based on the position distribution of each marking point in the target and the design parameters of each marking point. Then, the position and magnification of the virtual target formed relative to the target are determined by the focal length of the teleconverter and the distance between the teleconverter and the stereo target. Thus, the world coordinates of the marking point in the virtual target can be determined.

[0067] Step S150: Fitting the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera.

[0068] In this embodiment, after obtaining the world coordinates and image coordinates of each mark point in the virtual target, the focal length, principal point and initial values ​​of the distortion parameters of the camera can be calculated according to the perspective imaging model. In the perspective imaging model, there is no distortion parameter, so the initial value of the distortion parameter of the camera can be set to 0.

[0069] The perspective imaging model is also called the pinhole model. In the perspective imaging model, the world coordinates (X, Y, Z) of any point P in space and the image coordinates (u, v) of the image point of the point in the camera satisfy the perspective imaging formula:

[0070]

[0071] Where [X, Y, Z, 1] is the homogeneous coordinates of the spatial point in the world coordinate system, corresponding to the homogeneous pixel coordinates [u, v, 1] in the camera coordinate system. [R, T] is a pair of rotation matrices and translation matrices between the world coordinate system and the camera coordinate system. (f u , f v ) represents the normalized focal length in the u direction (horizontal direction) and the v direction (vertical direction) in the camera coordinate system, which is the pixel focal length of the camera. 0 , v 0 ) is the coordinate of the camera principal point. Substitute the world coordinates of all the marking points on the virtual target and the image coordinates corresponding to these marking points into the perspective imaging formula, and solve the simultaneous equations composed of these marking points into the perspective imaging formula to obtain the camera focal length (f u , f v ) and the principal point coordinates (u 0 , v 0 ) in the perspective imaging model.

[0072] In some embodiments, when the target image includes a marker point, the rotation of the camera with the two rotating mechanisms is equivalent to the camera not moving and the marker point rotating around the camera in two directions, and the motion trajectory of the marker point forms a spherical surface with the camera optical center as the sphere center. Since the camera sees a virtual image of the marker point through the teleconverter, the motion trajectory of the virtual image forms a spherical surface with a larger radius with the camera optical center as the sphere center, which is a virtual spherical surface. Please refer to Fig.13 , Fig.13A schematic diagram of coordinates provided in an embodiment of the present application is shown. If the rectangular coordinates coincide with the origin of the spherical coordinates, the relationship between the midpoint P (X, Y, Z) of the rectangular coordinates and the midpoint P (R, φ, θ) of the spherical coordinates is: X = Rcosθcosφ, Y = Rcosθsinφ, Z = Rsinθ, wherein the ranges of R, θ, and φ are: R ≥ 0, -π / 2≤θ≤π / 2, -π≤φ≤π, φ is longitude, and θ is latitude.

[0073] In some embodiments, when the target image includes multiple feature points, the rotation of the camera with the two rotating mechanisms is equivalent to the rotation of the camera in two directions without the feature points moving around the camera, and the motion trajectories of the multiple feature points form multiple spherical surfaces with the optical center of the camera as the sphere center. Since the camera sees the virtual image of the feature point through the teleconverter, the motion trajectory of the virtual image forms multiple spherical surfaces with the optical center of the camera as the sphere center, which are multiple virtual spherical surfaces. Among them, if the rectangular coordinates coincide with the origin of the spherical coordinates, the relationship between the rectangular coordinates and the spherical coordinates is the same as the above-mentioned coordinate relationship, which will not be repeated here.

[0074] In some real-time methods, for a camera, point O is the optical center of the camera. A point P outside the optical center can be represented by X, Y, Z, or by R, φ, θ. If the points in the world coordinates are all on a sphere, they can also be represented by only φ, θ. When the camera is a pinhole camera without distortion, the image point on the photoelectric coupling plane is only related to the focal length f, φ, θ. When the camera is a pinhole model with radial distortion, the image point on the photoelectric coupling plane is also related to the distortion parameters. The specific type of the camera is not limited here.

[0075] Step S160: optimizing the focal length, principal point and initial values ​​of distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera.

[0076] In this embodiment, after obtaining the initial values ​​of the focal length, principal point, and distortion parameters of the camera, the Levenberg-Marquardt algorithm can be used to optimize the initial values ​​of the focal length, principal point, and distortion parameters of the camera to determine the imaging model parameters of the camera.

[0077] In some embodiments, the Levenberg-Marquardt algorithm is used to minimize the sum of the squares of the differences between the projection values ​​of the marking points on the virtual target and the measured values ​​of the marking points on the virtual target. The projection values ​​of the marking points are calculated based on the image coordinates corresponding to the marking points according to the projection model based on the world coordinates of each marking point in the virtual target; the measured values ​​refer to the coordinates of each marking point in the target image corresponding to the image coordinate system after the virtual target image is obtained by measuring the virtual target image formed by shooting the target with a camera and magnifying it with a teleconverter. Therefore, the Levenberg-Marquardt algorithm can be used to minimize the sum of the squares of the differences between the projection values ​​of the marking points on the virtual target and the measured values ​​of the marking points on the virtual target, and the focal length, principal point and initial values ​​of the distortion parameters of the camera are continuously updated to obtain the optimal solution, so that the imaging model parameters of the camera can be determined.

[0078] Furthermore, the sum of the squares of the differences between the projection values ​​of the marking points on the virtual target and the measured values ​​of the marking points on the virtual target is minimized by the Levenberg-Marquardt algorithm, that is, the value of the objective function is minimized. Where N represents the number of marking points on the virtual target. is the projection value of the marker point, that is, the pixel coordinate of the marker point calculated according to the projection model, m j is the measured value of the marker point, that is, the pixel coordinate of the marker point measured from the target image.

[0079] A camera calibration method provided by an embodiment of the present application obtains multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each target image, superimposes the multiple target images to obtain a target composite image, determines the image coordinates of the marking point in the target composite image, and determines the world coordinates of the marking point in the virtual target; fits the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera; optimizes the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera, thereby achieving that when calibrating the camera, there is no need to move the target, only to collect images during the rotation of the rotating mechanism, thereby reducing the difficulty of camera calibration and completing fast and high-precision calibration of the camera in a smaller calibration space.

[0080] See also Fig.14 , Fig.14 The module block diagram of the camera calibration device 700 provided in the embodiment of the present application is shown in FIG. Fig.14The block diagram shown in FIG. 1 is used to illustrate that the camera calibration device 700 includes: a position information acquisition module 710, a target image acquisition module 720, an image coordinate determination module 730, a world coordinate determination module 740, a camera calibration module 750, and a parameter optimization module 760, wherein:

[0081] The position information acquisition module 710 is used to acquire a plurality of target images captured by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each of the target images.

[0082] The target image acquisition module 720 is used to acquire a target image. The target image is obtained by photographing a virtual target formed after the target is magnified by the teleconverter. The target image includes marking points in the virtual target.

[0083] The image coordinate determination module 730 is used to determine the image coordinates of the marking point in the target image according to each target image.

[0084] The world coordinate determination module 740 is used to determine the world coordinates of the marked point in the virtual target according to the parameters of the target carrier and the teleconverter.

[0085] Furthermore, the world coordinate determination module 740 includes: a marker point latitude and longitude coordinate determination submodule, wherein:

[0086] The submodule for determining the longitude and latitude coordinates of the marking point is used to determine the longitude and latitude coordinates of the marking point in each target image according to the position information of the rotating mechanism when collecting each target image, wherein the rotation angle φ of the rotating mechanism on the camera axis is used as the longitude coordinate of the center point of the target image, and the rotation angle θ of the rotating mechanism on the second rotating mechanism rotation axis is used as the latitude coordinate of the center point of the target image.

[0087] Furthermore, the world coordinate determination module 740 includes: a spherical radius determination submodule and a marker point world coordinate determination submodule, wherein:

[0088] The spherical radius determination submodule is used to obtain the spherical radius corresponding to the center point of each marking point in the virtual target according to the focal length of the teleconverter and the distance between the teleconverter and the target carrier.

[0089] The marking point world coordinate determination submodule determines the world coordinate of each marking point in the virtual target according to the latitude and longitude coordinates of each marking point in the target synthetic image and the spherical radius corresponding to each marking point.

[0090] The camera calibration module 750 is used to fit the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain the initial values ​​of the focal length, principal point and distortion parameters of the camera.

[0091] The parameter optimization module 760 is used to optimize the focal length, principal point and initial values ​​of distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera.

[0092] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0093] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0094] See also Fig.15 , which shows a structural block diagram of an electronic device 600 provided in an embodiment of the present application. The electronic device 600 in the present application may include one or more of the following components: a processor 610, a memory 620, and one or more applications, wherein the one or more applications may be stored in the memory 620 and configured to be executed by one or more processors 610, and the one or more programs are configured to execute the method described in the foregoing method embodiment.

[0095] Among them, the processor 610 may include one or more processing cores. The processor 610 uses various interfaces and lines to connect various parts within the entire electronic device 600, and executes various functions and processes data of the electronic device 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 620, and calling data stored in the memory 620. Optionally, the processor 610 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 610 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 610, but may be implemented separately through a communication chip.

[0096] The memory 620 may include a random access memory (RAM) or a read-only memory (ROM). The memory 620 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 600 during use.

[0097] See also Fig.16 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0098] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0099] In summary, the camera calibration system and method provided by the embodiment of the present application obtains multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each of the target images, obtains the target image, the target image is obtained by photographing the target and magnifying it through the teleconverter to form a virtual target, the target image includes the marking point in the virtual target, determines the image coordinates of the marking point in the target image according to each target image, determines the world coordinates of the marking point in the virtual target according to the position information of the rotating mechanism corresponding to the target and the parameters of the teleconverter, fits the image coordinates and the world coordinates according to the perspective imaging model of the camera, obtains the focal length, principal point and initial values ​​of the distortion parameters of the camera, and optimizes the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera. When calibrating the camera, it is not necessary to move the target, but only to collect images during the rotation of the rotating mechanism, thereby reducing the difficulty of camera calibration and completing the fast and high-precision calibration of the camera in a smaller calibration space.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera calibration system, characterized in that: The device comprises a target carrier, a teleconverter, a camera, a rotating mechanism, a servo driver and an electronic device, wherein the teleconverter is arranged between the camera and the target carrier, and the camera and the servo driver are electrically connected to the electronic device; The target carrier includes a plane, the plane is used to form a target, and the target includes at least one marking point; The camera is installed on the rotating mechanism, and is used to photograph the virtual target formed after the target is magnified by the teleconverter to obtain a target image, wherein the target image includes the marking points in the virtual target; The rotating mechanism comprises two rotating axes, and the optical axis of the camera is coaxial with one of the rotating axes; The servo driver is used to drive the rotating mechanism to rotate and send the position information of the rotating mechanism to the electronic device; The electronic device is used to obtain multiple target images taken by the camera during the rotation of the rotating mechanism and the position information of the rotating mechanism corresponding to each of the target images, superimpose the multiple target images to obtain a target composite image, determine the image coordinates of the marking point in the target composite image according to the target composite image, determine the world coordinates of the marking point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter, fit the image coordinates and the world coordinates according to the perspective imaging model of the camera, obtain the focal length, principal point and initial values ​​of the distortion parameters of the camera, and optimize the focal length, principal point and initial values ​​of the distortion parameters of the camera through an algorithm to determine the imaging model parameters of the camera.

2. The system according to claim 1, characterized in that The rotating mechanism includes a first rotating mechanism having a first rotating axis and a second rotating mechanism having a second rotating axis, the first rotating axis and the second rotating axis are perpendicular to and intersect each other, the optical center of the camera coincides with the intersection of the first rotating axis and the second rotating axis, and the optical axis of the camera is coaxial with the first rotating axis.

3. The system according to claim 2, characterized in that The servo driver is used to drive the first rotating mechanism and the second rotating mechanism to rotate within a preset amplitude range, and send the rotation angle φ of the first rotating mechanism and the rotation angle θ of the second rotating mechanism as position information to the electronic device.

4. The system according to claim 2, characterized in that The rotating mechanism also includes a first rotary encoder installed on the first rotating mechanism and a second rotary encoder installed on the second rotating mechanism. The first rotary encoder and the second rotary encoder are both electrically connected to the servo driver. The first rotary encoder is used to obtain the rotational angle position of the first rotating mechanism, and the second rotary encoder is used to obtain the rotational angle position of the second rotating mechanism.

5. The system according to claim 1, characterized in that The optical axis of the teleconverter is perpendicular to the plane, and the distance between the teleconverter and the target carrier is smaller than the focal length of the teleconverter.

6. The system according to claim 1, characterized in that The plane of the target carrier is used to carry a physical target, or to display a target image, or to carry an electronic chart card displaying a target pattern.

7. The system according to claim 1, characterized in that The marking points include one or a combination of circular spots, field grids, BMW patterns, chessboards and ChArUco.

8. A camera calibration method, characterized in that: A camera is calibrated based on the camera calibration system according to any one of claims 1 to 7, wherein the method is applied to an electronic device and comprises: Acquire a plurality of target images captured by the camera during the rotation of the rotating mechanism and position information of the rotating mechanism corresponding to each of the target images, wherein the target image is obtained by capturing a virtual target formed after the target is magnified by the teleconverter, and the target image includes a marking point in the virtual target; Superimposing the multiple target images to obtain a target composite image, and determining the image coordinates of the marking point in the target composite image according to the target composite image; Determining the world coordinates of the marked point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter; Fitting the image coordinates and the world coordinates according to the perspective imaging model of the camera to obtain initial values ​​of the focal length, principal point and distortion parameters of the camera; The focal length, principal point and initial values ​​of distortion parameters of the camera are optimized by an algorithm to determine the imaging model parameters of the camera.

9. The method according to claim 8, characterized in that The step of superimposing the multiple target images to obtain a target composite image, and determining the image coordinates of the marking point in the target composite image according to the target composite image includes: Determine the image coordinates of each marking point according to the position of the image of the marking point in the target composite image; According to the position information of the rotating mechanism when collecting each target image, the longitude and latitude coordinates of the marking point in the target composite image are determined: wherein the rotation angle φ of the first rotating mechanism is used as the longitude coordinate of the marking point, and the rotation angle θ of the second rotating mechanism is used as the latitude coordinate of the center point of the target composite image.

10. The method according to claim 9, characterized in that The determining the world coordinates of the marking point in the virtual target according to the position information of the rotating mechanism and the parameters of the teleconverter includes: According to the focal length of the teleconverter and the distance between the teleconverter and the target carrier, the spherical radius corresponding to each marking point in the virtual target is obtained; The world coordinates of each marking point in the virtual target are determined according to the longitude and latitude coordinates of each marking point in the target composite image and the spherical radius corresponding to each marking point.

Images