Camera calibration method and device

Abstract

The embodiment of the present invention provides a camera calibration method and device, the method comprising: obtaining a calibration model for calibrating camera internal and external parameters, and pixel coordinate points of an image captured by the camera; wherein the calibration model for calibrating camera internal and external parameters is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system; according to the pixel coordinate points of the captured image and the calibration model for calibrating camera internal and external parameters, the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system are obtained to complete the calibration of the camera. Multi-camera calibration based on Gaussian process is performed through a pre-trained calibration model, so that the pixel coordinate points of the image captured by the camera can be input into the calibration model, and the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system can be directly obtained to complete the calibration of the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a camera calibration method and a camera calibration device. Background Art

[0002] The clarity and pixel count of camera imaging are largely determined by imaging technology, and the most important focus of imaging technology is camera calibration, which is to calculate the distance and parameters of various objects in the real world through the imaging of the camera on the sensor. Among them, camera calibration is the process of determining the camera's internal and external parameters.

[0003] At present, camera calibration usually requires taking photos at different angles based on a calibration plate, and manually specifying several points on the photos for correspondence. This camera calibration method is cumbersome and difficult to automate. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a camera calibration method and a corresponding camera calibration device that overcome the above problems or at least partially solve the above problems.

[0005] An embodiment of the present invention discloses a camera calibration method, the method comprising:

[0006] Obtain a calibration model for calibrating the internal and external parameters of the camera, and the pixel coordinate points of the image captured by the camera; wherein the calibration model for calibrating the internal and external parameters of the camera is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system;

[0007] According to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera, the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system are obtained to complete the calibration of the camera.

[0008] Optionally, obtaining the spatial point coordinates of each pixel point of the captured image in a spatial coordinate system according to the pixel coordinate points of the captured image and the calibration model for calibrating the internal and external parameters of the camera includes:

[0009] Input the pixel points of the captured image into the calibration model, and output the spatial angle relationship of each spatial point in the camera coordinate system;

[0010] The coordinates of the optical center of the camera in the spatial coordinate system are obtained, and the spatial angle relationship of each spatial point in the camera coordinate system is transformed using the coordinates of the optical center of the camera in the spatial coordinate system to obtain the spatial point coordinates of each pixel point in the spatial coordinate system.

[0011] Optionally, the generation process of the calibration model is as follows:

[0012] Obtaining pixel coordinates of each square corner point in the captured image sample training data, and obtaining the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system;

[0013] The pixel coordinates of each grid corner point in the image sample training data are used as the first input item of the calibration model, and the spatial angle relationship of each grid corner point in the image sample training data in the camera coordinate system is used as the first output item, and the first input item and the first output item are input into the calibration model to train the calibration model.

[0014] Optionally, the step of obtaining pixel coordinates of each grid corner point in the captured image sample data in the image sample training data includes:

[0015] Get the calibration data of the geometric grid calibration plate;

[0016] The calibration data of the geometric grid calibration plate is used to call an extraction function for grid corner points to obtain pixel coordinates of pixel coordinates of each grid corner point in the geometric grid calibration plate of the captured image.

[0017] Optionally, the obtaining of the spatial angle relationship between each grid corner point in the image sample training data in the camera coordinate system includes:

[0018] The spatial coordinate system is established with the projection point of the vehicle's rear axle center on the ground as the origin;

[0019] Obtaining the spatial coordinates of the first square corner point in the geometric square calibration plate in the spatial coordinate system, and calculating the spatial coordinates of each square corner point in the geometric square calibration plate based on the spatial coordinates of the first square corner point;

[0020] The conversion formula between the spatial coordinate system and the camera coordinate system of the camera is used, and the spatial coordinates of each grid corner point in the geometric grid calibration plate are used to calculate the spatial angle relationship of each grid corner point in the camera coordinate system.

[0021] Optionally, after training the calibration model, the method further includes:

[0022] Obtaining pixel coordinates of each square corner point in the captured image sample test data, and obtaining the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system;

[0023] The pixel coordinates of each square corner point in the image sample test data are used as the second input item of the calibration model, and the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system is used as the second output item, and the second input item and the second output item are input into the trained calibration model to test the trained calibration model.

[0024] Optionally, the second input item and the second output item are input into a trained calibration model, and the trained calibration model is tested, including:

[0025] Using the second input item and the second output item as inputs to the trained calibration model to obtain a fitting result;

[0026] Using the second output item to find the spatial vector of the spatial straight line passing through the camera origin of the camera, and using the spatial vector of the spatial straight line of the camera origin of the camera to solve the equation of the straight line passing through different camera origins;

[0027] The spatial straight line passing through the camera origin of the camera and the distance center of the straight line equations passing through different camera origins are used to determine the accuracy between the pixel coordinates of each grid corner point in the image sample test data and the fitting result, thereby completing the test of the trained calibration model.

[0028] The embodiment of the present invention further discloses a camera calibration device, the device comprising:

[0029] A calibration model acquisition module, used to acquire a calibration model for calibrating the internal and external parameters of the camera, and pixel coordinate points of the image captured by the camera; wherein the calibration model for calibrating the internal and external parameters of the camera is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system;

[0030] The camera calibration module is used to obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system according to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera to complete the calibration of the camera.

[0031] Optionally, the camera calibration module includes:

[0032] A spatial angle relationship output submodule is used to input the pixel points of the captured image into the calibration model and output the spatial angle relationship of each spatial point in the camera coordinate system;

[0033] The spatial point coordinate determination submodule is used to obtain the coordinates of the camera optical center in the spatial coordinate system, and use the coordinates of the camera optical center in the spatial coordinate system to transform the spatial angle relationship of each spatial point in the camera coordinate system to obtain the spatial point coordinates of each pixel point in the spatial coordinate system.

[0034] Optionally, the device further comprises:

[0035] The calibration model generation module is used to obtain a calibration model for calibrating the internal and external parameters of the camera by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system.

[0036] Optionally, the calibration model generation module includes:

[0037] The image sample training data acquisition submodule is used to obtain the pixel coordinates of each square corner point in the captured image sample training data, and to obtain the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system;

[0038] The calibration model training submodule is used to use the pixel coordinates of each grid corner point in the image sample training data as the first input item of the calibration model, and use the spatial angle relationship of each grid corner point in the image sample training data in the camera coordinate system as the first output item, and use the first input item and the first output item to input into the calibration model to train the calibration model.

[0039] Optionally, the image sample training data acquisition submodule includes:

[0040] A calibration data acquisition unit, used to acquire calibration data of a geometric grid calibration plate;

[0041] The pixel coordinate determination unit is used to call the extraction function for the grid corner points using the calibration data of the geometric grid calibration plate to obtain the pixel coordinates of the pixel coordinates of each grid corner point of the captured image in the geometric grid calibration plate.

[0042] Optionally, the image sample training data acquisition submodule includes:

[0043] A spatial coordinate system establishing unit, used to establish a spatial coordinate system with the projection point of the center of the rear axle of the vehicle on the ground as the origin;

[0044] A spatial coordinate determination unit, used to obtain the spatial coordinates of the first square corner point in the geometric square calibration plate in the spatial coordinate system, and to calculate the spatial coordinates of each square corner point in the geometric square calibration plate based on the spatial coordinates of the first square corner point;

[0045] The spatial angle relationship determination unit is used to use the conversion formula between the spatial coordinate system and the camera coordinate system of the camera, and the spatial coordinates of each grid corner point in the geometric grid calibration plate to calculate the spatial angle relationship for each grid corner point in the camera coordinate system.

[0046] Optionally, it also includes:

[0047] The calibration model testing module is used to test the calibration model obtained after training the calibration model.

[0048] Optionally, the calibration model testing module includes:

[0049] The image sample test data acquisition submodule is used to obtain the pixel coordinates of each square corner point in the captured image sample test data, and to obtain the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system;

[0050] The calibration model testing submodule is used to use the pixel coordinates of each square corner point in the image sample test data as the second input item of the calibration model, and use the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system as the second output item, and use the second input item and the second output item to input the trained calibration model to test the trained calibration model.

[0051] Optionally, the calibration model testing submodule includes:

[0052] A fitting result output unit, used to input the second input item and the second output item into the trained calibration model to obtain a fitting result;

[0053] a spatial orientation acquisition unit, configured to use the second output item to obtain a spatial vector of a spatial straight line passing through the camera origin of the camera, and use the spatial vector of the spatial straight line of the camera origin of the camera to solve equations of straight lines passing through different camera origins;

[0054] The calibration model testing unit is used to use the spatial straight line passing through the camera origin of the camera and the distance center of the straight line equation passing through different camera origins to determine the accuracy between the pixel coordinates of each grid corner point in the image sample test data and the fitting result, so as to complete the testing of the trained calibration model.

[0055] An embodiment of the present invention also discloses a vehicle, comprising: the camera calibration device, a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of any one of the camera calibration methods are implemented.

[0056] An embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the camera calibration methods are implemented.

[0057] The embodiments of the present invention include the following advantages:

[0058] In an embodiment of the present invention, by obtaining a calibration model for calibrating the internal and external parameters of the camera, the obtained calibration model can be obtained by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system, and then the pixel coordinate points of the image captured by the camera can be directly input into the pre-fitted calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, so as to complete the calibration of the camera. That is, a multi-camera calibration based on a Gaussian process is performed through a pre-trained calibration model, so that the pixel coordinate points of the image captured by the camera can be input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, and the uncertain parameters are fitted into the calibration model to avoid the influence of the uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of the steps of a camera calibration method provided by an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of the conversion between the spatial angle relationship and the spatial point coordinates provided by an embodiment of the present invention;

[0061] Figure 3 is a flowchart of another camera calibration method provided by an embodiment of the present invention;

[0062] Figure 4 is a schematic diagram of establishing a spatial coordinate system provided by an embodiment of the present invention;

[0063] Figure 5 It is a structural block diagram of a camera calibration device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] In the image measurement process and computer vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging. These geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. The process of solving this parameter is called camera calibration.

[0066] Among them, the camera calibration process includes solving the camera's internal and external parameters.

[0067] At present, the calibration of the camera can be achieved by the traditional Zhang Zhengyou calibration method, which is mainly through the geometric calibration board, such as the pixel points on the chessboard and the calibrated three-dimensional space coordinates. Through the mapping relationship between the pixel points and their corresponding space coordinate points, the camera's internal and external parameters can be obtained. Among them, the factors that affect the camera's calculation of three-dimensional space coordinates after calibration include the internal parameter matrix and the external parameter matrix. The internal parameter matrix K can be mainly expressed as the camera focal length (fx, fy) and distortion parameters. The external parameter matrix is ​​mainly R and T. At this time, the three-dimensional space point coordinates (X, Y, Z) corresponding to the pixel point on the chessboard can be calculated through the internal parameter matrix K, the external parameter matrix R and T, and the pixel coordinates (u, v) on the geometric calibration board. However, when the traditional Zhang Zhengyou calibration method is used to calibrate the camera, for example, after the traditional binocular calibration is used for the panoramic fisheye camera and the body perimeter camera, the error of the obtained three-dimensional point is relatively large.

[0068] One of the core ideas of the embodiments of the present invention is to propose a technical concept of fitting uncertain parameters to a Gaussian model for calibration, mainly through a multi-camera calibration based on a Gaussian process using a pre-trained calibration model, wherein the pre-trained calibration model is obtained by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system, so that the pixel coordinate points of the image taken by the camera can be input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, and the uncertain parameters are fitted to the Gaussian model used to train the calibration model to avoid the influence of uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration.

[0069] Reference Figure 1 , shows a flowchart of the steps of a camera calibration method provided by an embodiment of the present invention, focusing on the calibration process of the camera internal and external parameters, that is, the process of using the calibration model, which may specifically include the following steps:

[0070] Step 101, obtaining a calibration model for calibrating the internal and external parameters of the camera, and pixel coordinate points of an image captured by the camera;

[0071] In an embodiment of the present invention, the pixel coordinate points of the image taken by the camera are input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, and the uncertain parameters are fitted into the calibration model to avoid the influence of the uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration.

[0072] In order to calibrate the camera, that is, to obtain the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, it can be achieved by inputting the pixel coordinate points of the image taken by the camera into the calibration model used to calibrate the internal and external parameters of the camera.

[0073] In one embodiment of the present invention, a calibration model for calibrating the internal and external parameters of a camera can be obtained, so as to obtain the three-dimensional geometric position of the pixel coordinate points of the image captured by the camera in space through the obtained calibration model. The obtained calibration model can be a pre-trained Gaussian model, which can be obtained by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system.

[0074] Specifically, when determining the three-dimensional geometric position of the pixel coordinate point of the image in space, the three-dimensional space point coordinates (X, Y, Z) corresponding to this pixel point are calculated mainly through the intrinsic parameter matrix K, the extrinsic parameter matrices R and T, and the pixel coordinates (u, v) on the geometric calibration plate.

[0075] [u, v]K[R, T] = [C, Y, Z]

[0076] From the process of calculating the spatial point coordinates (X, Y, Z) from the image pixels (u, v), it can be seen that it is mainly necessary to determine the intrinsic parameter matrix K and the extrinsic parameter matrices R and T. The extrinsic parameter matrices R and T can jointly describe how to convert a point from the spatial coordinate system to the camera coordinate system. For the extrinsic parameter matrix T, it can represent the coordinates of the optical center of the camera in the spatial coordinate system, that is, it describes the position of the spatial origin in the camera coordinate system. In the automobile design model, it is also relatively easy to obtain the accurate coordinate position of a certain vehicle-mounted camera in the spatial coordinate system; for the extrinsic parameter matrix R and the intrinsic parameter matrix K, R represents the rotation matrix of the camera relative to the spatial coordinate system. This value is difficult to measure, and the real lens has radial and tangential distortion. K refers to the distortion parameter, which needs to be calculated by the distortion model polynomial to obtain the result. Specifically, it can be composed of focal length, coordinate axis tilt parameters, etc. The intrinsic parameter matrix needs to be calibrated separately from the extrinsic parameter matrix in the model, and a fitting polynomial is also required. Even if the calibration of the internal and external parameters of the camera is achieved by training the model, the trained model is an internal parameter model that determines the internal parameters of the camera and an external parameter model that determines the external parameters of the camera respectively, and the internal and external parameters of the camera cannot be obtained at the same time through the trained model.

[0077] In practical applications, the values ​​of the external parameter matrix T, which are relatively easy to determine, can be used as known values ​​to construct a calibration model. The calibration model can be constructed by fitting the uncertain parameters into the Gaussian model by jointly calibrating the K and R parameters. Specifically, the horizontal and vertical coordinates of each spatial point on the image can be fitted with the spatial angle relationship of each spatial point in the camera coordinate system to fit the uncertain parameters into the Gaussian model to obtain a calibration model, so as to avoid the influence of the uncertain parameters on the camera calibration.

[0078] Step 102, according to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera, obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system to complete the calibration of the camera.

[0079] In one embodiment of the present invention, after obtaining a pre-trained calibration model for calibrating a camera, the pixel coordinate points of the image taken by the camera can be input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, thereby improving the accuracy of camera calibration while realizing the automation of camera calibration.

[0080] Specifically, after the pixel coordinate points of the image taken by the camera are input into the calibration model, the calibration model can output the spatial angle relationship of each spatial point in the camera coordinate system. The output spatial angle relationship in the camera coordinate system may refer to the relative angle of each spatial point in the camera coordinate system, and this relative angle can be determined by the longitude and latitude of each spatial point in the camera coordinate system.

[0081] In practical applications, the value of the easily determined extrinsic parameter matrix T can be used as a known value. Then, the coordinates of the optical center of the camera in the spatial coordinate system, that is, the extrinsic parameter matrix T, can be obtained. Then, the coordinates of the optical center of the camera in the spatial coordinate system (that is, the extrinsic parameter matrix T) can be used to transform the spatial angle relationship of each spatial point in the camera coordinate system to obtain the spatial point coordinates of each pixel point in the spatial coordinate system, so as to determine the corresponding relationship between the three-dimensional geometric positions of the pixel coordinate points of the image in space and complete the calibration of the camera.

[0082] Specifically, refer to Figure 2 , showing a schematic diagram of the conversion of the spatial angle relationship and the spatial point coordinates provided by an embodiment of the present invention. Each point (x, y, z) in the space can be represented by latitude and longitude in the camera coordinate system, that is, the spatial point coordinates (x, y, z) can be converted to the spatial angle relationship (latitude, longitude) and vice versa.

[0083] In an embodiment of the present invention, by obtaining a calibration model for calibrating the internal and external parameters of the camera, the obtained calibration model can be obtained by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system, and then the pixel coordinate points of the image captured by the camera can be directly input into the pre-fitted calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, so as to complete the calibration of the camera. That is, a multi-camera calibration based on a Gaussian process is performed through a pre-trained calibration model, so that the pixel coordinate points of the image captured by the camera can be input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, and the uncertain parameters are fitted into the calibration model to avoid the influence of the uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration.

[0084] Reference Figure 3 , shows a flowchart of another camera calibration method provided by an embodiment of the present invention, focusing on the generation process of a calibration model for calibrating camera internal and external parameters, which may specifically include the following steps:

[0085] Step 301, training a calibration model according to the pixel coordinates of each square corner point in the captured image sample training data and the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system;

[0086] In an embodiment of the present invention, the pixel coordinate points of the image taken by the camera are input into the calibration model to directly obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system, and the uncertain parameters are fitted into the calibration model to avoid the influence of the uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration.

[0087] In order to calibrate the camera, that is, to obtain the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, the pixel coordinate points of the image taken by the camera can be input into the calibration model used to calibrate the internal and external parameters of the camera. The obtained calibration model can be a pre-trained Gaussian model, which can be obtained by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system.

[0088] In practical applications, the values ​​of the easily determined external parameter matrix T can be taken as known values ​​and a calibration model can be constructed. The calibration model can be constructed by fitting the uncertain parameters into the Gaussian model by jointly calibrating the K and R parameters.

[0089] In an embodiment of the present invention, the model function expression of the calibration model obtained through parameter fitting and Gaussian process can be as follows:

[0090] [u, v] [GaussianModel] [T] = [X, Y, Z]

[0091] Among them, (u, v) can be the pixel coordinates of a certain pixel point on the geometric calibration board, Gaussian Model can be any Gaussian model that can implement Gaussian process and perform parameter fitting, the external parameter matrix T can be the coordinates of the optical center of the camera in the spatial coordinate system, that is, it describes the position of the spatial origin in the camera coordinate system, and (X, Y, Z) can be the three-dimensional space point coordinates corresponding to this pixel point in the spatial coordinate system. The calibration model constructed above does not involve the determination of the uncertain parameter internal parameter matrix K and the external parameter matrix R that affect the camera calibration, which can improve the accuracy of the constructed calibration model for camera calibration.

[0092] Specifically, the horizontal and vertical coordinates of each spatial point on the image, that is, the pixel coordinates, can be fitted with the spatial angle relationship of each spatial point in the camera coordinate system to fit the uncertain parameters to the Gaussian model to obtain a calibration model, so as to avoid the influence of uncertain parameters on camera calibration.

[0093] In one embodiment of the present invention, in the process of training a calibration model used to calibrate the internal and external parameters of a camera, the pixel coordinates of each grid corner point in the captured image sample training data can be obtained, as well as the spatial angle relationship of each grid corner point in the image sample training data in the camera coordinate system, so as to train the calibration model according to the acquired image sample training data.

[0094] Specifically, the pixel coordinates of each grid corner point in the image sample training data can be used as the first input item of the calibration model, and the spatial angle relationship of each grid corner point in the image sample training data in the camera coordinate system can be used as the first output item, and then the first input item and the first output item determined above are input into the Gaussian model to train the calibration model. The Gaussian model used for construction can be any common Gaussian model that can realize the Gaussian process and perform parameter fitting, and this is not limited by the embodiment of the present invention.

[0095] In practical applications, the captured image sample training data contains a batch of pixel coordinates (xi, yi) of the grid corner points. (xi, yi) is used as the training input, and the longitude and latitude of each grid corner point are used as the training outputs and input into the Gaussian model for training.

[0096] Among them, the pixel coordinates (xi, yi) of the grid corner points are used as the training input, that is, the first input item. When obtaining the pixel coordinates of each grid corner point, a geometric grid calibration plate can be used. For example, the calibration data of the chessboard is used to obtain the images taken by the AVM panoramic fisheye camera and multiple DDS cameras around the vehicle body, and the extraction function cv2.findChessboardCornersSB() for the grid corner points is called to obtain the pixel coordinates (xi, yi) of the pixel coordinates of each grid corner point of the captured image in the geometric grid calibration plate. If the image of the grid corner point cannot be recognized, relevant information can be manually added to the grid corner point.

[0097] Taking the longitude and latitude of each square corner point as the training output, that is, the first output item, when obtaining the spatial angle relationship of each square corner point in the camera coordinate system, we can first do the following Figure 4As shown, based on the vehicle's IMU (Inertial Measurement Unit) detecting and measuring the vehicle's current acceleration and rotational motion, a spatial coordinate system is established with the projection point of the vehicle's rear axle center on the ground as the origin. Then, a geometric square calibration plate, such as a chessboard calibration data, can be used to record the spatial coordinates of the first square corner point in the lower left corner in the spatial coordinate system, and the spatial coordinates (X, Y, Z) of each square corner point in the geometric square calibration plate are calculated based on the spatial coordinates of the first square corner point. The conversion formula between the spatial coordinate system and the camera coordinate system of the camera is used, and the spatial coordinates of each square corner point in the geometric square calibration plate are used to calculate the spatial angle relationship for each square corner point in the camera coordinate system, that is, Figure 2 The longitude and latitude of each grid corner point are calculated as shown.

[0098] The conversion formula based on longitude, latitude and (X, Y, Z) can be shown as follows:

[0099] dx = X - cam_x

[0100] dy=Y-cam_y

[0101] dz=Z-cam_z

[0102] ra=np.sqrt(dx*dx+dy*dy+dz*dz)

[0103] latitude = np.arccos(dz / ra)

[0104] longitude = np.arctan2(dy,dx)

[0105] Specifically, (cam_x, cam_y, cam_z) can be the coordinates of the optical center of the camera in the spatial coordinate system, which is used to describe the position of the spatial origin in the camera coordinate system, and (X, Y, Z) can be the coordinates of the three-dimensional spatial point corresponding to a certain pixel point in the spatial coordinate system. Among them, dx can refer to the perpendicular distance from the point of a certain spatial point projected on the xy plane to the x-axis, which is specifically determined by the difference between the horizontal axis coordinate X of a certain pixel point in the spatial coordinate system and the horizontal axis coordinate cam_x of the corresponding position of the spatial origin of the camera coordinate system in the spatial coordinate system. dy can refer to the perpendicular distance from the point of a certain spatial point projected on the xy plane to the y-axis, which is specifically determined by the difference between the vertical axis coordinate Y of a certain pixel point in the spatial coordinate system and the vertical axis coordinate cam_y of the corresponding position of the spatial origin of the camera coordinate system in the spatial coordinate system. dz can refer to the perpendicular distance from the point of a certain spatial point projected on the xz plane to the z-axis, which is specifically determined by the vertical axis coordinate Z of a certain pixel point in the spatial coordinate system. It is determined by the difference between the vertical axis coordinate cam_z in the spatial coordinate system and the position corresponding to the spatial origin of the camera coordinate system. ra can be the length of the edge in the figure composed of dx, dy and dz. np.sqrt() returns the square root of dx*dx, dy*dy and dz*dz. The longitude of a spatial point in the camera coordinate system can be achieved by taking the inverse cosine of dz / ra using np.arccos(). The latitude of a spatial point in the camera coordinate system can be determined by the inverse tangent function np.arctan2(), which is the plane angle (i.e., radian value) between the line segment from the origin (0,0) to (dy,dx) and the positive direction of the x-axis.

[0106] Step 302, testing the trained calibration model according to the pixel coordinates of each square corner point in the captured image sample test data and the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system.

[0107] After the calibration model is trained, the calibration model obtained after the training can be tested to ensure and improve the accuracy of the calibration model when calibrating the camera.

[0108] Specifically, you can re-enable a batch of data (xi, yi, longitude, latitude) input into the model for testing, view the fitting results, and test the trained calibration model.

[0109] In practical applications, the reactivated batch of data may refer to captured image sample test data, which may include pixel coordinates of each grid corner point and the spatial angle relationship of each grid corner point in the camera coordinate system. At this time, the obtained pixel coordinates of each grid corner point may be used as the second input item of the calibration model, and the obtained spatial angle relationship of each grid corner point in the camera coordinate system may be used as the second output item. The determined second input item and second output item are input into the trained calibration model to test the trained calibration model.

[0110] In one embodiment of the present invention, a second input item and a second output item can be input into a trained calibration model to obtain a fitting result. The fitting result obtained may refer to the three-dimensional spatial coordinates obtained after calibration by the calibration model, which can be used to determine the equation of a straight line passing through different camera origins. At this time, the second output item can be used to find the spatial vector of the spatial straight line passing through the camera origin of the camera, and the spatial vector of the spatial straight line of the camera origin of the camera can be used to solve the equation of the straight line passing through different camera origins.

[0111] When determining the accuracy between the pixel coordinates of each square corner point in the image sample test data and the fitting result, that is, the three-dimensional spatial coordinates obtained after calibration by the calibration model, the distance center can be determined by the spatial straight line passing through the camera origin of the camera and the straight line equation passing through different camera origins, so as to test the trained calibration model through the determined distance center value.

[0112] Specifically, for the image sample test data, the spatial vector [m, n, p] of the spatial straight line passing through the camera origin can be calculated based on the spatial angle relationship of each grid corner point in the image sample test data in the camera coordinate system, that is, the latitude and longitude values. The spatial vector calculation process can be as follows:

[0113] m=np.sin(latitude_vals)*np.sin(longitude_vals)

[0114] n=np.sin(latitude_vals)*np.cos(longitude_vals)

[0115] p = np.cos(latitude_vals)

[0116] Among them, the sin function and / or cos function of latitude and longitude are mainly realized through the numpy module (a very fast mathematical library mainly used for array calculations).

[0117] After the calculation of the above spatial vectors, the equations of the straight lines passing through the origins of different cameras (such as avm and dds7) are solved through the above obtained spatial vectors [m,n,p]. Specifically, the coordinates of the optical center of each camera in the spatial coordinate system are known. At this time, the coordinates of the position corresponding to the origin of the camera coordinate system in the spatial coordinate system can be obtained, such as (cam_x, cam_y, cam_z). At this time, the commonly used mathematical formula point-to-point formula can be used to determine the vector [m,n,p] and the fixed point, that is, the camera origin (c The straight line equation of (am_x, cam_y, cam_z), for example, the straight line equation passing through the camera origin can be (x-cam_x) / m=(y-cam_y) / n=(z-cam_z) / p; then the accuracy of the two three-dimensional spatial coordinates (x', y', z') of the test point and the three-dimensional spatial coordinates during calibration can be calculated. The accuracy can be determined by solving the center value of the distance between two skew lines in the above-mentioned straight lines passing through different camera origins, that is, using the distance formula between the two straight lines for calculation.

[0118] It should be noted that the lower the determined distance center value, the higher the accuracy of the pixel coordinates of each grid corner point in the image sample test data and the fitting result, that is, the three-dimensional space coordinates obtained after calibration by the calibration model. When the determined distance center value exceeds a certain test threshold, it means that the accuracy of the current calibration model is not high enough. The calibration model can be continuously trained and the trained calibration model can be continuously tested until the determined distance center value is lower than the test threshold. The embodiments of the present invention are not limited to this.

[0119] In an embodiment of the present invention, a multi-camera calibration is performed based on a Gaussian process through a pre-trained calibration model, so that the pixel coordinate points of the image taken by the camera can be input into the calibration model, and the spatial point coordinates of each pixel point of the taken image in the spatial coordinate system can be directly obtained, and the uncertain parameters are fitted into the calibration model to avoid the influence of the uncertain parameters on the camera calibration, thereby improving the accuracy of the camera calibration while realizing the automation of the camera calibration.

[0120] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0121] Reference Figure 5 , shows a structural block diagram of a camera calibration device provided by an embodiment of the present invention, which may specifically include the following modules:

[0122] The calibration model acquisition module 501 is used to acquire a calibration model for calibrating the internal and external parameters of the camera, and the pixel coordinate points of the image captured by the camera; wherein the calibration model for calibrating the internal and external parameters of the camera is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system;

[0123] The camera calibration module 502 is used to obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system according to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera to complete the calibration of the camera.

[0124] In one embodiment of the present invention, the camera calibration module 502 may include the following submodules:

[0125] A spatial angle relationship output submodule is used to input the pixel points of the captured image into the calibration model and output the spatial angle relationship of each spatial point in the camera coordinate system;

[0126] The spatial point coordinate determination submodule is used to obtain the coordinates of the camera optical center in the spatial coordinate system, and use the coordinates of the camera optical center in the spatial coordinate system to transform the spatial angle relationship of each spatial point in the camera coordinate system to obtain the spatial point coordinates of each pixel point in the spatial coordinate system.

[0127] In one embodiment of the present invention, the device may further include the following modules:

[0128] The calibration model generation module is used to obtain a calibration model for calibrating the internal and external parameters of the camera by fitting the horizontal and vertical coordinates of each spatial point on the image with the spatial angle relationship of each spatial point in the camera coordinate system.

[0129] In one embodiment of the present invention, the calibration model generation module may include the following submodules:

[0130] The image sample training data acquisition submodule is used to obtain the pixel coordinates of each square corner point in the captured image sample training data, and to obtain the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system;

[0131] The calibration model training submodule is used to use the pixel coordinates of each grid corner point in the image sample training data as the first input item of the calibration model, and use the spatial angle relationship of each grid corner point in the image sample training data in the camera coordinate system as the first output item, and use the first input item and the first output item to input into the calibration model to train the calibration model.

[0132] In one embodiment of the present invention, the image sample training data acquisition submodule may include the following units:

[0133] A calibration data acquisition unit, used to acquire calibration data of a geometric grid calibration plate;

[0134] The pixel coordinate determination unit is used to call the extraction function for the grid corner points using the calibration data of the geometric grid calibration plate to obtain the pixel coordinates of the pixel coordinates of each grid corner point of the captured image in the geometric grid calibration plate.

[0135] In one embodiment of the present invention, the image sample training data acquisition submodule may include the following units:

[0136] A spatial coordinate system establishing unit, used to establish a spatial coordinate system with the projection point of the center of the rear axle of the vehicle on the ground as the origin;

[0137] A spatial coordinate determination unit, used to obtain the spatial coordinates of the first square corner point in the geometric square calibration plate in the spatial coordinate system, and to calculate the spatial coordinates of each square corner point in the geometric square calibration plate based on the spatial coordinates of the first square corner point;

[0138] The spatial angle relationship determination unit is used to use the conversion formula between the spatial coordinate system and the camera coordinate system of the camera, and the spatial coordinates of each grid corner point in the geometric grid calibration plate to calculate the spatial angle relationship for each grid corner point in the camera coordinate system.

[0139] In one embodiment of the present invention, the following modules may also be included:

[0140] The calibration model testing module is used to test the calibration model obtained after training the calibration model.

[0141] In one embodiment of the present invention, the calibration model test module may include the following submodules:

[0142] The image sample test data acquisition submodule is used to obtain the pixel coordinates of each square corner point in the captured image sample test data, and to obtain the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system;

[0143] The calibration model testing submodule is used to use the pixel coordinates of each square corner point in the image sample test data as the second input item of the calibration model, and use the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system as the second output item, and use the second input item and the second output item to input the trained calibration model to test the trained calibration model.

[0144] In one embodiment of the present invention, the calibration model test submodule may include the following units:

[0145] A fitting result output unit, used to input the second input item and the second output item into the trained calibration model to obtain a fitting result;

[0146] a spatial orientation acquisition unit, configured to use the second output item to obtain a spatial vector of a spatial straight line passing through the camera origin of the camera, and use the spatial vector of the spatial straight line of the camera origin of the camera to solve equations of straight lines passing through different camera origins;

[0147] The calibration model testing unit is used to use the spatial straight line passing through the camera origin of the camera and the distance center of the straight line equation passing through different camera origins to determine the accuracy between the pixel coordinates of each grid corner point in the image sample test data and the fitting result, so as to complete the testing of the trained calibration model.

[0148] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0149] An embodiment of the present invention further provides a vehicle, comprising:

[0150] The present invention comprises the camera calibration device, a processor, a memory and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the camera calibration method embodiment is implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0151] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned camera calibration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0152] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0153] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0154] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0158] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0159] A camera calibration method and a camera calibration device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A camera calibration method, characterized in that: The method comprises: Obtain a calibration model for calibrating the internal and external parameters of the camera, and the pixel coordinate points of the image captured by the camera; wherein the calibration model for calibrating the internal and external parameters of the camera is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system; the spatial angle relationship refers to the relative angle of each spatial point in the camera coordinate system, and the relative angle is determined by the longitude and latitude of each spatial point in the camera coordinate system; the generation process of the calibration model is as follows: obtain the pixel coordinates of each square corner point in the captured image sample training data, and obtain the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system; use the pixel coordinates of each square corner point in the image sample training data as the first input item of the calibration model, and use the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system as the first output item, and use the first input item and the first output item to input into the calibration model to train the calibration model; According to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera, the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system are obtained to complete the calibration of the camera.

2. The method according to claim 1, characterized in that The step of obtaining the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system according to the pixel coordinate points of the captured image and the calibration model for calibrating the internal and external parameters of the camera comprises: Input the pixel points of the captured image into the calibration model, and output the spatial angle relationship of each spatial point in the camera coordinate system; The coordinates of the optical center of the camera in the spatial coordinate system are obtained, and the spatial angle relationship of each spatial point in the camera coordinate system is transformed using the coordinates of the optical center of the camera in the spatial coordinate system to obtain the spatial point coordinates of each pixel point in the spatial coordinate system.

3. The method according to claim 1, characterized in that The step of obtaining pixel coordinates of each grid corner point in the captured image sample data in the image sample training data includes: Get the calibration data of the geometric grid calibration plate; The calibration data of the geometric grid calibration plate is used to call an extraction function for grid corner points to obtain pixel coordinates of pixel coordinates of each grid corner point in the geometric grid calibration plate of the captured image.

4. The method according to claim 3, characterized in that The step of obtaining the spatial angle relationship between each grid corner point in the image sample training data in the camera coordinate system includes: The spatial coordinate system is established with the projection point of the vehicle's rear axle center on the ground as the origin; Obtaining the spatial coordinates of the first square corner point in the geometric square calibration plate in the spatial coordinate system, and calculating the spatial coordinates of each square corner point in the geometric square calibration plate based on the spatial coordinates of the first square corner point; The conversion formula between the spatial coordinate system and the camera coordinate system of the camera is used, and the spatial coordinates of each grid corner point in the geometric grid calibration plate are used to calculate the spatial angle relationship of each grid corner point in the camera coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that: After training the calibration model, the method further includes: Obtaining pixel coordinates of each square corner point in the captured image sample test data, and obtaining the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system; The pixel coordinates of each square corner point in the image sample test data are used as the second input item of the calibration model, and the spatial angle relationship of each square corner point in the image sample test data in the camera coordinate system is used as the second output item, and the second input item and the second output item are input into the trained calibration model to test the trained calibration model.

6. The method according to claim 5, characterized in that The second input item and the second output item are input into the trained calibration model, and the trained calibration model is tested, including: Using the second input item and the second output item as inputs to the trained calibration model to obtain a fitting result; Using the second output item to find the spatial vector of the spatial straight line passing through the camera origin of the camera, and using the spatial vector of the spatial straight line of the camera origin of the camera to solve the equation of the straight line passing through different camera origins; The spatial straight line passing through the camera origin of the camera and the distance center of the straight line equations passing through different camera origins are used to determine the accuracy between the pixel coordinates of each grid corner point in the image sample test data and the fitting result, thereby completing the test of the trained calibration model.

7. A camera calibration device, characterized in that: The device comprises: A calibration model acquisition module, used to acquire a calibration model for calibrating the internal and external parameters of a camera, and pixel coordinate points of an image captured by the camera; wherein the calibration model for calibrating the internal and external parameters of a camera is obtained by fitting the pixel coordinates of each spatial point in the image with the spatial angle relationship of each spatial point in the camera coordinate system; the spatial angle relationship refers to the relative angle of each spatial point in the camera coordinate system, and the relative angle is determined by the longitude and latitude of each spatial point in the camera coordinate system; the generation process of the calibration model is as follows: acquiring the pixel coordinates of each square corner point in the captured image sample training data, and acquiring the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system; using the pixel coordinates of each square corner point in the image sample training data as the first input item of the calibration model, and using the spatial angle relationship of each square corner point in the image sample training data in the camera coordinate system as the first output item, and using the first input item and the first output item to input into the calibration model to train the calibration model; The camera calibration module is used to obtain the spatial point coordinates of each pixel point of the captured image in the spatial coordinate system according to the pixel coordinate points of the captured image and the calibration model used to calibrate the internal and external parameters of the camera to complete the calibration of the camera.

8. A vehicle, characterized in that: include: The camera calibration device, processor, memory and computer program stored in the memory and capable of running on the processor as claimed in claim 7, wherein the computer program, when executed by the processor, implements the steps of the camera calibration method as claimed in any one of claims 1 to 6.

Images