Camera calibration data real-time correction method, device, computer equipment and storage medium

Abstract

The present application relates to a method, device, computer equipment and storage medium for real-time correction of camera calibration data. The method includes: obtaining the current ground expression in the vehicle body coordinate system in real time; querying the calibrated ground expression in the vehicle body coordinate system during calibration, the camera coordinate system during calibration, the vehicle body coordinate system during calibration and the vehicle coordinate system during calibration; calculating the vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression; obtaining the camera-body conversion matrix of the vehicle body coordinate system during calibration and the camera coordinate system during calibration; and performing real-time correction of the camera calibration data according to the vehicle body coordinate system conversion matrix, the camera-body conversion matrix, the vehicle body coordinate system during calibration and the vehicle coordinate system during calibration. The present method can correct the camera calibration data in real time.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle technology, and in particular to a method, device, computer equipment and storage medium for real-time correction of camera calibration data. Background Art

[0002] The camera calibration data integrated in the panoramic view system are all fixed calibration parameters. This parameter can be obtained by calibration at a certain calibration site. However, in actual use, the actual calibration parameters of the camera change in real time according to the vehicle condition, road conditions, etc. This phenomenon is particularly obvious when the vehicle is bumpy. Therefore, fixed calibration parameters will cause the panoramic view system to use inaccurate calibration parameters in actual use, which will lead to reduced performance of various algorithms. Summary of the invention

[0003] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for real-time correction of camera calibration data that can improve the accuracy of camera calibration in order to address the above technical problems.

[0004] A method for real-time correction of camera calibration data, the method comprising:

[0005] Get the current ground expression in the vehicle body coordinate system in real time;

[0006] Query the calibration ground expression under the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration;

[0007] Calculating a vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression;

[0008] Obtaining a camera-to-body conversion matrix of a body coordinate system during the calibration and a camera coordinate system during the calibration;

[0009] The camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration.

[0010] In the above embodiment, since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. The change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at the time of calibration and the camera coordinate system at the time of calibration, the camera calibration data can be corrected in real time. In this way, because the camera calibration is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving the performance robustness.

[0011] In one embodiment, the real-time correction of the camera calibration data according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration includes:

[0012] The conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration is calculated according to the body coordinate system during calibration and the vehicle coordinate system during calibration;

[0013] The vehicle body coordinate system transformation matrix, the camera body transformation matrix and the product of the transformation relationship are calculated to correct the camera calibration data in real time.

[0014] In the above embodiment, since the expression of the ground in the vehicle body coordinate system and the expression of the ground in the vehicle coordinate system obtained after the relationship between the vehicle body coordinate system and the vehicle coordinate system is converted should be consistent at any time, the conversion relationship between the vehicle body coordinate system and the vehicle coordinate system during calibration can be calculated in advance based on the vehicle body coordinate system during calibration and the vehicle coordinate system during calibration, thereby facilitating the subsequent real-time calibration of the camera.

[0015] In one embodiment, after the camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, the method further includes:

[0016] The panoramic surround stitching image is corrected according to the corrected camera calibration data.

[0017] In the above embodiment, the camera calibration data is corrected in real time, so that the panoramic view stitching image is corrected according to the externally corrected camera calibration data of the calibrated camera, so that the panoramic view stitching image is more realistic and more stable.

[0018] In one embodiment, the method further comprises:

[0019] Define the body coordinate system and vehicle coordinate system during calibration in a fixed scene;

[0020] The camera is calibrated based on the defined body coordinate system during calibration and the vehicle coordinate system during calibration to obtain a calibrated ground expression under the body coordinate system during calibration and a camera coordinate system during calibration.

[0021] In the above embodiment, in order to ensure the accuracy of subsequent real-time calibration, the camera calibration data may be corrected in real time in a fixed scene in advance to improve the accuracy of the initial calibration, thereby laying a foundation for the accuracy of subsequent real-time calibration.

[0022] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system includes:

[0023] Get the current image captured by each camera;

[0024] Collecting feature points on the ground in real time according to the current image, and obtaining three-dimensional information of the feature points;

[0025] The current ground expression is obtained by fitting according to the three-dimensional information of the feature points.

[0026] In the above embodiment, the current ground expression is obtained by means of feature points, so that the ground coordinate system conversion matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, laying the foundation for the subsequent real-time calibration of the camera.

[0027] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system includes:

[0028] Get the current image captured by each camera;

[0029] Acquire the height of each camera according to the current image by using an optical flow tracking method;

[0030] The current ground expression is calculated based on the height of each camera.

[0031] In the above embodiment, the current ground expression is obtained by optical flow tracking, so that the ground coordinate system transformation matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, laying the foundation for the subsequent real-time calibration of the camera.

[0032] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system includes:

[0033] The current ground expression in the vehicle body coordinate system is obtained through the sensors installed on the vehicle.

[0034] In the above embodiment, the current ground expression is acquired by means of a sensor, so that the ground coordinate system conversion matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, thus laying a foundation for subsequent real-time calibration of the camera.

[0035] A real-time correction device for camera calibration data, the device comprising:

[0036] The current ground expression acquisition module is used to obtain the current ground expression in the vehicle body coordinate system in real time;

[0037] A data query module, used to query the calibration ground expression under the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration;

[0038] A vehicle body coordinate system transformation matrix calculation module, used to calculate a vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression;

[0039] A camera-body conversion matrix calculation module, used to obtain the camera-body conversion matrix of the body coordinate system during the calibration and the camera coordinate system during the calibration;

[0040] The real-time calibration module is used to perform real-time correction on the camera calibration data according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration and the vehicle coordinate system during calibration.

[0041] In the above embodiment, since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. The change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at the time of calibration and the camera coordinate system at the time of calibration, the camera calibration data can be corrected in real time. In this way, because the camera calibration is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving the performance robustness.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0043] In the above embodiment, since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. The change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at the time of calibration and the camera coordinate system at the time of calibration, the camera calibration data can be corrected in real time. In this way, because the camera calibration is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving the performance robustness.

[0044] A computer-readable storage medium stores a computer program, which implements the steps of any one of the above methods when executed by a processor.

[0045] In the above embodiment, since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. The change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at the time of calibration and the camera coordinate system at the time of calibration, the camera calibration data can be corrected in real time. In this way, because the camera calibration is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving the performance robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A diagram showing an application environment of a method for real-time correction of camera calibration data in an embodiment;

[0047] Figure 2 is a relationship diagram of various coordinate systems in one embodiment;

[0048] Figure 3 is a schematic flow chart of a method for real-time correction of camera calibration data in one embodiment;

[0049] FIG4( a ) is a schematic diagram of a camera after initial calibration in one embodiment;

[0050] FIG4( b ) is a schematic diagram showing changes in camera calibration after a bumpy vehicle in the embodiment shown in FIG4( a );

[0051] FIG4(c) is a schematic diagram of the conversion relationship after the vehicle in FIG4(b) is bumped;

[0052] Figure 5 is a schematic diagram of a camera height acquisition process in one embodiment;

[0053] Figure 6 is a schematic diagram of vehicle condition changes in one embodiment;

[0054] Figure 7 is a schematic flow chart of a method for real-time correction of camera calibration data in another embodiment;

[0055] Figure 8 It is a structural block diagram of a real-time correction device for camera calibration data in one embodiment;

[0056] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The real-time correction method for camera calibration data provided in this application can be applied to Figure 1 In the application environment shown. Among them, the camera 102 communicates with the controller 104 installed on the vehicle. Specifically, the controller 104 can receive the image captured by the camera 102, and the controller can obtain the current ground expression in the vehicle body coordinate system in real time, and then obtain the calibrated ground expression in the vehicle body coordinate system during calibration, the camera coordinate system during calibration, the vehicle body coordinate system during calibration, and the vehicle coordinate system during calibration. In this way, the controller 104 can calculate the vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression, and then obtain the camera-body conversion matrix of the vehicle body coordinate system during calibration and the camera coordinate system during calibration. In this way, the controller 104 corrects the camera calibration data in real time according to the vehicle body coordinate system transformation matrix, the camera-body conversion matrix, the vehicle body coordinate system during calibration, and the vehicle coordinate system during calibration. Since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. This change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at calibration and the camera coordinate system at calibration, the camera calibration data can be corrected in real time, thereby improving the performance robustness.

[0059] Specifically, before introducing the present application in detail, several coordinate systems involved in the present application are first described, including the vehicle body coordinate system C veh , vehicle coordinate system (not shown) and camera coordinate system Where i indicates the i-th camera on the vehicle. For details, see Figure 2 , Figure 2 is a relationship diagram of various coordinate systems in an embodiment. veh The body coordinate system C is not the same as the vehicle coordinate system. veh It is a coordinate system fixed at a certain point on the vehicle body, which is used to express the position and posture of the vehicle body, including the position and posture of the camera integrated on the vehicle body. The vehicle body coordinate system C veh It can be placed anywhere for easy use. The vehicle coordinate system is generally fixed at the projection of the center of the vehicle's rear axle on the ground. Its function is to describe the position and posture of individual parts and sensors, as well as the movement of the vehicle. Camera coordinate system is the coordinate system of the camera itself.

[0060] In one embodiment, Figure 3 As shown, a real-time correction method for camera calibration data is provided, and the method is applied to Figure 1 The controller in the example is used to illustrate the following steps:

[0061] S302: Acquire the current ground expression in the vehicle body coordinate system in real time.

[0062] Specifically, during the driving process of the vehicle, the body coordinate system changes compared with the vehicle coordinate system due to the bumps of the vehicle, etc., which causes the ground expression in the body coordinate system to change. Therefore, the controller needs to re-acquire the current ground expression in the body coordinate system.

[0063] Optionally, there may be multiple ways to obtain the current ground expression, and the specific way of the current ground expression is not limited here. Any method that can obtain the current ground expression can be used. For example, a point cloud of ground feature points can be obtained through feature point detection, and then the ground expression can be parsed, or the camera height can be obtained through optical flow tracking, and then the ground expression can be obtained based on the camera height. You can even get the real-time camera height by adding a sensor, and then get the ground expression based on the camera height. As long as the real-time movement of the vehicle body relative to the ground can be obtained, the method is not limited.

[0064] S304: querying the calibration ground expression in the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration.

[0065] Specifically, the camera needs to be calibrated before the vehicle goes on the road. The calibration can be done in a fixed environment, so that the camera coordinate system during calibration can be obtained. In order to perform real-time correction on the camera calibration data later, the calibration ground expression under the body coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration need to be obtained and stored. The body coordinate system during calibration and the vehicle coordinate system during calibration are defined by the user. The vehicle coordinate system is generally defined as follows: the origin is the projection of the rear axle center on the ground, the positive direction of the X axis is pointing to the front of the vehicle, the positive direction of the Y axis is pointing to the right side of the vehicle, and the positive direction of the Z axis is perpendicular to the ground. The body coordinate system is also defined manually, and its purpose is to express the posture of the body. Because the body is rigidly connected to the camera, if all the installed cameras are regarded as a whole (which can be called a camera system), then the defined body coordinate system actually represents the posture of the camera system. Therefore, this body coordinate system can be defined at any position (as long as it does not change after definition), or directly defined at the center point of the camera connection line, and the direction can follow the direction of the vehicle coordinate system. In addition, after defining these two coordinate systems, what can be obtained at the calibration time are: the posture of each camera in the vehicle body coordinate system (the conversion relationship from the camera to the vehicle body coordinate system), the posture of the camera in the vehicle coordinate system (initial calibration), and the posture of the vehicle body coordinate system in the vehicle coordinate system (conversion relationship), which will be used in subsequent steps.

[0066] S306: Calculate the vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression.

[0067] Specifically, combined Figure 4(a) to Figure 4(c) As shown, FIG. 4(a) is a schematic diagram of the camera after initial calibration in an embodiment; FIG. 4(b) is a schematic diagram of the camera calibration change after the vehicle in the embodiment shown in FIG. 4(a) bumps; FIG. 4(c) is a schematic diagram of the conversion relationship after the vehicle in FIG. 4(b) bumps. In FIG. 4(a), after the initial calibration of the camera is completed, it is assumed that there is a plane n1 representing the camera calibration, and n1 is the camera plane normal vector. After the vehicle bumps, the body coordinate system changes, and the camera calibration also changes. Assuming that the actual calibration at a certain moment is n2, the conversion relationship between the actual calibration at this moment and the initial calibration is the conversion relationship between n1 and n2. In order to facilitate the rotation of n2, the ground is rotated at the same time so that n1 and n2 coincide. At this time, the conversion relationship between the unrotated ground normal vector A and the rotated ground normal vector B is the conversion relationship between n1 and n2. Therefore, the controller can calculate the body coordinate system transformation matrix based on the current ground expression and the calibrated ground expression.

[0068] The transformation matrix of the body coordinate system can be expressed as:

[0069]

[0070] Among them, G calib It is the calibrated ground expression in the vehicle body coordinate system. is the current ground representation in the body coordinate system.

[0071] S308: Obtain the camera-body conversion matrix of the body coordinate system during calibration and the camera coordinate system during calibration.

[0072] Specifically, the controller also needs to obtain the camera-body conversion matrix of the body coordinate system and the camera coordinate system during calibration. The camera and the body are rigidly connected. When the vehicle moves, the movement of the camera relative to the body can be ignored. Assume that a body coordinate system C is fixed somewhere on the body. veh , the coordinate system of camera i is Then the transformation matrix of the two is an invariant and can be obtained through initial calibration. Optionally, the camera-body conversion matrix can be calculated in advance, and the camera-body conversion matrix can be directly obtained when the camera calibration is actually calculated in real time.

[0073] In addition, the acquisition of the body coordinate system transformation matrix and the camera body transformation matrix can be optionally performed through multi-threading, and the first thread calculates the body coordinate system transformation matrix, and the second thread queries the camera body transformation matrix, which can improve processing efficiency.

[0074] S310: Correct the camera calibration data in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration.

[0075] Specifically, the relationship between the body coordinate system and the vehicle coordinate system changes when the vehicle is moving. If it is in the vehicle coordinate system, the expression of the ground should not change, because the vehicle coordinate system itself is attached to the ground. Therefore, a closed loop can be formed. The expression of the ground in the body coordinate system, after the relationship between the body coordinate system and the vehicle coordinate system is converted, the expression of the ground in the vehicle coordinate system should be consistent at any time:

[0076]

[0077]

[0078] in, is the current ground expression at time k, G calib To calibrate the ground representation, is the transformation relationship between the body coordinate system and the vehicle coordinate system, is the vehicle coordinate system at the current time k. The conversion relationship between the body coordinate system and the vehicle coordinate system is the body coordinate system transformation matrix, that is:

[0079]

[0080] And because the camera coordinate system and the body coordinate system have the following relationship:

[0081]

[0082] in, is the camera coordinate system of the i-th camera at time k, is the body coordinate system at time k, is the transformation relationship between the vehicle body coordinate system and the camera coordinate system at time k, which is consistent with that in the initial calibration.

[0083] Therefore, according to the above formula we can get:

[0084]

[0085] In this way, the corrected position and posture of the i-th camera at time k can be obtained.

[0086] In the above embodiment, since the body coordinate system changes with the vehicle coordinate system when the vehicle moves, the camera coordinate system also changes. Therefore, after the ground expression is robustly acquired, the body coordinate system transformation matrix can be obtained according to the change of the ground expression. The change matrix can represent the change of the camera coordinate system. In this way, according to the camera-body transformation matrix of the body coordinate system at the time of calibration and the camera coordinate system at the time of calibration, the camera calibration data can be corrected in real time. In this way, because the camera calibration is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving the performance robustness.

[0087] In one of the embodiments, the camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, including: calculating the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration according to the body coordinate system during calibration and the vehicle coordinate system during calibration; calculating the product of the body coordinate system transformation matrix, the camera body conversion matrix, and the conversion relationship to correct the camera calibration data in real time.

[0088] Specifically, the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration can be pre-calculated and stored in the controller, so that the controller can directly read the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration as needed, thereby reducing the calculation steps, reducing repeatability, and thus improving processing efficiency.

[0089] The reason why the conversion relationship between the body coordinate system and the vehicle coordinate system during calibration is required is that the expression of the ground in the body coordinate system and the expression of the ground in the vehicle coordinate system obtained after the conversion between the body coordinate system and the vehicle coordinate system should be consistent at any time. In this way, at the calibration time, the ground expression is calibrated, and the expression of the ground in the vehicle coordinate system can be obtained after the conversion between the body coordinate system and the vehicle coordinate system. This expression is consistent at any time. And according to the formula above:

[0090]

[0091] In this way, After that, the other items are known, that is, the corrected pose of the i-th camera at time k can be obtained.

[0092] In the above embodiment, since the expression of the ground in the vehicle body coordinate system and the expression of the ground in the vehicle coordinate system obtained after the relationship between the vehicle body coordinate system and the vehicle coordinate system is converted should be consistent at any time, the conversion relationship between the vehicle body coordinate system and the vehicle coordinate system during calibration can be calculated in advance based on the vehicle body coordinate system during calibration and the vehicle coordinate system during calibration, thereby facilitating the subsequent real-time calibration of the camera.

[0093] In one of the embodiments, after the camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, it also includes: correcting the panoramic surround stitching image according to the corrected camera calibration data.

[0094] Specifically, after calibrating each camera, the controller can input the corrected camera calibration data into the panoramic view module, so that the panoramic view module can correct the panoramic view stitching image according to the corrected camera calibration data. For example, the panoramic view module can stitch the current images acquired by each camera according to the corrected camera calibration data, so that the stitching accuracy is higher and the obtained panoramic view stitching image is more realistic, so that when it is displayed to the user, it can ensure that subsequent operations based on the panoramic view stitching image are safer. In other embodiments, the corrected camera calibration data can also be output to other application modules to ensure the accuracy of data calculation of other application modules. In this embodiment, only the panoramic view module is used as an example for explanation.

[0095] In the above embodiment, the camera calibration data is corrected in real time, so that the panoramic view stitching image obtained by stitching the current image according to the externally acquired and corrected camera calibration data of the calibrated camera is more realistic and stable.

[0096] In one of the embodiments, the method also includes: defining a body coordinate system during calibration and a vehicle coordinate system during calibration in a fixed scene; calibrating the camera based on the defined body coordinate system during calibration and the vehicle coordinate system during calibration to obtain a calibrated ground expression under the body coordinate system during calibration and a camera coordinate system during calibration.

[0097] Specifically, the initial calibration of the camera can be performed in a fixed scene, such as a scene with regular texture features, so that the controller can perform calibration according to the image captured by the camera, for example, it may include: first, define the body coordinate system during calibration and the vehicle coordinate system during calibration in a fixed scene, obtain the current frame image, the previous frame image, and the mileage of the vehicle from the previous frame image to the current frame image captured by the camera, calculate the initial pose of each camera corresponding to the current frame image according to the current frame image, the previous frame image, and the mileage, and obtain the initial height of each camera corresponding to the current frame. Perform global optimization based on the initial pose, initial height, and mileage of each camera to minimize the global reprojection error and obtain the camera pose of each camera corresponding to the current frame. For example, first calculate the initial pose of each camera. Specifically, the controller obtains the pose change H of the camera in the world coordinate system based on the feature matching between two frames of a single camera. cam , combined with the vehicle odometer to obtain the vehicle's posture change H veh , and then find the pose estimate H of a single camera cam2veh . Because the vehicle can only provide planar motion, the above process can only calculate the initial translation value of the plane, and cannot obtain the initial height value of each camera. Therefore, it is necessary to calculate and estimate the initial height of each camera. Finally, after the controller obtains the initial value of the full degree of freedom of each camera, it performs global optimization to minimize the global reprojection error and solve the optimal solution of all current camera poses. There are many algorithms for realizing the initial pose of the camera, such as PnP, using vanishing points, using optical flow tracking, etc., which can all obtain robust initial values ​​of the camera pose. By combining the odometer information of the vehicle, the height estimate of each camera can be solved by projection transformation, or other methods can be used, such as fitting the ground equation with the point cloud, adding sensors to directly obtain the camera height, etc., the method is not limited. The odometer information of the vehicle also provides scale constraints to ensure that the pose optimization is within a reasonable range in the global optimization step.

[0098] In this way, after the controller obtains the camera pose, that is, the camera coordinate system, it records the calibrated ground expression in the body coordinate system during calibration, the body coordinate system defined during calibration in a fixed scene, and the vehicle coordinate system defined during calibration in a fixed scene, laying the foundation for subsequent real-time calibration of the camera.

[0099] In the above embodiment, in order to ensure the accuracy of subsequent real-time calibration, the camera may be calibrated in advance in a fixed scene to improve the accuracy of the initial calibration, thereby laying a foundation for the accuracy of subsequent real-time calibration.

[0100] Specifically, the following will introduce several current methods of ground expression. Depending on the actual integrated system, the method of obtaining the ground expression may be different, but as long as it can robustly describe the changes of the ground relative to the vehicle body, the method is not limited to the several solutions listed here. For example, if the surround view system itself has an algorithm that uses feature points (such as SLAM, etc.), it can simply restore the 3D information directly and fit the ground expression.

[0101] In one of the embodiments, the current ground expression in the vehicle body coordinate system is obtained in real time, including: obtaining the current image captured by each camera; collecting feature points on the ground in real time according to the current image, and obtaining three-dimensional information of the feature points; fitting according to the three-dimensional information of the feature points to obtain the current ground expression.

[0102] Specifically, the controller obtains the current image captured by each camera; based on the current image, feature points on the ground are collected in real time, or point cloud data on the ground is collected, so that the three-dimensional information of these feature points can be obtained, and then the current ground expression can be fitted through the least squares method.

[0103] In the above embodiment, the current ground expression is obtained by means of feature points, so that the ground coordinate system conversion matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, laying the foundation for the subsequent real-time calibration of the camera.

[0104] In one of the embodiments, the current ground expression in the vehicle body coordinate system is obtained in real time, including: obtaining the current image captured by each camera; obtaining the height of each camera according to the current image by optical flow tracking method; and calculating the current ground expression according to the height of each camera.

[0105] Specifically, see Figure 5 As shown, Figure 5 A schematic diagram of a camera height acquisition process in one embodiment.

[0106] Among them, the controller can obtain the ending feature point in the current image and the corresponding starting feature point from the previous frame image. The ending feature point and the starting feature point are mutually matching feature points and are located on the camera. In this way, the controller can calculate the height of each camera based on the distance between the starting point and the ending point and the vehicle's mileage through the similarity principle.

[0107] in Figure 5 In, x (1)is the starting feature point, X (2) is the termination feature point. (1) is the point on the ground plane corresponding to the starting feature point, P (2) is the point on the ground plane corresponding to the termination feature point, where the camera height can be:

[0108]

[0109] The optical flow method is used to calculate the camera height. After the height of each camera is calculated, it is easy to derive the ground expression.

[0110] In the above embodiment, the current ground expression is obtained by optical flow tracking, so that the ground coordinate system transformation matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, laying the foundation for the subsequent real-time calibration of the camera.

[0111] In one of the embodiments, obtaining the current ground expression in the vehicle body coordinate system in real time includes: obtaining the current ground expression in the vehicle body coordinate system through a sensor installed on the vehicle.

[0112] Specifically, the sensor is used to first obtain the height of each camera, and then derive the ground expression according to the height of the camera.

[0113] In the above embodiment, the current ground expression is acquired by means of a sensor, so that the ground coordinate system conversion matrix, that is, the vehicle body coordinate system transformation matrix, can be obtained according to the calibrated ground expression during calibration, thus laying a foundation for subsequent real-time calibration of the camera.

[0114] Specifically, see Figure 6 and Figure 7 As shown, Figure 6 Schematic diagram of vehicle condition changes in one embodiment. Figure 7 The present invention is a flowchart of a method for real-time correction of camera calibration data in another embodiment.

[0115] During vehicle operation, Figure 6 As shown in , the vehicle or road conditions may change, causing the vehicle to bump and the camera calibration to change. Figure 7As shown, this embodiment uses image processing or other methods to obtain the expression of the ground in the field of view of each camera in real time, and comprehensively optimizes to obtain the expression of the real-time ground relative to the vehicle body, that is, the current ground expression. The controller solves the real-time correction matrix of each camera posture by comparing it with the ground expression of the calibration site obtained during calibration, and then obtains the corrected posture of each camera at the current moment. The focus of this embodiment is the posture correction after obtaining the ground expression, and there are many ways to obtain the ground expression or similar methods that can express the movement of the ground relative to the vehicle body. For example, the point cloud of ground feature points can be obtained through feature point detection, and then the ground equation can be analyzed, or the height of the camera can be obtained through optical flow tracking, and even the real-time camera height can be obtained by adding sensors. As long as the real-time movement of the vehicle body relative to the ground can be obtained, the method is not limited. Because the camera and the vehicle body are rigidly connected, the movement of the camera relative to the vehicle body can be ignored when the vehicle moves. Assume that a vehicle body coordinate system C is fixed somewhere on the vehicle body. veh , the coordinate system of camera i is Then the transformation matrix of the two is an invariant and can be obtained through initial calibration. During correction, it is assumed that the ground expression at time k is obtained by the above method as During the initial calibration, the ground expression of the calibration scene can be obtained as G calib . The real-time conversion relationship between the two can be obtained Then, the transformation relationship of the vehicle body coordinate system at time k relative to the calibration time can be obtained: Then the corrected position of camera i at time k can be obtained

[0116] In the above embodiments, the performance robustness of various surround-view based algorithms can be significantly improved. Because the calibration data is corrected in real time according to the actual road conditions, various algorithms can obtain more accurate calibration data, thereby improving performance robustness. In this embodiment, there is no restriction on the way to obtain the current ground expression, and it has strong adaptability, and can obtain stable performance output for different hardware and algorithm conditions. In this embodiment, the user experience is significantly improved. From the user level, a more realistic and stable surround stitching image can be seen, which improves the user experience. In this embodiment, the camera correction process algorithm flow is simple, occupies little resources, and has high real-time performance.

[0117] It should be understood that although Figure 2 and Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 7At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0118] In one embodiment, Figure 8 As shown, a real-time correction device for camera calibration data is provided, comprising: a current ground expression acquisition module 100, a data query module 200, a vehicle body coordinate system transformation matrix calculation module 300, a camera body conversion matrix calculation module 400 and a real-time calibration module 500, wherein:

[0119] The current ground expression acquisition module 100 is used to acquire the current ground expression in the vehicle body coordinate system in real time;

[0120] The data query module 200 is used to query the calibration ground expression under the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration;

[0121] A vehicle body coordinate system transformation matrix calculation module 300 is used to calculate a vehicle body coordinate system transformation matrix according to a current ground expression and a calibrated ground expression;

[0122] The camera-body conversion matrix calculation module 400 is used to obtain the camera-body conversion matrix of the body coordinate system during calibration and the camera coordinate system during calibration;

[0123] The real-time calibration module 500 is used to correct the camera calibration data in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration and the vehicle coordinate system during calibration.

[0124] In one embodiment, the real-time calibration module 500 may include:

[0125] A conversion relationship calculation unit, used to calculate the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration according to the body coordinate system during calibration and the vehicle coordinate system during calibration;

[0126] The real-time calibration unit is used to calculate the body coordinate system transformation matrix, the camera body transformation matrix and the product of the transformation relationship to correct the camera calibration data in real time.

[0127] In one embodiment, the camera calibration data real-time correction device further includes:

[0128] An acquisition module is used to obtain the current image acquired by each camera;

[0129] A stitching module is used to stitch the current image according to the external parameters of the calibrated camera to obtain a panoramic stitching image;

[0130] The display module is used to display the panoramic stitching image.

[0131] In one embodiment, the camera calibration data real-time correction device further includes:

[0132] The initial calibration module is used to define the body coordinate system and the vehicle coordinate system during calibration in a fixed scene; based on the defined body coordinate system and the vehicle coordinate system during calibration, the camera is calibrated to obtain the calibrated ground expression under the body coordinate system and the camera coordinate system during calibration.

[0133] In one embodiment, the current ground expression acquisition module 100 includes:

[0134] The feature point acquisition unit is used to acquire the current image acquired by each camera; acquire the feature points on the ground in real time according to the current image, and acquire the three-dimensional information of the feature points;

[0135] The first current ground expression acquisition unit is used to obtain the current ground expression by fitting according to the three-dimensional information of the feature points.

[0136] In one embodiment, the current ground expression acquisition module 100 includes:

[0137] A height acquisition unit is used to acquire the current image captured by each camera; the height of each camera is acquired according to the current image by optical flow tracking method;

[0138] The second current ground expression acquisition unit is used to calculate the current ground expression according to the height of each camera.

[0139] In one embodiment, the current ground expression acquisition module 100 includes:

[0140] The third current ground expression acquisition unit is used to acquire the current ground expression in the vehicle body coordinate system through a sensor installed on the vehicle.

[0141] For the specific definition of the camera calibration data real-time correction device, please refer to the definition of the camera calibration data real-time correction method above, which will not be repeated here. Each module in the above-mentioned camera calibration data real-time correction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0142] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for real-time correction of camera calibration data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0143] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: acquiring a current ground expression in a vehicle body coordinate system in real time; querying a calibrated ground expression in the vehicle body coordinate system during calibration, a camera coordinate system during calibration, a vehicle body coordinate system during calibration, and a vehicle coordinate system during calibration; calculating a vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression; acquiring a camera-body conversion matrix of the vehicle body coordinate system during calibration and a camera coordinate system during calibration; and correcting the camera calibration data in real time according to the vehicle body coordinate system transformation matrix, the camera-body conversion matrix, the vehicle body coordinate system during calibration, and the vehicle coordinate system during calibration.

[0145] In one embodiment, the real-time correction of the camera calibration data is implemented when the processor executes a computer program according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, including: calculating the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration according to the body coordinate system during calibration and the vehicle coordinate system during calibration; calculating the product of the body coordinate system transformation matrix, the camera body conversion matrix, and the conversion relationship to perform real-time correction on the camera calibration data.

[0146] In one embodiment, after the real-time correction of the camera calibration data according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration is implemented when the processor executes the computer program, it also includes: correcting the panoramic surround stitching image according to the corrected camera calibration data.

[0147] In one embodiment, when the processor executes the computer program, the following steps are also implemented: defining a body coordinate system during calibration and a vehicle coordinate system during calibration in a fixed scene; calibrating the camera based on the defined body coordinate system during calibration and the vehicle coordinate system during calibration to obtain a calibrated ground expression under the body coordinate system during calibration and a camera coordinate system during calibration.

[0148] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system implemented when the processor executes a computer program includes: acquiring the current image captured by each camera; acquiring feature points on the ground in real time based on the current image, and acquiring three-dimensional information of the feature points; and fitting the three-dimensional information of the feature points to obtain the current ground expression.

[0149] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system implemented when the processor executes a computer program includes: acquiring the current image captured by each camera; acquiring the height of each camera based on the current image through an optical flow tracking method; and calculating the current ground expression based on the height of each camera.

[0150] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system achieved when the processor executes the computer program includes: acquiring the current ground expression in the vehicle body coordinate system through a sensor installed on the vehicle.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a current ground expression in a vehicle body coordinate system in real time; querying a calibrated ground expression in the vehicle body coordinate system during calibration, a camera coordinate system during calibration, a vehicle body coordinate system during calibration, and a vehicle coordinate system during calibration; calculating a vehicle body coordinate system transformation matrix based on the current ground expression and the calibrated ground expression; obtaining a camera-body conversion matrix of the vehicle body coordinate system during calibration and a camera coordinate system during calibration; and correcting the camera calibration data in real time based on the vehicle body coordinate system transformation matrix, the camera-body conversion matrix, the vehicle body coordinate system during calibration, and the vehicle coordinate system during calibration.

[0152] In one embodiment, the computer program implemented when executed by the processor performs real-time correction of the camera calibration data according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, including: calculating the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration according to the body coordinate system during calibration and the vehicle coordinate system during calibration; calculating the product of the body coordinate system transformation matrix, the camera body conversion matrix, and the conversion relationship to perform real-time correction of the camera calibration data.

[0153] In one embodiment, after the computer program is executed by the processor and the camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, the computer program also includes: correcting the panoramic surround stitching image according to the corrected camera calibration data.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: defining a body coordinate system during calibration and a vehicle coordinate system during calibration in a fixed scene; calibrating the camera based on the defined body coordinate system during calibration and the vehicle coordinate system during calibration to obtain a calibrated ground expression under the body coordinate system during calibration and a camera coordinate system during calibration.

[0155] In one embodiment, the computer program implemented by the processor when executed by the processor obtains the current ground expression in real time in the vehicle body coordinate system, including: obtaining the current image captured by each camera; collecting feature points on the ground in real time according to the current image, and obtaining the three-dimensional information of the feature points; fitting according to the three-dimensional information of the feature points to obtain the current ground expression.

[0156] In one embodiment, the computer program implemented by the processor when executed by the processor obtains the current ground expression in real time in the vehicle body coordinate system, including: obtaining the current image captured by each camera; obtaining the height of each camera according to the current image by optical flow tracking method; and calculating the current ground expression according to the height of each camera.

[0157] In one embodiment, the real-time acquisition of the current ground expression in the vehicle body coordinate system implemented when the computer program is executed by the processor includes: acquiring the current ground expression in the vehicle body coordinate system through a sensor installed on the vehicle.

[0158] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only, Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for real-time correction of camera calibration data, the method comprising: Get the current ground expression in the vehicle body coordinate system in real time; Query the calibrated ground expression under the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration, wherein the body coordinate system is a coordinate system fixed at a certain point on the body, used to express the posture of the body, including the posture of the camera integrated on the body; the vehicle coordinate system is fixed at the projection of the rear axle center of the vehicle on the ground, used to describe the posture of each part, sensor, and the movement of the vehicle; Calculating a vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression; Obtaining a camera-to-body conversion matrix of a body coordinate system during the calibration and a camera coordinate system during the calibration; Correcting the camera calibration data in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration; The real-time correction of the camera calibration data according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration includes: The conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration is calculated according to the body coordinate system during calibration and the vehicle coordinate system during calibration; The vehicle body coordinate system transformation matrix, the camera body transformation matrix and the product of the transformation relationship are calculated to correct the camera calibration data in real time.

2. The method according to claim 1, characterized in that After the camera calibration data is corrected in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration, the method further includes: The panoramic surround stitching image is corrected according to the corrected camera calibration data.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Define the body coordinate system and vehicle coordinate system during calibration in a fixed scene; The camera is calibrated based on the defined body coordinate system during calibration and the vehicle coordinate system during calibration to obtain a calibrated ground expression under the body coordinate system during calibration and a camera coordinate system during calibration.

4. The method according to claim 1, characterized in that: The real-time acquisition of the current ground expression in the vehicle body coordinate system includes: Get the current image captured by each camera; Collecting feature points on the ground in real time according to the current image, and obtaining three-dimensional information of the feature points; The current ground expression is obtained by fitting according to the three-dimensional information of the feature points.

5. The method according to claim 1, characterized in that The real-time acquisition of the current ground expression in the vehicle body coordinate system includes: Get the current image captured by each camera; Acquire the height of each camera according to the current image by using an optical flow tracking method; The current ground expression is calculated based on the height of each camera.

6. The method according to claim 1, characterized in that The real-time acquisition of the current ground expression in the vehicle body coordinate system includes: The current ground expression in the vehicle body coordinate system is obtained through the sensors installed on the vehicle.

7. A real-time correction device for camera calibration data, characterized in that: The device comprises: The current ground expression acquisition module is used to obtain the current ground expression in the vehicle body coordinate system in real time; A data query module is used to query the calibrated ground expression under the body coordinate system during calibration, the camera coordinate system during calibration, the body coordinate system during calibration, and the vehicle coordinate system during calibration, wherein the body coordinate system is a coordinate system fixed at a certain point on the body, used to express the posture of the body, including the posture of the camera integrated on the body; the vehicle coordinate system is fixed at the projection of the rear axle center of the vehicle on the ground, used to describe the posture of each part, sensor, and the movement of the vehicle; A vehicle body coordinate system transformation matrix calculation module, used to calculate a vehicle body coordinate system transformation matrix according to the current ground expression and the calibrated ground expression; A camera-body conversion matrix calculation module, used to obtain the camera-body conversion matrix of the body coordinate system during the calibration and the camera coordinate system during the calibration; A real-time calibration module, used for correcting the camera calibration data in real time according to the body coordinate system transformation matrix, the camera body conversion matrix, the body coordinate system during calibration, and the vehicle coordinate system during calibration; The real-time calibration module comprises: A conversion relationship calculation unit, used to calculate the conversion relationship between the body coordinate system during calibration and the vehicle coordinate system during calibration according to the body coordinate system during calibration and the vehicle coordinate system during calibration; The real-time calibration unit is used to calculate the product of the vehicle body coordinate system transformation matrix, the camera body transformation matrix and the transformation relationship to perform real-time correction on the camera calibration data.

8. The device according to claim 7, characterized in that The device also includes: The initial calibration module is used to define the body coordinate system and the vehicle coordinate system during calibration in a fixed scene; based on the defined body coordinate system and the vehicle coordinate system during calibration, the camera is calibrated to obtain the calibrated ground expression under the body coordinate system and the camera coordinate system during calibration.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images