Camera calibration method performed on the basis of vehicle position estimation

Abstract

To provide a camera calibration method performed on the basis of vehicle position estimation considering that swing of a vehicle body during operation, a non-flat road surface, or a different load causes a variation in an external parameter of a camera system, which causes an error in sensing information.SOLUTION: A camera calibration method performed on the basis of vehicle position estimation includes: acquiring the speed, acceleration, angular velocity, and angular acceleration of a vehicle chassis, and estimating initial six-degree of freedom positions of the present vehicle chassis of the vehicle; detecting a feature point set of an image object in a vehicle peripheral image and vanishing points of all cameras in a sound view system, and performing matching on the basis of a feature point set of a semantic map on the periphery of the vehicle to calculate the six-degree of freedom positions of all the cameras, wherein a mass-spring-dumper model is connected between the vehicle body and the vehicle chassis; completing position estimation of the six-degree of freedom positions of the vehicle body and the vehicle chassis; and performing calibration of the six-degree of freedom positions of the cameras with the vehicle chassis as reference coordinates.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention belongs to a camera calibration method based on vehicle position estimation, and particularly refers to a dynamic camera calibration method that applies high-precision 3D map information and position estimation, and is used to achieve the function of combining image position estimation and dynamic camera calibration, and is applicable to the design of an automatic driving system or an Advanced Driver Assistance System (ADAS).

Background Art

[0002] The operation mode of an autonomous driving vehicle (AD) mainly arranges various sensors based on different application needs at different positions in the autonomous driving vehicle, and uses these sensors to detect various driving information during the driving of the autonomous driving vehicle, and provides it to the automatic driving system as a reference for the control command plan, and then controls the stable driving of the autonomous driving vehicle.

[0003] However, the fact that an autonomous driving vehicle is actually a non-rigid structure composed of many parts means that when the autonomous driving vehicle moves, there is relative movement between the parts that make it up, and the relative displacement of different parts can cause the shaking of different sensors on it. For example, depending on the vehicle type, the front of a articulated vehicle clearly shakes during driving, and the height of a logistics vehicle varies due to the load. In addition, even on an uneven road surface, it may cause the shaking of the vehicle body and cause the sensing error of the sensor. These shakes and uneven road surfaces cause the sensor to deviate from the position parameters set in the automatic driving system, resulting in detection errors in the motion state of external objects by the autonomous driving vehicle, such as the relative distance and speed between the external object and the autonomous driving vehicle. The estimation error of the position of the external object of the autonomous driving vehicle easily leads to the inability of the automatic driving system to calculate the optimal control command, and due to the excessive change of the object state, the autonomous driving vehicle has large changes in speed and acceleration, and may even collide with other objects.

[0004] Taking conventional autonomous driving systems or ADAS as an example, usually, a camera mounted on the vehicle body is used to obtain the position of an external object, and the position parameters set by the autonomous driving system are determined by an object detection and position estimation method based on an image. However, since the vehicle body and the vehicle chassis of the vehicle are actually not a rigid structure but are connected by a suspension system, the external parameters (Extrinsics) of the six-degree-of-freedom position of the camera with respect to the coordinate reference point of the vehicle chassis of the camera arranged on the vehicle body are likely to vary due to the sway of the vehicle body, different loads, and the inclination of the road surface, resulting in changes in the rotation angle and displacement of the camera with respect to the vehicle chassis, and possibly causing detection errors in the distance between the object and the vehicle in the image. Therefore, it may cause abnormalities in the autonomous driving system or ADAS.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] One embodiment of the present invention provides a camera calibration method based on vehicle position estimation, which combines the position estimation of the vehicle chassis by multiple sensor fusions and the position estimation of the vehicle body based on an image, performs iterative operations on the dynamic model by adding the vehicle body and the vehicle chassis, completes the position estimation of the six-degree-of-freedom position of the vehicle body and the vehicle chassis, and calibrates the external parameters of the camera with the vehicle chassis as the reference coordinate.

Means for Solving the Problems

[0007] The means for solving the problems of the present invention are A. Vehicle chassis position estimation using multiple sensor fusions - Obtain the speed information of the vehicle chassis, the speed and acceleration of the three-dimensional displacement of the vehicle chassis, and the angular velocity and angular acceleration of the three-dimensional rotation angle of the vehicle chassis, and estimate the initial six-degree-of-freedom position of the current vehicle chassis of the vehicle. B. Vehicle body position estimation based on images - Use a surround view system to obtain the surrounding images near the vehicle body of the vehicle, detect the set of feature points of the image objects in the images and the vanishing points of all cameras of the surround view system, and match them with the set of feature points of the semantic map near the vehicle to calculate the six-degree-of-freedom positions of all the cameras. C. Iterative calculation of the relative motion model between the vehicle body and the vehicle chassis - Use a mass-spring-damper model to estimate the three-dimensional relative displacement between the vehicle body and the vehicle chassis, refer to the road surface normal vector and the rotation angle of the vehicle chassis, iteratively fine-tune the six-degree-of-freedom position of the vehicle chassis, complete the position estimation of the six-degree-of-freedom positions of the vehicle body and the vehicle chassis, and finally, with the vehicle chassis as the reference coordinate, calibrate the external parameters of the camera. This includes the above steps.

Advantages of the Invention

[0008] Compared with the prior art, the present invention Introduces a mass-spring-damper model for the automatic driving system or ADAS to describe the relative displacement between the vehicle body and the vehicle chassis, and with the vehicle chassis as the reference coordinate, calibrates the external parameters of the camera in real time to solve the problems of the rotation angle and displacement changes of the camera relative to the vehicle chassis caused by the sway of the vehicle body, the unevenness of the road surface, and the difference in load, and the problems of perception and position estimation errors, and achieves the function of combining image position estimation and dynamic camera calibration.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0010] <Embodiment> Referring to FIG. 1, a camera calibration method based on vehicle position estimation provided by the present invention is shown. The system architecture of the dynamic camera calibration method includes a vehicle 100 having an upper body 110 and a lower vehicle chassis 120, as shown in FIGS. 2 and 3. A wheel speed meter 210 and an inertial measurement unit 220 are installed on the vehicle chassis 120, and a surround view system 300 including a plurality of cameras 310 is installed on the upper body 110. The plurality of cameras 310 are distributed around the upper body 110 to acquire vehicle surrounding images. Further, between the upper body 110 and the vehicle chassis 120, a mass-spring-damper model 230 is connected to calculate the change in the extension amount of the mass-spring-damper with the vehicle chassis 120 as the reference coordinate to estimate the three-dimensional relative displacement between the upper body 110 and the vehicle chassis 120.

[0011] With the above configuration, the correction method of the dynamic camera 310 includes at least Step A of estimating the initial six-degree-of-freedom position, speed, and acceleration of the vehicle chassis 120 by estimating the position of the vehicle chassis 120 through sensor fusion of a plurality of sensors (for example, wheel speed meter 210, inertial measurement unit 220); Step B of estimating the position of the upper body 110 based on an image, matching the detected features of the surround view system 300 using a set of feature points of a semantic map, and calculating the position estimation of the six-degree-of-freedom positions of all the cameras 310; In the iterative calculation of the relative motion model between the vehicle body 110 and the vehicle chassis 120, the position estimation of the six-degree-of-freedom position of the vehicle body 110 and the vehicle chassis 120 is completed, and finally, with the vehicle chassis 120 as the reference coordinate, step C of calibrating the external parameters of all the cameras 310 is included.

[0012] Based on the above, the detailed workflow of the present invention will be described as follows.

[0013] Regarding the position estimation of the vehicle chassis 120 in the fusion of a plurality of sensors (for example, the wheel speed meter 210 and the inertial measurement unit 220) (step A), the speed information of the vehicle chassis 120 is sensed using the wheel speed meter 210, and the speed and acceleration of the three-dimensional displacement of the vehicle chassis 120 and the angular speed and angular acceleration of the three-dimensional rotation angle of the vehicle chassis 120 are sensed using the inertial measurement unit 220, and the initial six-degree-of-freedom position of the current vehicle chassis 120 of the vehicle 100 is estimated (step A01).

[0014] On the other hand, regarding the position estimation of the vehicle body 110 based on an image (step B), using the surround view system 300, peripheral images near the vehicle body 110 of the vehicle 100, such as various road surface markings such as road signs and crosswalks, are acquired (step B01), the features of the road surface in the image are detected to form a set of feature points of the image object (step B02), and the vanishing points of all the cameras 310 of the surround view system 300 are detected (step B03), the initial estimated values of the rotation angles of all the cameras 310 are calculated (step B04), and by matching with the set of feature points of the semantic map near the vehicle 100 obtained by searching from the high-precision three-dimensional map information 400, the six-degree-of-freedom positions of all the cameras 310 are calculated (step B05).

[0015] Next, perform iterative calculations (step C) on the relative motion model between the vehicle body 110 and the vehicle chassis 120, use the mass-spring-damper model 230 to estimate the three-dimensional relative displacement between the vehicle body 110 and the vehicle chassis 120, and refer to the road surface normal vector and the rotation angle of the vehicle chassis 120 to iteratively fine-tune the six-degree-of-freedom position of the vehicle chassis 120 (step C01), estimate the six-degree-of-freedom position of the vehicle body 110 and the surround view system 300 thereon (step C02), and complete the estimation of the six-degree-of-freedom position of the vehicle chassis 120 (step C03). Finally, with the vehicle chassis 120 as the reference coordinate, calibrate the external parameters of all the cameras 310 of the surround view system 300 (step C04).

[0016] In the method described above, the estimation of the initial six-degree-of-freedom position of the current vehicle chassis 120 may be calculated based on the speed and acceleration information of the vehicle 100 calculated by the wheel speed meter 210 and the inertial measurement unit 220, with the position of the vehicle chassis 120 at the previous sampling time point as the reference point (step A03), to calculate the position of the vehicle chassis 120 at the current sampling time point. Similarly, for the estimation of the six-degree-of-freedom position of all the cameras 310, when the feature point set of the image object and the feature point set of the semantic map are matched, with the position of the vehicle body 110 at the previous sampling time point as the reference point (step B06), the six-degree-of-freedom positions of all the cameras 310 at the current sampling time point may be calculated.

[0017] Also, the road surface normal vector is generated by searching for high-precision three-dimensional map (HD map) information 400 at the position of the vehicle chassis 120. Also, the feature set of the semantic map near the vehicle 100 includes the set of edges of road markings and road signs, and is generated by searching for high-precision three-dimensional map information 400 at the position of the vehicle chassis 120. In addition, the set of the vanishing points and the feature points of the image object of all the cameras 310 of the surround view system 300 is calculated and generated based on a plurality of image features in the vehicle surrounding image.

[0018] Furthermore, the six-degree-of-freedom external parameters of all the cameras 310 of the surround view system 300 calculate the pitch and yaw angles for each camera 310 based on the vanishing points of four or more of the cameras 310 in the surround view system 300. The four cameras 310 are all arranged on the vehicle body 110, and the camera 310 and the vehicle body 110 are in a rigid body connection relationship. The pitch and yaw angles of the four cameras 310 are used to determine the initial value of the three-dimensional rotation angle of the vehicle body 110. Matching is performed based on the set of feature points of the semantic map near the vehicle 100 and the set of feature points of the image object detected by the camera 310, and the six-degree-of-freedom position of the camera 310 and the six-degree-of-freedom position of the vehicle body 110 are calculated.

[0019] Note that the six-degree-of-freedom position of the vehicle body 110 and the six-degree-of-freedom position of the vehicle chassis 120 are based on the initial value of the six-degree-of-freedom position of the vehicle body 110, the initial value of the six-degree-of-freedom position of the vehicle chassis 120, the initial values of the speed and acceleration of the vehicle chassis 120, and the road surface normal vector obtained by searching for high-precision three-dimensional map information 400 according to the position of the vehicle chassis 120. Using a mass-spring-damper model to depict the relative motion between the vehicle body 110 and the vehicle chassis 120, and performing iterations until the external forces applied to the vehicle body 110 and the vehicle chassis 120 approximate the dynamic changes of the mass-spring-damper model, continuously correct the dynamic numerical values of the mass-spring-damper model 230 including the position, speed, and acceleration, generate the six-degree-of-freedom position of the vehicle body 110 and the six-degree-of-freedom position of the vehicle chassis 120, and calculate the six-degree-of-freedom external parameters of the camera 310 of the surround view system 300.

Description of Reference Numerals

[0020] 100 Vehicle 110 Vehicle body 120 Vehicle chassis 210 Wheel speed meter 220 Inertial measurement unit 230 Mass-spring-damper model 300 Surround view system 310 Camera 400 High-precision 3D map information

Claims

1. A camera calibration method based on vehicle position estimation, comprising: Obtaining a velocity and an acceleration of a three-dimensional displacement of a vehicle chassis, and an angular velocity and an angular acceleration of a three-dimensional rotation angle of the vehicle chassis, and estimating an initial six-degree-of-freedom position of the vehicle chassis at the current time of the vehicle; a surround view system having a plurality of cameras is installed on the vehicle, an image of the surroundings near the body of the vehicle is acquired using the surround view system, a set of feature points of an image object in the vehicle surroundings image and vanishing points of all of the cameras of the surround view system are detected, and the set of feature points is matched with a semantic map of the surroundings of the vehicle to calculate six-degree-of-freedom positions of all of the cameras; and a mass-spring-dump model is connected between the body and the vehicle chassis, which estimates a three-dimensional relative displacement between the body and the vehicle chassis using the surround view system as a reference point; Iterating the six-degree-of-freedom position of the vehicle chassis according to the road surface normal vector and the rotation angle of the chassis to complete the position estimation of the six-degree-of-freedom position of the body and the vehicle chassis; calibrating a six-degree-of-freedom position of the camera using the vehicle chassis as a reference coordinate; A camera calibration method based on vehicle position estimation, comprising:

2. 2. A camera calibration method based on vehicle position estimation as described in claim 1, wherein the estimation of the initial six-degree-of-freedom position of the current vehicle chassis 120 is performed by calculating the position of the vehicle chassis at the current sampling time point using the position of the vehicle chassis at the previous sampling time point as a reference point based on information of the vehicle speed and acceleration calculated by a wheel speedometer and an inertial measurement unit.

3. 2. The method for camera calibration based on vehicle position estimation according to claim 1, wherein the road surface normal vector is generated by searching high-precision three-dimensional map information at the position of the vehicle chassis.

4. 2. The method for camera calibration based on vehicle position estimation according to claim 1, wherein the feature set of the semantic map near the vehicle includes a set of edges of road markings and edges of road signs, and is generated by searching high-precision three-dimensional map information at the position of the vehicle chassis.

5. The camera calibration method based on vehicle position estimation as described in claim 1, wherein a set of the vanishing points and the feature points of the image object of all the cameras of the surround view system is calculated and generated based on the features of multiple images in the vehicle surroundings image.

6. 2. The camera calibration method based on vehicle position estimation as claimed in claim 1, further comprising: calculating pitch and yaw angles for each of the six-degree-of-freedom positions of all of the cameras of the surround view system based on the vanishing points of the multiple cameras in the surround view system, the multiple cameras being disposed on the vehicle body and being rigidly connected to the vehicle body, the pitch and yaw angles of the multiple cameras being used to determine an initial value of a three-dimensional rotation angle of the vehicle body, and performing matching based on a set of feature points of the semantic map near the vehicle and a set of feature points of the image object detected and obtained by the camera, thereby calculating the six-degree-of-freedom positions of the cameras and the six-degree-of-freedom position of the vehicle body.

7. 2. The camera calibration method based on vehicle position estimation according to claim 1, wherein the six degrees of freedom position of the vehicle body and the six degrees of freedom position of the vehicle chassis are determined by using a mass-spring-dump model to describe the relative motion between the vehicle body and the vehicle chassis based on an initial value of the six degrees of freedom position of the vehicle body, an initial value of the six degrees of freedom position of the vehicle chassis, initial values ​​of the velocity and acceleration of the vehicle chassis, and the road surface normal vector obtained by searching high-precision three-dimensional map information using the position of the vehicle chassis, and iteratively modifying dynamic values ​​of the mass-spring-dump model including position, velocity, and acceleration until an external force applied to the vehicle body and the vehicle chassis approximates the dynamic change of the mass-spring-dump model, thereby generating the six degrees of freedom position of the vehicle body and the six degrees of freedom position of the vehicle chassis, and calculating the six degrees of freedom external parameters of the camera of the surround view system.

Images