Online calibration method, device and equipment of vehicle, storage medium and program product

Abstract

The invention relates to an online calibration method and device for a vehicle, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an image acquired by a camera of a vehicle and laser point cloud data acquired by a laser radar; performing calibration feature extraction on the image, predicting a vanishing point position in the image according to the extracted camera calibration feature, and calculating a camera external parameter of the current online calibration according to the vanishing point position; converting the laser point cloud data into a depth map according to relative external parameters between the camera and the laser radar obtained through previous online calibration, and performing image feature extraction on the depth map to obtain point cloud features; and calculating relative external parameters between the current online calibration camera and the laser radar based on correlation features between the image features and the point cloud features, and calculating external parameters of the current online calibration laser radar according to the external parameters of the current online calibration camera and the relative external parameters between the camera and the laser radar. And an online calibration result can be quickly and accurately obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an online calibration method, device, computer device, computer-readable storage medium, and computer program product for a vehicle. Background Art

[0002] Autonomous driving technology or vehicle assisted driving technology usually requires real-time online calibration of the relative external parameters between multi-source sensors to better accurately correlate different source data, which can effectively make up for the defects of a single sensor, thereby improving vehicle performance and safety. However, this also introduces the correlation problem between multi-source data, that is, the calibration between multi-source sensors. In particular, the correlation between two-dimensional and three-dimensional data such as cameras and lidars is more difficult to achieve.

[0003] Online calibration is to automatically calibrate the sensors and control systems in real time or regularly during vehicle operation using the real-time data collected by the sensors in real time, without disassembling or stopping the vehicle. In related technologies, the online calibration process basically extracts some artificially designed specific features, such as structures like lines or poles, for correlation. It is greatly restricted by the environment and highly dependent on the environment, and it is impossible to quickly and accurately obtain the online calibration result. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide an online calibration method, device, computer device, computer-readable storage medium, and computer program product for a vehicle. By adopting an end-to-end calibration method based on feature extraction, it can reduce the dependence on the environment and can quickly and accurately obtain the online calibration result.

[0005] In a first aspect, this application provides an online calibration method for a vehicle, including:

[0006] Obtain the sensor data required for the current online calibration, where the sensor data includes the images collected by the camera of the vehicle and the lidar point cloud data collected by the lidar of the vehicle;

[0007] Extract calibration features from the image, predict the vanishing point position in the image according to the extracted camera calibration features, and calculate the external parameters of the camera for the current online calibration according to the vanishing point position;

[0008] Convert the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features;

[0009] Based on the correlation features between the image features and the point cloud features, calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration. According to the camera extrinsic parameters of the current online calibration and the relative extrinsic parameters between the camera and the lidar, calculate the lidar extrinsic parameters of the current online calibration.

[0010] In one embodiment, the extracting calibration features from the image, predicting the position of the vanishing point in the image according to the extracted camera calibration features, and calculating the camera extrinsic parameters of the current online calibration according to the position of the vanishing point include:

[0011] Extract image features from the image to obtain image features;

[0012] Extract camera calibration features from the image features to obtain camera calibration features;

[0013] Predict the position of the vanishing point where the horizontal lines in the image intersect according to the camera calibration features, and calculate the camera extrinsic parameters of the current online calibration according to the position of the vanishing point.

[0014] In one embodiment, the method further includes:

[0015] Calculate the correlation features between the image features and the point cloud features;

[0016] The calculating the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features includes:

[0017] Perform corresponding point matching based on the correlation features, predict the matching corresponding points between the laser point cloud data and the image, and calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the matching corresponding points.

[0018] In one embodiment, the method further includes:

[0019] Obtain the vehicle pose obtained from the previous online calibration, the matching corresponding points between the laser point cloud data and the image, and the correlation features for multiple rounds of iterative optimization to obtain the vehicle pose of the current online calibration;

[0020] Obtain the positioning information of the vehicle according to the vehicle pose of the current online calibration and the map data.

[0021] In one embodiment, the method further includes:

[0022] Perform depth estimation on the image to obtain the estimated depth value of each pixel in the image;

[0023] An estimated depth map corresponding to the image is obtained according to the estimated depth values of each pixel in the image;

[0024] According to the camera internal parameters and the estimated depth map corresponding to the image, the image features are projected into the three-dimensional space to obtain three-dimensional image features;

[0025] According to the relative external parameters between the camera and the lidar obtained from the previous online calibration, after converting the three-dimensional image features from the camera coordinate system to the point cloud coordinate system, they are fused with the point cloud features to obtain fused features, and the obstacle perception result of the current online calibration is obtained based on the fused features.

[0026] In one embodiment, the sensor data further includes millimeter-wave point cloud data collected by the millimeter-wave radar of the vehicle, and the millimeter-wave point cloud data includes the vehicle speed. Converting the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration includes:

[0027] The vehicle speed is merged with the lidar point cloud data to obtain point cloud data carrying the vehicle speed;

[0028] According to the relative external parameters between the camera and the lidar obtained from the previous online calibration, coordinate transformation is performed on the point cloud data carrying the vehicle speed to obtain point cloud data in the camera coordinate system;

[0029] The point cloud in the camera coordinate system is projected onto the image plane according to the camera internal parameters to obtain a depth map.

[0030] In a second aspect, the present application further provides an online calibration device for a vehicle, including:

[0031] A data acquisition module, configured to acquire sensor data required for the current online calibration, where the sensor data includes an image collected by a camera of the vehicle and lidar point cloud data collected by a lidar of the vehicle;

[0032] A camera external parameter determination module, configured to extract calibration features from the image, predict the position of the vanishing point in the image according to the extracted camera calibration features, and calculate the external parameters of the camera for the current online calibration according to the position of the vanishing point;

[0033] A point cloud data processing module, configured to convert the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map to obtain point cloud features;

[0034] The lidar extrinsic parameter determination module is used to calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0035] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Obtain the sensor data required for the current online calibration. The sensor data includes the images collected by the camera of the vehicle and the lidar point cloud data collected by the lidar of the vehicle;

[0037] Extract calibration features from the image, predict the vanishing point position in the image according to the extracted camera calibration features, and calculate the camera extrinsic parameters for the current online calibration according to the vanishing point position;

[0038] Convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features;

[0039] Based on the correlation features between the image features and the point cloud features, calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0040] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0041] Obtain the sensor data required for the current online calibration. The sensor data includes the images collected by the camera of the vehicle and the lidar point cloud data collected by the lidar of the vehicle;

[0042] Extract calibration features from the image, predict the vanishing point position in the image according to the extracted camera calibration features, and calculate the camera extrinsic parameters for the current online calibration according to the vanishing point position;

[0043] Convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features;

[0044] Calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0045] In a fifth aspect, the present application also provides a computer program product, including a computer program, which when executed by a processor implements the following steps:

[0046] Obtain the sensor data required for the current online calibration, where the sensor data includes the images collected by the camera of the vehicle and the lidar point cloud data collected by the lidar of the vehicle;

[0047] Extract calibration features from the image, predict the position of the vanishing point in the image according to the extracted camera calibration features, and calculate the camera extrinsic parameters for the current online calibration according to the position of the vanishing point;

[0048] Convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map to obtain point cloud features;

[0049] Calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0050] The above online calibration method, device, computer device, computer-readable storage medium and computer program product for the vehicle, by obtaining the images collected by the camera of the vehicle and the lidar point cloud data collected by the lidar of the vehicle required for the current online calibration, extracting calibration features from the images, predicting the position of the vanishing point in the images according to the extracted camera calibration features, calculating the camera extrinsic parameters for the current online calibration according to the position of the vanishing point, converting the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extracting image features from the depth map to obtain point cloud features; calculating the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculating the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar, provides an end-to-end online calibration process for the camera and the lidar based on feature extraction, and can quickly and accurately obtain the online calibration result. Description of the Drawings

[0051] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the accompanying drawings required for the description of the embodiments of the present application or related technologies. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.

[0052] Figure 1 It is an application environment diagram of an online calibration method for a vehicle in an embodiment;

[0053] Figure 2 It is a schematic flowchart of an online calibration method for a vehicle in an embodiment;

[0054] Figure 3 It is a schematic flowchart of determining the external camera parameters for the current online calibration in an embodiment;

[0055] Figure 4 It is a schematic flowchart of obstacle perception for a vehicle in an embodiment;

[0056] Figure 5 It is a block diagram of an online calibration method for a vehicle in an embodiment;

[0057] Figure 6 It is a structural block diagram of an online calibration device for a vehicle in an embodiment;

[0058] Figure 7 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The online calibration process of an autonomous vehicle ensures the accuracy of its data and the reliability of its system by calibrating sensors and control systems in real time. The sensors include the vehicle's cameras, radars, lidars, millimeter-wave radars, inertial measurement units (IMUs), etc. Among them, the vehicle's cameras are used to collect image data in real time, capturing images and videos. The vehicle's radars are used to collect three-dimensional point cloud data in real time to obtain the distances and / or speeds of objects in front of or around the vehicle. The inertial measurement unit (IMU) is used to record the vehicle's real-time acceleration and angular velocity. Among them, the joint calibration of lidar and camera is a key step in autonomous driving. The joint calibration process of lidar and camera can be used to determine the relative extrinsic parameters between the lidar and the camera, that is, the coordinate transformation matrix between the lidar coordinate system and the camera coordinate system, including the rotation matrix and the translation vector, so that the data of the two can be aligned in the same coordinate system.

[0061] The pose estimation of the vehicle can not only provide the positioning position and attitude information of the vehicle, but also provide the motion state of the vehicle, so as to realize path planning and improve driving safety and reliability. In the embodiments of the present application, the position and attitude of the vehicle can be determined by fusing the images collected by the camera and the point cloud data collected by the lidar.

[0062] The vehicle's perception system can detect static and dynamic obstacles around the vehicle to obtain the surrounding environmental information, providing an information source for the downstream path planning and control modules to avoid obstacles. In the embodiments of the present application, after determining the relative extrinsic parameters between the camera and the lidar, the point cloud data can be projected onto the image plane, or the image data can be mapped into the point cloud space. After the spatial alignment of the two types of data, features are extracted respectively, and after feature fusion, object detection is realized based on the fused features to obtain the environmental perception result around the vehicle.

[0063] In the related art, the online calibration process basically extracts some artificially designed specific features, such as line or pole structures, for association. It is greatly restricted by the environment and highly dependent on the environment, and it is impossible to obtain the online calibration result quickly and accurately. In the related art, the calibration, positioning, and perception systems of the vehicle are designed independently of each other. Each system publishes the information it extracts, and other modules subscribe to the information they are interested in to achieve the purpose of synchronous collaborative work between different systems. However, this method has low timeliness. Each system requires complex, rule-based post-processing algorithms to improve the stability of the algorithm. At the same time, each system extracts features based on artificially designed targets, and there is often information loss during the information transmission process, resulting in inaccurate results and affecting the performance and safety of the vehicle.

[0064] The online calibration method for vehicles provided by the embodiments of the present application can adopt an end-to-end calibration method based on feature extraction, which can reduce the dependence on the environment and quickly and accurately obtain the online calibration results, including the extrinsic parameters of the camera, the extrinsic parameters of the lidar, and the relative extrinsic parameters between the camera and the lidar. In some embodiments, the online calibration method for vehicles provided by the embodiments of the present application can estimate the vehicle pose based on the extracted features, with higher robustness. In some embodiments, the online calibration method for vehicles provided by the embodiments of the present application realizes the integration of online calibration, positioning, and perception of the vehicle based on multi-source sensor data, can achieve the coordination of online calibration, real-time positioning, and perception, give full play to the role of each network, and effectively reduce the dependence on the CPU (Central Processing Unit), concentrate all tasks on the GPU (Graphics Processing Unit), improve performance, and improve accuracy at the same time.

[0065] The online calibration method for vehicles provided by the embodiments of the present application can be applied to an application environment such as Figure 1 shown. In some embodiments, the computer device 102 can perform multiple rounds of online calibration on the vehicle. Each time online calibration is performed, the computer device 102 can obtain the sensor data for the current online calibration. The sensor data includes the images collected by the vehicle's camera and the lidar point cloud data collected by the vehicle's lidar; extract the calibration features from the images, predict the position of the vanishing point in the image according to the extracted camera calibration features, and calculate the extrinsic parameters of the camera for the current online calibration according to the position of the vanishing point; convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract the image features from the depth map to obtain the point cloud features; calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the extrinsic parameters of the lidar for the current online calibration according to the extrinsic parameters of the camera for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0066] In some embodiments, the computer device 102 is a device on a vehicle. The computer device 102 implements online calibration, positioning, and perception of the vehicle, and based on the obtained online calibration results, estimated pose, and perception information, implements driving planning and decision-making for the vehicle. In some embodiments, the computer device may also be other computer devices that are not on the vehicle. The other computer device receives image and point cloud data from the vehicle, implements online calibration, positioning, and perception of the vehicle, and based on the obtained online calibration results, estimated pose, and perception information, implements driving planning and decision-making for the vehicle, and sends the planning and decision-making results to the vehicle. In some embodiments, the computer device 102 can be, but is not limited to, various personal computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, intelligent vehicle-mounted devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0067] In an exemplary embodiment, as Figure 2 shown, an online calibration method for a vehicle is provided. Taking the method applied to the Figure 1 computer device therein as an example, it includes the following steps 202 to 208. Among them:

[0068] Step 202, obtain the sensor data required for the current online calibration. The sensor data includes the images collected by the vehicle's camera and the laser point cloud data collected by the vehicle's lidar.

[0069] Specifically, the computer device can continuously perform multiple rounds of online calibration on the vehicle based on the real-time collected sensor data. For each online calibration process, the computer device can obtain the sensor data required for the current online calibration, and can also obtain the extrinsic calibration results of the sensors, the vehicle pose, and the perception information of the previous online calibration process. These information combined with the sensor data required for the current online calibration can obtain the calibration results of the current online calibration.

[0070] During the vehicle's driving process, the vehicle's sensors collect data in real time. Specifically, the vehicle's camera continuously collects images to obtain image data. The vehicle's lidar collects laser point cloud data. The point cloud data collected by the lidar is a set of three-dimensional coordinates of the object surface obtained through laser ranging technology, which has the characteristics of high precision, discreteness, and multi-dimension. It is a set composed of a large number of discrete three-dimensional points. Each point represents the position of an object surface in space, that is, the three-dimensional coordinates (x, y, z) of each point represent the position of the point in space.

[0071] In some embodiments, a computer device on a vehicle may obtain sensor data required for the current on-line calibration. In some embodiments, when the computer device is not a device on the vehicle, the computer device may obtain the sensor data required for the current on-line calibration from the vehicle's devices through a network.

[0072] Step 204: Extract calibration features from the image, predict the position of the vanishing point in the image based on the extracted camera calibration features, and calculate the extrinsic parameters of the camera for the current on-line calibration based on the position of the vanishing point.

[0073] During the on-line calibration process, in order to estimate in real time the extrinsic parameters (i.e., the rotation matrix and the translation vector) of the camera relative to the vehicle coordinate system, that is, the camera extrinsic parameters, the computer device extracts calibration features from the images captured by the camera, that is, extracts feature points (such as lane lines, markers, ground textures, etc.) that can be used for camera calibration from the images. Usually, the road boundary lines or lane lines on both sides of the road where the vehicle travels are parallel. In the images captured by the vehicle's camera, the road boundary lines or lane lines are not parallel in the image, and the point where these lines intersect in the image is called the vanishing point. The intersection point (vanishing point) of the lane lines in the image can provide pose information about the installation of the camera, that is, the attitude of the camera relative to the vehicle.

[0074] Specifically, the computer device can determine the position (u, v) of the vanishing point in the image by extracting calibration features from the image. The position of the vanishing point is the point where the horizontal lines (such as road boundary lines or lane lines) in the image intersect. Given the known intrinsic parameters of the camera, the extrinsic parameters of the camera, that is, the extrinsic parameters of the camera for the current on-line calibration, can be deduced based on the position (u, v) of the vanishing point.

[0075] Step 206: Convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous on-line calibration, extract image features from the depth map, and obtain point cloud features.

[0076] The extrinsic parameters of the lidar are the position and orientation of the lidar relative to the vehicle coordinate system, which are used to align the lidar point cloud data collected by the lidar with the data of other sensors of the vehicle. It should be noted that before starting the online calibration, the computer device can give an initial extrinsic parameter of the camera and an initial extrinsic parameter of the lidar. According to the initial extrinsic parameter of the camera and the initial extrinsic parameter of the lidar, the relative extrinsic parameter between the initial camera and the lidar can be calculated. After the online calibration starts, the extrinsic parameter of the camera, the extrinsic parameter of the lidar, and the relative extrinsic parameter between the camera and the lidar will all be updated. For the first online calibration process, the computer device can convert the lidar point cloud data into a depth map according to the relative extrinsic parameter between the initial camera and the lidar, extract image features from the depth map, and obtain point cloud features. For non-first online calibration processes, the computer device can convert the lidar point cloud data into a depth map according to the relative extrinsic parameter between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features.

[0077] Convert the lidar point cloud data into a depth map according to the relative extrinsic parameter between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features, so as to convert the lidar point cloud data in the lidar coordinate system to the camera coordinate system, which is convenient for calculating the relative extrinsic parameter between the camera and the lidar in the current online calibration. A depth map is an image representing the distance between each point in the scene and the camera. In the depth map, the value of each pixel represents the distance between the corresponding point in the scene and the camera.

[0078] Step 208, calculate the relative extrinsic parameter between the camera and the lidar in the current online calibration based on the correlation feature between the image feature and the point cloud feature, and calculate the extrinsic parameter of the lidar in the current online calibration according to the extrinsic parameter of the camera and the relative extrinsic parameter between the camera and the lidar in the current online calibration.

[0079] After obtaining the point cloud features, the computer device can further calculate the correlation feature between the image feature and the point cloud feature. According to this correlation feature, calculate the relative extrinsic parameter between the camera and the lidar in the current online calibration, realize the update of the relative extrinsic parameter between the camera and the lidar, and thus, calculate the extrinsic parameter of the lidar in the current online calibration according to the extrinsic parameter of the camera and the relative extrinsic parameter between the camera and the lidar in the current online calibration, and realize the update of the extrinsic parameter of the lidar.

[0080] The above online calibration method for a vehicle obtains the images captured by the vehicle's camera and the lidar point cloud data collected by the vehicle required for the current online calibration, extracts calibration features from the images, predicts the vanishing point position in the images based on the extracted camera calibration features, calculates the external parameters of the camera for the current online calibration according to the vanishing point position, converts the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, extracts image features from the depth map to obtain point cloud features; based on the correlation features between the image features and the point cloud features, calculates the relative external parameters between the camera and the lidar for the current online calibration, and calculates the external parameters of the lidar for the current online calibration according to the external parameters of the camera for the current online calibration and the relative external parameters between the camera and the lidar. Based on feature extraction, it provides a collaborative end-to-end online calibration process for the camera and the lidar, and can quickly and accurately obtain the online calibration result.

[0081] In some embodiments, as Figure 3 shown, step 204, extracting calibration features from the image, predicting the vanishing point position in the image based on the extracted camera calibration features, and calculating the external parameters of the camera for the current online calibration according to the vanishing point position, includes:

[0082] Step 302, extracting image features from the image to obtain image features.

[0083] Optionally, the computer device can extract features from the image through an image feature extraction model based on deep learning to obtain image features. The embodiment of the present application does not limit the model structure of the image feature extraction model based on deep learning. The image features are preliminary features reflecting the global information of the image.

[0084] Step 304, extracting camera calibration features from the image features to obtain camera calibration features.

[0085] Optionally, the computer device can further extract higher-order features related to camera calibration from the image features through a feature extraction model based on deep learning, that is, camera calibration features.

[0086] Step 306, predicting the vanishing point position where the horizontal lines in the image intersect according to the camera calibration features, and calculating the external parameters of the camera for the current online calibration according to the vanishing point position.

[0087] The horizontal lines in the image refer to road feature points available for camera calibration, such as lane lines, road boundary lines, etc. In this embodiment, by extracting image features and calibration features from the image obtained in the current online calibration, predicting the vanishing point position where the horizontal lines in the image intersect, and calculating the external parameters of the camera for the current online calibration according to the vanishing point position, the real-time update of the external parameters of the camera can be realized.

[0088] In some embodiments, the method further includes:

[0089] Calculating the correlation features between the image features and the point cloud features;

[0090] Based on the correlation features between the image features and the point cloud features, calculating the relative external parameters between the current online calibrated camera and the lidar, including: performing corresponding point matching based on the correlation features, predicting the matching corresponding points between the laser point cloud data and the image, and calculating the relative external parameters between the current online calibrated camera and the lidar based on the matching corresponding points.

[0091] Optionally, the computer device can calculate the inner product between the image features and the point cloud features to obtain the correlation information between the two features, that is, the correlation features. The correlation features express the similarity between the point cloud data and the image data, so they can be used for feature alignment, predicting the matching corresponding points between the laser point cloud data and the image, and calculating the relative external parameters between the current online calibrated camera and the lidar according to these matching corresponding points.

[0092] In some embodiments, after obtaining the correlation features between the image features and the point cloud features, the computer device further extracts the features of the correlation features through a deep learning model (such as convGRU), and estimates the three-degree-of-freedom angular external parameters between the camera and the lidar based on the extraction results.

[0093] In this embodiment, through the correlation features between the image features and the point cloud features, the relevant matching corresponding points between the two can be found, so as to realize the real-time update of the relative external parameters between the camera and the lidar.

[0094] In some embodiments, the method further includes: obtaining the vehicle pose obtained from the previous online calibration, the matching corresponding points between the laser point cloud data and the image, and the correlation features for multi-round iterative optimization to obtain the vehicle pose of the current online calibration; obtaining the positioning information of the vehicle according to the vehicle pose of the current online calibration and the map data.

[0095] Specifically, the computer device can further extract features from the correlation features through a deep learning model. For the extracted features, perform feature matching on the image and the point cloud, predict the matching corresponding points between the laser point cloud data and the image, and calculate the vehicle pose of the current online calibration based on the predicted matching corresponding points between the laser point cloud data and the image using the BA (Bundle Adjustment) optimization method. Then, obtain the positioning information of the vehicle according to the vehicle pose of the current online calibration and the map data, and perform real-time positioning on the vehicle, thus realizing the end-to-end positioning process and also realizing the integration of vehicle online calibration and positioning.

[0096] In some embodiments, such asFigure 4 As shown, the method further includes the step of performing obstacle perception on the vehicle, specifically including:

[0097] Step 402, perform depth estimation on the image to obtain the estimated depth value of each pixel in the image.

[0098] Step 404, obtain the estimated depth map corresponding to the image according to the estimated depth value of each pixel in the image.

[0099] Step 406, project the image features into the three-dimensional space according to the camera internal parameters and the estimated depth map corresponding to the image to obtain three-dimensional image features.

[0100] Step 408, after converting the three-dimensional image features from the camera coordinate system to the point cloud coordinate system according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, fuse them with the point cloud features to obtain fused features, and obtain the obstacle perception result of the current online calibration based on the fused features.

[0101] In this embodiment, by fusing the image features with the point cloud features after converting them from the camera coordinate system to the point cloud coordinate system, three-dimensional fused features are obtained. Based on these three-dimensional fused features, dynamic targets and static obstacles can be detected to obtain the perception detection result, thereby realizing online calibration and perception integration.

[0102] In some embodiments, the sensor data further includes millimeter-wave point cloud data collected by the vehicle's millimeter-wave radar. The millimeter-wave point cloud data includes the vehicle speed. Converting the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration includes: merging the vehicle speed with the lidar point cloud data to obtain point cloud data carrying the vehicle speed; performing coordinate transformation on the point cloud data carrying the vehicle speed according to the relative external parameters between the camera and the lidar obtained from the previous online calibration to obtain the point cloud data in the camera coordinate system; projecting the point cloud in the camera coordinate system onto the image plane according to the camera internal parameters to obtain a depth map.

[0103] Among them, the millimeter-wave point cloud data is three-dimensional space information collected by the millimeter-wave radar, usually including data such as the distance, azimuth, height, and speed of the target. Therefore, the vehicle speed can be extracted from the millimeter-wave point cloud data, and the vehicle speed is merged with the lidar point cloud data to obtain point cloud data carrying the vehicle speed, which is beneficial to vehicle positioning and perception. By performing coordinate transformation on the point cloud data carrying the vehicle speed according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, the point cloud data in the camera coordinate system is obtained, thereby realizing the conversion of the point cloud data into a depth map.

[0104] Such as Figure 5 As shown, it is a block diagram of an online calibration method for a vehicle in an embodiment. Refer to Figure 5For each online calibration process, the input sensor data includes the images captured by the vehicle's camera, the lidar point cloud data collected by the vehicle's lidar, and the millimeter-wave point cloud data collected by the millimeter-wave radar. For the input image sequence, image feature extraction is performed based on a deep learning model. For the input lidar point cloud data and millimeter-wave point cloud data, through the relative external parameters between the camera and the lidar and the camera internal parameters, they are converted into depth maps in the camera coordinate system, and then image feature extraction is performed through the deep learning model to obtain point cloud features. Next, for the image features of the image sequence, higher-order camera calibration feature extraction is further performed through the deep learning model to obtain camera calibration features, and the external parameters of the camera for the current online calibration are calculated based on the camera calibration features. In addition, the correlation features between the image features and the point cloud features are calculated through a feature correlation module (such as CorrBlock). Based on the correlation features, further features are extracted through a convolutional network (such as convGRU), and the matching corresponding points between the lidar point cloud data and the image are predicted based on the extracted features. Based on the matching corresponding points, the relative external parameters between the camera and the lidar for the current online calibration are calculated, so that the external parameters of the lidar for the current online calibration can be calculated according to the external parameters of the camera for the current online calibration and the relative external parameters between the camera and the lidar. Based on the features extracted by convGRU and combined with bundle adjustment optimization, the vehicle pose change between frames is estimated, and through iterative solution, the vehicle pose is obtained to complete the motion estimation. Through the deep learning model, feature fusion is performed on the extracted image features and point cloud features to obtain fusion features, and the obstacle perception result for the current online calibration is obtained based on the fusion features.

[0105] In a detailed embodiment, an online calibration method for a vehicle is provided, including the following steps:

[0106] 1. Obtain the sensor data required for the current online calibration. The sensor data includes the images captured by the vehicle's camera, the lidar point cloud data collected by the vehicle's lidar, and the millimeter-wave point cloud data collected by the millimeter-wave radar. The millimeter-wave point cloud data includes the vehicle speed.

[0107] 2. Perform image feature extraction on the images to obtain image features.

[0108] 3. Perform camera calibration feature extraction on the image features to obtain camera calibration features.

[0109] 4. Predict the position of the vanishing point where the horizontal lines in the image intersect according to the camera calibration features, and calculate the external parameters of the camera for the current online calibration according to the vanishing point position.

[0110] 5. Combine the vehicle speed with the lidar point cloud data to obtain point cloud data carrying the vehicle speed.

[0111] 6. Based on the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, perform coordinate transformation on the point cloud data with vehicle speed to obtain the point cloud data in the camera coordinate system.

[0112] 7. Project the point cloud in the camera coordinate system onto the image plane according to the camera intrinsic parameters to obtain a depth map, and perform image feature extraction on the depth map to obtain point cloud features.

[0113] 8. Calculate the correlation features between the image features and the point cloud features.

[0114] 9. Based on the correlation features, perform corresponding point matching to predict the matching corresponding points between the lidar point cloud data and the image, and calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the matching corresponding points.

[0115] 10. According to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar, calculate the lidar extrinsic parameters for the current online calibration.

[0116] 11. Obtain the vehicle pose, the matching corresponding points between the lidar point cloud data and the image, and the correlation features obtained from the previous online calibration for multiple rounds of iterative optimization to obtain the vehicle pose for the current online calibration.

[0117] 12. According to the vehicle pose for the current online calibration and the map data, obtain the positioning information of the vehicle.

[0118] 13. Perform depth estimation on the image to obtain the estimated depth value of each pixel in the image.

[0119] 14. According to the estimated depth value of each pixel in the image, obtain the estimated depth map corresponding to the image.

[0120] 15. According to the camera intrinsic parameters and the estimated depth map corresponding to the image, project the image features into three-dimensional space to obtain three-dimensional image features.

[0121] 16. According to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, after converting the three-dimensional image features from the camera coordinate system to the point cloud coordinate system, fuse them with the point cloud features to obtain fusion features, and obtain the obstacle perception result for the current online calibration based on the fusion features.

[0122] The above online calibration method for the vehicle, based on deep learning feature extraction, provides an end-to-end online calibration process for the camera and the lidar, provides an end-to-end positioning method, and integrates the online calibration, positioning, and perception of the vehicle. It can quickly and accurately obtain the online calibration result, improve the vehicle performance, and at the same time improve the accuracy of various tasks.

[0123] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0124] Based on the same inventive concept, an embodiment of the present application further provides an on-vehicle online calibration device for implementing the above-mentioned on-vehicle online calibration method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the on-vehicle online calibration device provided below can refer to the limitations on the on-vehicle online calibration method in the above text, and will not be repeated here.

[0125] In an exemplary embodiment, as Figure 6 shown, an on-vehicle online calibration device 600 is provided, including: a data acquisition module 602, a camera extrinsic parameter determination module 604, a point cloud data processing module 606, and a lidar extrinsic parameter determination module 608, where:

[0126] The data acquisition module 602 is configured to acquire sensor data required for the current online calibration. The sensor data includes images collected by the vehicle's camera and lidar point cloud data collected by the vehicle's lidar;

[0127] The camera extrinsic parameter determination module 604 is configured to extract calibration features from the image, predict the vanishing point position in the image according to the extracted camera calibration features, and calculate the camera extrinsic parameters for the current online calibration according to the vanishing point position;

[0128] The point cloud data processing module 606 is configured to convert the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map, and obtain point cloud features;

[0129] The lidar extrinsic parameter determination module 608 is configured to calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar.

[0130] In some embodiments, the camera extrinsic parameter determination module 604 is further configured to extract image features from the image to obtain image features; extract camera calibration features from the image features to obtain camera calibration features; predict the position of the vanishing point where the horizontal lines in the image intersect according to the camera calibration features, and calculate the extrinsic parameters of the camera for the current online calibration according to the position of the vanishing point.

[0131] In some embodiments, the lidar extrinsic parameter determination module 608 is further configured to calculate the correlation features between the image features and the point cloud features; perform corresponding point matching based on the correlation features to predict the matching corresponding points between the lidar point cloud data and the image, and calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the matching corresponding points.

[0132] In some embodiments, the vehicle online calibration device 600 further includes:

[0133] A positioning module, configured to obtain the vehicle pose obtained from the previous online calibration, the matching corresponding points between the lidar point cloud data and the image, and the correlation features for multiple rounds of iterative optimization to obtain the vehicle pose for the current online calibration; obtain the positioning information of the vehicle according to the vehicle pose for the current online calibration and the map data.

[0134] In some embodiments, the vehicle online calibration device 600 further includes:

[0135] A perception module, configured to perform depth estimation on the image to obtain the estimated depth value of each pixel in the image; obtain the estimated depth map corresponding to the image according to the estimated depth value of each pixel in the image; project the image features into the three-dimensional space according to the camera intrinsic parameters and the estimated depth map corresponding to the image to obtain three-dimensional image features; fuse the three-dimensional image features with the point cloud features after converting the three-dimensional image features from the camera coordinate system to the point cloud coordinate system according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration to obtain fusion features, and obtain the obstacle perception result for the current online calibration based on the fusion features.

[0136] In some embodiments, the sensor data further includes millimeter-wave point cloud data collected by the millimeter-wave radar of the vehicle, and the millimeter-wave point cloud data includes the vehicle speed. The lidar extrinsic parameter determination module 608 is further configured to merge the vehicle speed with the lidar point cloud data to obtain point cloud data carrying the vehicle speed; perform coordinate transformation on the point cloud data carrying the vehicle speed according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration to obtain point cloud data in the camera coordinate system; project the point cloud in the camera coordinate system onto the image plane according to the camera intrinsic parameters to obtain a depth map.

[0137] The above-mentioned online calibration device 600 of the vehicle obtains the images collected by the vehicle's camera and the lidar point cloud data collected by the vehicle required for the current online calibration, extracts calibration features from the images, predicts the vanishing point position in the images based on the extracted camera calibration features, calculates the external parameters of the camera for the current online calibration according to the vanishing point position, converts the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, extracts image features from the depth map to obtain point cloud features; based on the correlation features between the image features and the point cloud features, calculates the relative external parameters between the camera and the lidar for the current online calibration, and calculates the external parameters of the lidar for the current online calibration according to the external parameters of the camera for the current online calibration and the relative external parameters between the camera and the lidar. Based on feature extraction, an end-to-end online calibration process for the camera and the lidar is provided, and the online calibration result can be obtained quickly and accurately.

[0138] Each module in the above-mentioned online calibration device 600 of the vehicle can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0139] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 7 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it realizes an online calibration method for a vehicle.

[0140] Those skilled in the art can understand that Figure 7 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps of the online calibration method for a vehicle are implemented.

[0142] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the online calibration method for a vehicle are implemented.

[0143] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the online calibration method for a vehicle are implemented.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0145] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0146] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded in this application.

[0147] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An online calibration method for a vehicle, characterized in that, The method includes: Obtaining sensor data required for the current online calibration, where the sensor data includes images collected by a camera of the vehicle and lidar point cloud data collected by the lidar of the vehicle; Performing calibration feature extraction on the image, predicting the vanishing point position in the image according to the extracted camera calibration features, and calculating the external parameters of the camera for the current online calibration according to the vanishing point position; Converting the lidar point cloud data into a depth map according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, performing image feature extraction on the depth map, and obtaining point cloud features; Calculating the relative external parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculating the external parameters of the lidar for the current online calibration according to the external parameters of the camera for the current online calibration and the relative external parameters between the camera and the lidar; 2. The method according to claim 1, characterized in that The performing calibration feature extraction on the image, predicting the vanishing point position in the image according to the extracted camera calibration features, and calculating the external parameters of the camera for the current online calibration includes: Performing image feature extraction on the image to obtain image features; Performing camera calibration feature extraction on the image features to obtain camera calibration features; Predicting the vanishing point position where the horizontal lines in the image intersect according to the camera calibration features, and calculating the external parameters of the camera for the current online calibration according to the vanishing point position; 3. The method according to claim 1, characterized in that, The method further includes: Calculating the correlation features between the image features and the point cloud features; The calculating the relative external parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features includes: Performing corresponding point matching based on the correlation features, predicting the matching corresponding points between the lidar point cloud data and the image, and calculating the relative external parameters between the camera and the lidar for the current online calibration based on the matching corresponding points; 4. The method according to claim 3, wherein The method further includes: Obtaining the vehicle pose obtained from the previous online calibration, the matching corresponding points between the lidar point cloud data and the image, and the correlation features for multiple rounds of iterative optimization to obtain the vehicle pose for the current online calibration; Obtaining the positioning information of the vehicle according to the vehicle pose for the current online calibration and the map data; 5. The method according to claim 1, wherein The method further includes: Performing depth estimation on the image to obtain the estimated depth value of each pixel in the image; Obtaining the estimated depth map corresponding to the image according to the estimated depth value of each pixel in the image; Projecting the image features into a three-dimensional space according to the camera internal parameters and the estimated depth map corresponding to the image to obtain three-dimensional image features; Converting the three-dimensional image features from the camera coordinate system to the point cloud coordinate system according to the relative external parameters between the camera and the lidar obtained from the previous online calibration, fusing them with the point cloud features to obtain fusion features, and obtaining the obstacle perception result for the current online calibration based on the fusion features; 6. The method according to claim 1, characterized in that, The sensor data further includes millimeter-wave point cloud data collected by a millimeter-wave radar of the vehicle. The millimeter-wave point cloud data includes the vehicle speed. Transforming the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration includes: Combining the vehicle speed with the lidar point cloud data to obtain point cloud data carrying the vehicle speed; Performing coordinate transformation on the point cloud data carrying the vehicle speed according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration to obtain point cloud data in the camera coordinate system; Projecting the point cloud in the camera coordinate system onto the image plane according to the camera intrinsic parameters to obtain a depth map.

7. An on-line calibration device for a vehicle, characterized in that, The device includes: A data acquisition module, configured to acquire sensor data required for the current online calibration. The sensor data includes an image collected by a camera of the vehicle and lidar point cloud data collected by a lidar of the vehicle; A camera extrinsic parameter determination module, configured to extract calibration features from the image, predict the position of a vanishing point in the image according to the extracted camera calibration features, and calculate the camera extrinsic parameters for the current online calibration according to the position of the vanishing point; A point cloud data processing module, configured to transform the lidar point cloud data into a depth map according to the relative extrinsic parameters between the camera and the lidar obtained from the previous online calibration, extract image features from the depth map to obtain point cloud features; A lidar extrinsic parameter determination module, configured to calculate the relative extrinsic parameters between the camera and the lidar for the current online calibration based on the correlation features between the image features and the point cloud features, and calculate the lidar extrinsic parameters for the current online calibration according to the camera extrinsic parameters for the current online calibration and the relative extrinsic parameters between the camera and the lidar; 8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images