Joint calibration method and system for vehicle sensors, vehicle and storage medium

Abstract

The application relates to a joint calibration method and system of a vehicle sensor, a vehicle and a storage medium, the method comprising: receiving a first plurality of image data generated by an image sensor and a second plurality of point cloud data generated by a laser radar; aligning the first plurality of image data and the second plurality of point cloud data in time; performing scene recognition on the aligned image data, and screening a third plurality of image data according to a result of the scene recognition; calibrating the image sensor and the laser radar by using the third plurality of image data and contour information in the point cloud data aligned therewith, so as to obtain a third plurality of calibration results; and deleting outliers in the third plurality of calibration results to generate valid calibration results.

Description

TECHNICAL FIELD

[0001] The present application relates to calibration of vehicle sensors, in particular, to a joint calibration method and system of vehicle sensors, a vehicle and a storage medium. BACKGROUND

[0002] In recent years, autonomous driving and assisted driving technologies have developed rapidly. LiDAR and image sensors (cameras) are the two most important sensors in the field of autonomous driving perception. The accurate extrinsic parameters between them are the basis for the work of sensor fusion and other modules. The environmental information obtained by the camera is regular, orderly and dense, but the camera is only sensitive to light and has no image distance information. In order to make up for the weakness of the camera, the LiDAR can accurately obtain the distance of the object, but the point cloud is more sparse than the image information. Therefore, the fusion of these two sensors enables autonomous vehicles to better understand the external environmental information, and has become the focus of research in recent years. The sensor fusion algorithm needs accurate extrinsic parameters to convert the coordinates between the two sensors to achieve data alignment. Therefore, online automatic calibration algorithms with high precision and high robustness become more and more important.

[0003] The current mainstream LiDAR camera extrinsic parameter calibration method highly depends on high-precision calibration targets and specific environments. In traditional calibration, calibration features are manually detected, extracted and matched from images and point clouds. These methods are time-consuming and laborious, and are not feasible in various outdoor scenarios. In addition, sensor calibration is mostly performed only once, and it is assumed that the extrinsic calibration remains unchanged for the remaining life cycle of the sensor suite. Due to the existence of calibration errors and various situations during driving, the calibration results may change.

[0004] Therefore, it is necessary to propose an online calibration algorithm to reduce the conditions required for calibration and reduce the calibration error. SUMMARY

[0005] Embodiments of the present application provide a joint calibration method and system of vehicle sensors, a vehicle and a storage medium, for jointly calibrating a camera sensor and a LiDAR of a vehicle in an online manner.

[0006] According to an aspect of the present application, a joint calibration method of vehicle sensors is provided. The method comprises: receiving a first plurality of image data generated by an image sensor and a second plurality of point cloud data generated by a lidar; aligning the first plurality of image data and the second plurality of point cloud data in time; performing scene recognition on the aligned image data, and screening a third plurality of image data according to the result of scene recognition; calibrating the image sensor and the lidar using the third plurality of image data and contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results; and deleting outliers in the third plurality of calibration results to generate valid calibration results.

[0007] In some embodiments of the present application, optionally, aligning the first plurality of image data and the second plurality of point cloud data in time comprises: using a unified time source to time stamp the first plurality of image data and the second plurality of point cloud data respectively; and aligning two frames of the first plurality of image data and the second plurality of point cloud data whose time stamp difference is below a threshold.

[0008] In some embodiments of the present application, optionally, aligning the first plurality of image data and the second plurality of point cloud data in time further comprises: discarding image data and point cloud data that fail to align.

[0009] In some embodiments of the present application, optionally, performing scene recognition on the aligned image data, and screening a third plurality of image data according to the result of scene recognition comprises: performing scene recognition on the aligned image data respectively by semantic segmentation to obtain semantic features; and screening the third plurality of image data according to the number of the semantic features and their positions in the image data.

[0010] In some embodiments of the present application, optionally, screening the third plurality of image data according to the number of the semantic features and their positions in the image data comprises: determining whether the number of the semantic features is within a preset range.

[0011] In some embodiments of the present application, optionally, screening the third plurality of image data according to the number of the semantic features and their positions in the image data comprises: determining whether the semantic features are uniformly distributed in the image.

[0012] In some embodiments of the present application, optionally, screening the third plurality of image data according to the number of the semantic features and their positions in the image data comprises: determining whether the objects corresponding to the semantic features exist mutual occlusion in the image.

[0013] In some embodiments of the present application, optionally, calibrating the image sensor and the lidar using the third multi-frame image data and the contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results comprises: extracting image contour information included in the image data based on the semantic features; extracting point cloud contour information according to the depth information in the point cloud data; and constructing an optimization problem about the matching degree of the image contour information and the point cloud contour information, and determining the third plurality of calibration results according to the solution of the optimization problem, wherein the third plurality of calibration results comprise extrinsic parameters for the image sensor and the lidar.

[0014] In some embodiments of the present application, optionally, removing outliers in the third plurality of calibration results to generate valid calibration results comprises: unsupervised classification of the third plurality of calibration results by an isolation forest algorithm and removal of isolated points therefrom as the outliers.

[0015] In some embodiments of the present application, optionally, the valid calibration results are average values of the calibration results retained in the third plurality of calibration results.

[0016] According to another aspect of the present application, there is provided a joint calibration system of vehicle sensors. The system comprises: a memory configured to store instructions; and a processor configured to execute the instructions so as to perform any one of the joint calibration methods of vehicle sensors as described above.

[0017] According to another aspect of the present application, there is provided a vehicle comprising any one of the joint calibration systems of vehicle sensors as described above.

[0018] According to another aspect of the present application, there is provided a computer-readable storage medium having instructions stored therein, wherein when the instructions are executed by a processor, the processor is caused to perform any one of the joint calibration methods of vehicle sensors as described above.

[0019] The joint calibration method of vehicle sensors, the joint calibration system of vehicle sensors, the vehicle and the computer-readable storage medium according to some embodiments of the present application can calibrate the edge features extracted from the discontinuity of lidar point cloud depth and the vehicle edge features segmented by semantic segmentation, and can remove obviously unreasonable values from a plurality of calibration results to generate valid calibration results. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects and advantages of the present application will become more apparent by describing in detail the following embodiments thereof in connection with the accompanying drawings, in which like reference numerals refer to like elements throughout the several views.

[0021] Figure 1 A joint calibration method of vehicle sensors is shown according to an embodiment of the present application;

[0022] Figure 2 A joint calibration system of vehicle sensors is shown according to an embodiment of the present application;

[0023] Figure 3 A joint calibration method of vehicle sensors is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] For brevity and illustrative purposes, the principles of the present application are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles can be equally applied to all types of joint calibration methods and systems of vehicle sensors, vehicles, and storage media, and can be implemented therein, any such variations not departing from the true spirit and scope of the present application.

[0025] In traditional calibration methods, a high-precision calibration board is usually used for joint calibration. In recent years, with the penetration of deep learning in various fields, a calibration method based on deep neural network has also been proposed. However, the above method has high requirements for the scene and is not suitable for adjusting external parameters in a regular driving scene to maximize mutual information and thus obtain optimal external parameters. The present application will propose an online calibration method which uses the edge features extracted from the depth discontinuity of the laser radar point cloud and the vehicle edge features segmented by semantic segmentation as the calibration target features, and optimizes the external parameter calibration results by alignment and minimization of the projection distance.

[0026] According to an aspect of the present application, a joint calibration method of vehicle sensors is provided. As shown in Figure 1 The joint calibration method of vehicle sensors 10 (hereinafter referred to as method 10) includes the following steps: in step S102, receiving a first plurality of image data generated by an image sensor and a second plurality of point cloud data generated by a laser radar; in step S104, aligning the first plurality of image data and the second plurality of point cloud data in time; in step S106, performing scene recognition on the aligned image data, and screening out a third plurality of image data according to the result of scene recognition; in step S108, calibrating the image sensor and the laser radar using the third plurality of image data and the contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results; and in step S110, deleting outliers in the third plurality of calibration results to generate valid calibration results. Through the above steps of method 10, simultaneous online calibration of the image sensor and the laser radar can be achieved, and the calibration does not depend on specific calibration objects (e.g., a chessboard). The working principles of each step of method 10 will be described in detail below.

[0027] The method 10 receives, in step S102, a first plurality of frames of image data generated by the image sensor and a second plurality of frames of point cloud data generated by the lidar. The terms "first plurality", "second plurality", "first plurality of frames" and "second plurality of frames" in this application are only intended to distinguish the subject matter that follows, and do not necessarily mean that they are not equal in value. In addition, unless otherwise stated or can be concluded from the context that the opposite is true, "plurality" and "plurality of frames" generally refer to "at least two" and "at least two frames". That is, for the accuracy of calibration, the image data and the point cloud data generated within a period of time are received in step S102. In an ideal case, for the convenience of calibration, the image sensor and the lidar should always acquire one frame of image data and one frame of point cloud data "simultaneously". However, in general, the sampling frequencies of the image sensor and the lidar are not the same, so the number of frames of image data and point cloud data collected at the same time is different.

[0028] To solve the problem of mismatching sampling frequencies of the image sensor and the lidar, the method 10 of this application can align the first plurality of frames of image data and the second plurality of frames of point cloud data received in step S102 in time in step S104. In other words, the purpose of step S104 is to align the image data and the point cloud data with different frame rates in time, because the calibration object of joint calibration of the two sensors should be consistent, and if there is a deviation in the scene detected by the two sensors, the result of joint calibration will be deviated, and even it will be impossible to calibrate.

[0029] Ideally, each frame of point cloud data can find a corresponding frame of image data, because the sampling rate of point cloud data is likely to be lower than that of image data, so there are more image data available for fitting point cloud data. But in actual cases, it may not be possible to find a frame of image data or point cloud data that is sampled at the same time. Therefore, it is necessary to filter out the data that can be aligned in step S104.

[0030] In some embodiments of the present application, a unified time source can be used to time stamp the first multi-frame image data and the second multi-frame point cloud data respectively in step S104, and the two frames of the first multi-frame image data and the second multi-frame point cloud data whose time stamp difference is below a threshold value are aligned. Specifically, in order to make the image data and the point cloud data correspond to each other, a common clock source needs to be set. In step S104, this clock source can be used as the clock of the image sensor and the lidar, and the current time is recorded every time a frame of data is captured. In fact, even if each frame of image data and point cloud data is time stamped, there may be very few data sampled at exactly the same time. Therefore, in step S104, a time threshold (for example, 5ms) can be set, and the image data and the point cloud data whose time difference is below this threshold are "considered" to be sampled at the same time, or "temporally aligned".

[0031] As shown in Figure 3 When looking for a frame of image data corresponding to a frame of point cloud data on the time axis, as shown in

[0032] Since only the aligned image data and point cloud data are qualified to enter the subsequent steps of method 10, the image data and point cloud data that fail to align can also be discarded in step S104. Figure 3 As shown in

[0033] Method 10 performs scene recognition on the aligned image data in step S106, and selects a third multi-frame of image data according to the result of scene recognition. As shown in Figure 3 As shown in

[0034] In some embodiments of the present application, the aligned image data can be respectively subjected to scene recognition in step S106 to obtain semantic features, and the third plurality of image data can be screened according to the number of semantic features and the positions of the semantic features in the image data. For example, it can be determined whether the number of semantic features is within a preset range, whether the semantic features are uniformly distributed in the image, and whether the objects corresponding to the semantic features are mutually occluded in the image, and the third plurality of image data can be screened according to the above determination results. Further as shown in Figure 3 , the data screened by the scene classifier (scene recognition module) will be used as key frames (the number is the third plurality described above) to enter the subsequent steps of method 10 for further processing. The scene classifier (scene recognition module) is configured as a semantic segmentation network for semantic segmentation of images, and the results of the semantic segmentation network are summarized to determine whether there are enough semantic features in the scene. In some examples, it is required that the semantic features are uniformly distributed, and there are certain requirements for the distribution position to provide sufficient constraints. In addition, it is also necessary to determine whether the semantic results are clear enough and have no obvious occlusion. At the same time, scenes that are too dense or empty can also be excluded in order to carry out subsequent algorithms.

[0035] Method 10 calibrates the image sensor and the lidar in step S108 using the third plurality of image data and the contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results.

[0036] In some embodiments of the present application, the calibration process in step S108 can be implemented by the following steps: extracting image contour information included in the image data based on the semantic features; extracting point cloud contour information according to the depth information in the point cloud data; and constructing an optimization problem about the matching degree of the image contour information and the point cloud contour information, and determining the third plurality of calibration results according to the solution of the optimization problem, wherein the third plurality of calibration results include the extrinsic parameters of the image sensor and the lidar.

[0037] Specifically, the online calibration module (denoted as "calibration module" in the figure) shown in Figure 3 uses the semantic features visible in the lidar and the image sensor to establish an optimization problem, and constructs error terms and solves the optimization problem. Among them, the vehicle contour information after semantic segmentation is used for image data, and the lateral / longitudinal discontinuity information is used for point cloud data, and the optimization and solution of the automatic calibration algorithm are performed based on the vehicle contour as the reference feature. The final calibration result includes a third plurality of extrinsic parameters corresponding to the third plurality of image data (point cloud data). As shown in Figure 3 , the extrinsic parameters can include rotation parameters θ 1 , θ 2 , θ3 Translation parameter t 1 t 2 t 3 .

[0038] Method 10 removes outliers from the third plurality of calibration results in step S110 to generate valid calibration results. In some embodiments of this application, in step S110, the isolation forest algorithm can be used to perform unsupervised classification on the third plurality of calibration results and remove isolated points as outliers. Although the alignment of image data and point cloud data has been ensured as much as possible above, due to the limitations of the algorithm, some unreasonable extrinsic parameters are still generated in step S108. Step S110 aims to filter out these unreasonable parameters and retain only reasonable parameters. Specifically, it can be done by... Figure 3 The anomaly detector shown performs unsupervised learning classification on the third and subsequent calibration results, thereby eliminating obvious outliers. The implementation of the isolation forest algorithm can be carried out using existing techniques, which will not be elaborated upon here.

[0039] In some embodiments of this application, the effective calibration result is the average of the calibration results retained among the third or more calibration results. In step S110, calibration results that all meet the screening adjustment can be obtained. In some examples, calibration results generated within a certain period of time can be selected for smoothing (e.g., taking the average), and the smoothed result can be used as an effective calibration result generated within this period of time for use by the LiDAR and image sensor.

[0040] According to another aspect of this application, a joint calibration system for vehicle sensors is provided. For example... Figure 2 As shown, the joint calibration system 20 for vehicle sensors (hereinafter referred to as system 20) includes a memory 202 and a processor 204. The memory 202 is configured to store instructions, and the processor 204 is configured to execute instructions to perform any of the joint calibration methods for vehicle sensors described above.

[0041] According to another aspect of this application, a vehicle is provided, the vehicle including a joint calibration system of any of the vehicle sensors as described above. This application does not limit the vehicle's layout (e.g., wheeled vehicle, tracked vehicle, etc.) or the vehicle's driving force (e.g., electric motor drive, gasoline engine drive, etc.), and the vehicle covered by this application encompasses various vehicles currently known in the art as well as vehicles developed in the future.

[0042] According to another aspect of the present application, there is provided a computer readable storage medium having stored therein instructions which, when executed by a processor, cause the processor to perform any of the joint calibration methods of vehicle sensors as described above. The computer readable medium as used in the present application includes various types of computer storage media that can be accessed by a general purpose or special purpose computer. For example, computer readable media can include RAM, ROM, EEPROM, E 2 PROM, registers, hard disks, removable disks, CD-ROMs or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Disk typically magnetically copy data while discs optically copy data using a laser. Combinations thereof can also be included within the scope of computer readable media. A tangible storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0043] Some embodiments of the present application above propose an online calibration algorithm framework that is automatically run, which realizes the automatic running of calibration algorithms in all working conditions by introducing semantic information and combining scene information and outlier processing. It should be noted that the above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art can think of other feasible changes or replacements according to the technical range disclosed in the present application, and such changes or replacements are all included in the protection scope of the present application. The embodiments of the present application and the features in the embodiments can also be combined with each other without conflict. The protection scope of the present application is subject to the description of the claims.

Claims

1. A method of joint calibration of vehicle sensors, characterized in that, The method comprises: receiving a first plurality of frames of image data generated by an image sensor and a second plurality of frames of point cloud data generated by a lidar; aligning the first plurality of frames of image data and the second plurality of frames of point cloud data in time, wherein aligning the first plurality of frames of image data and the second plurality of frames of point cloud data in time comprises: time stamping the first plurality of frames of image data and the second plurality of frames of point cloud data respectively using a unified time source; and aligning two frames of the first plurality of frames of image data and the second plurality of frames of point cloud data whose time stamp difference is below a threshold value; performing scene recognition on the aligned image data and filtering a third plurality of frames of image data according to the result of the scene recognition; calibrating the image sensor and the lidar using the third plurality of frames of image data and contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results respectively; and removing outliers in the third plurality of calibration results to generate valid calibration results.

2. The method of claim 1, wherein, Aligning the first plurality of frames of image data and the second plurality of frames of point cloud data in time further comprises discarding image data and point cloud data that fail to align.

3. The method of claim 1, wherein, Performing scene recognition on the aligned image data and filtering a third plurality of frames of image data according to the result of the scene recognition comprises: performing scene recognition on the aligned image data respectively through semantic segmentation to obtain semantic features; and filtering the third plurality of frames of image data according to the number of the semantic features and their positions in the image data.

4. The method of claim 3, wherein, Filtering the third plurality of frames of image data according to the number of the semantic features and their positions in the image data comprises determining whether the number of the semantic features is within a preset range.

5. The method of claim 3, wherein, Filtering the third plurality of frames of image data according to the number of the semantic features and their positions in the image data comprises determining whether the semantic features are uniformly distributed in the image.

6. The method of claim 3, wherein, Filtering the third plurality of frames of image data according to the number of the semantic features and their positions in the image data comprises determining whether objects corresponding to the semantic features are mutually occluded in the image.

7. The method of claim 3, wherein, Calibrating the image sensor and the lidar using the third plurality of frames of image data and contour information in the point cloud data aligned therewith to obtain a third plurality of calibration results respectively comprises: extracting image contour information included in the image data based on the semantic features; extracting point cloud contour information according to depth information in the point cloud data; and constructing an optimization problem about the matching degree of the image contour information and the point cloud contour information, and determining the third plurality of calibration results according to the solution of the optimization problem, wherein the third plurality of calibration results comprise extrinsic parameters for the image sensor and the lidar.

8. The method of claim 1, wherein, Removing outliers in the third plurality of calibration results to generate valid calibration results comprises: performing unsupervised classification on the third plurality of calibration results through an isolation forest algorithm and removing isolated points therefrom as the outliers.

9. The method of claim 1, wherein, The valid calibration results are average values of the calibration results retained in the third plurality of calibration results.

10. A joint calibration system for vehicle sensors, characterized in that, The system comprises: a memory configured to store instructions; and a processor configured to execute the instructions so as to perform the method of any of claims 1-9.

11. A vehicle characterized by comprising: The vehicle comprises the system of claim 10.

12. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: The instructions, when executed by a processor, cause the processor to perform the method of any of claims 1-9.

Images