A self-supervised learning method for holographic data based on vehicle-road collaboration

Abstract

The present invention discloses a holographic data self-supervised learning method based on vehicle-road collaboration, which relates to the technical field of road traffic. Complete holographic data is obtained through roadside unit devices; holographic data of the road surface environment is collected, and a small amount of holographic data is manually labeled to train a perception detection model; through time synchronization, point cloud matching, and point cloud stitching methods, the original sensor data of the driverless vehicle on the same road surface at the same time is aligned with the holographic data of the roadside unit; through data alignment, the original sensor data perceived by the driverless vehicle in the same space-time can be matched with the results of the perception detection model trained by the holographic data of the roadside unit. The detection results of the holographic data perception detection model of the roadside unit are used as the annotation results of the perception data of the driverless vehicle in the same space-time for supervised learning to train the perception model at the driverless vehicle end; through the process of self-supervised learning of the driverless vehicle with the holographic data of the roadside unit in different scenarios and at different times, automatic and efficient model iteration is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of road traffic, and particularly to a self-supervised learning method for holographic data based on vehicle-road cooperation. Background Art

[0002] L4-level driverless depends on the high-precision perception of the surrounding environment by the system. At present, the mainstream perception algorithms mainly adopt the method of supervised learning, and the unsupervised learning is far from meeting the requirements of practical applications in terms of accuracy and efficiency. When using the unsupervised learning method for learning, it is currently difficult to compare with the results of supervised learning in terms of time accuracy and efficiency. When using the data migration method for learning, migrating the data set in other environments to the existing environment for learning, currently the effect of this method is the same as 1, and it is difficult to compare with the results of supervised learning in terms of accuracy and time efficiency. The most prominent feature of supervised learning is its dependence on a large amount of high-quality labeled data, and the labeled data usually takes a lot of time and effort.

[0003] The existing perception detection is based on single-frame images, point clouds or images + point clouds, and is carried out by means of manual annotation. For supervised learning, this requires a large number of manually annotated results. Manual annotation is difficult, cumbersome and boring, with low annotation efficiency, low and unstable annotation quality. Taking point cloud annotation as an example, due to the sparsity and occlusion problems of single-frame point clouds, manual annotation cannot accurately annotate the specific size of the actual object to be detected. Therefore, point cloud annotation is usually completed based on the experience of annotators, which also leads to inaccurate and unstable annotation results. In addition, the existing technology usually requires an extremely large number of annotation results, usually reaching more than one million in magnitude. Completing the above annotation volume requires a large amount of time, manpower and material resources, with a long annotation time and high annotation cost. Summary of the Invention

[0004] In view of the above technical problems, the present invention overcomes the disadvantages of the prior art and provides a self-supervised learning method for holographic data based on vehicle-road cooperation, including:

[0005] S1. Obtain complete holographic data through roadside unit devices;

[0006] S2. Collect holographic data of the road surface environment and a small amount of high-quality holographic data with manual annotation;

[0007] S3. Use a small amount of high-quality labeled data with manual annotation to train a deep learning artificial neural network;

[0008] S4. Through time synchronization, point cloud matching, and point cloud stitching methods, align the original data of the sensors of driverless vehicles on the same road surface at the same time with the holographic data of the roadside unit;

[0009] S5. Through data alignment, the original sensor data sensed by driverless vehicles in the same time and space can be matched with the perception detection results of the deep learning artificial neural network trained based on the holographic data of roadside units. The holographic data detection results of the roadside units are used as the annotation results of the perception data of driverless vehicles in the same time and space for supervised learning to train the perception model at the driverless vehicle end.

[0010] S6. Through the process of self-supervised learning of the detection results of driverless vehicles and the holographic data of roadside units in different scenarios and at different times, automatic and efficient model iteration is achieved.

[0011] The further limited technical solution of the present invention is:

[0012] In the above-mentioned self-supervised learning method for holographic data based on vehicle-road collaboration, in step S1, a roadside perception system is deployed by the roadside. Through time synchronization and sensor calibration, data from different devices and different types of sensors are fused at the data level to obtain holographic data information of the environment within the coverage range.

[0013] In the above-mentioned self-supervised learning method for holographic data based on vehicle-road collaboration, the roadside perception system includes cameras, lidar, and millimeter-wave radars.

[0014] In the above-mentioned self-supervised learning method for holographic data based on vehicle-road collaboration, step S1 is specifically:

[0015] Through GPS time synchronization, the original data of each sensor at the same moment is obtained. Since the frequencies of different sensors are different, taking 100 ms as a time window, the original data within the same time window is used as the original data at the same moment.

[0016] By using the original data at the same moment, the sensors of different roadside units are calibrated to obtain the spatial geometric relationship between different sensors.

[0017] Through the calibration parameters, the data of different sensors are pre-fused, and through multiple roadside units, the data volume is multiplied.

[0018] In the above-mentioned self-supervised learning method for holographic data based on vehicle-road collaboration, the sensor data includes image information, point cloud information, and millimeter-wave information.

[0019] In the above-mentioned self-supervised learning method for holographic data based on vehicle-road collaboration, in step S2, the pre-fused data of a small number of roadside units is annotated to obtain the annotation true values, and these true values are used to train a detection model.

[0020] The above-mentioned self-supervised learning method for holographic data based on vehicle-road cooperation, the input of the detection model includes point cloud information, image information, and radar information.

[0021] The above-mentioned self-supervised learning method for holographic data based on vehicle-road cooperation, step S4 is specifically as follows:

[0022] Through GPS satellite timing, the sensor data on the driverless vehicle and the sensor data of the roadside unit are under the same clock source to ensure the unity of time;

[0023] Taking the self-positioning information of the driverless vehicle as the initial value, align the point cloud data on the driverless vehicle with the holographic point cloud data on the roadside unit.

[0024] The above-mentioned self-supervised learning method for holographic data based on vehicle-road cooperation uses point cloud registration technology to splice two point cloud data from different sources in space.

[0025] The above-mentioned self-supervised learning method for holographic data based on vehicle-road cooperation, in step S5, fuse the annotation boxes on the roadside unit data, which also includes the data of the driverless vehicle. Taking the roadside unit annotation box as the ground truth, provide supervision information for the data of the driverless vehicle.

[0026] The beneficial effects of the present invention are:

[0027] (1) Through the self-supervised learning method for holographic data based on vehicle-road cooperation, the present invention can provide a large amount of high-quality annotation data for the driverless vehicle only by using a small amount of high-precision annotation data and the vehicle-road cooperation method. Through supervised learning, on the basis of a large amount of automatically extracted high-precision data, the purpose of quickly iterating the model is achieved;

[0028] (2) By using the holographic data generated by the roadside unit, based on the annotation results of a small amount of holographic data, first train a relatively high-level detection model on the holographic data, and use the detection results of this detection model on the holographic data as supervision to self-annotate the data collected by the driverless vehicle itself;

[0029] (3) Compared with the common single-frame point cloud data, the holographic data in the present invention has the advantages of high point cloud density, more complete and clear contours of detected objects, and fused image texture information and millimeter-wave speed information. By obtaining high-quality original data, the learning difficulty of the detection model is greatly reduced, and the model can quickly converge with only a small amount of artificial high-quality annotation. Specific embodiments

[0030] A self-supervised learning method for holographic data based on vehicle-road cooperation provided in this embodiment includes:

[0031] S1. Obtain complete holographic data through roadside unit devices. Install a roadside perception system (cameras, lidar, millimeter-wave radar) by the roadside. Through time synchronization and sensor calibration, fuse the data of different devices and different types of sensors at the data level to obtain holographic data information of the environment within the coverage range. Specifically:

[0032] Through GPS time synchronization, obtain the raw data of each sensor at the same moment. Different sensors have different frequencies. Taking 100 ms as a time window, use the raw data within the same time window as the raw data at the same moment;

[0033] By using the raw data at the same moment, calibrate the sensors of different roadside units to obtain the spatial geometric relationship between different sensors. The sensor data includes image information, point cloud information, and millimeter-wave information;

[0034] Through the calibration parameters, perform pre-fusion on the data of different sensors. For example, expand the point cloud data from three channels of x, y, and z to multi-dimensional data such as x, y, z, intensity, ring, timestamp, r, g, b, etc. Through multiple roadside units, multiply the data volume;

[0035] S2. Collect holographic data of the road surface environment, use a small amount of high-quality holographic data manually labeled to train a perception detection model. Label the pre-fusion data of a small number of roadside units to obtain labeled ground truth, and use this ground truth to train a detection model. The input of the detection model includes point cloud information, image information, and radar information;

[0036] S3. Use a small amount of high-quality labeled data manually labeled to train a deep learning artificial neural network;

[0037] S4. Through time synchronization, point cloud matching, and point cloud stitching methods, align the raw data of the sensors of the driverless vehicle on the same road surface at the same moment with the holographic data of the roadside unit. Specifically:

[0038] Through GPS satellite time synchronization, make the sensor data on the driverless vehicle and the sensor data of the roadside unit under the same clock source to ensure the unity of time;

[0039] Taking the self-positioning information of the driverless vehicle as the initial value, align the point cloud data on the driverless vehicle with the holographic point cloud data on the roadside unit, and use point cloud registration technology to splice the two pieces of point cloud data from different sources in space;

[0040] S5. Through data alignment, the original sensor data perceived by driverless vehicles in the same time and space can be matched with the perception detection results of the deep learning artificial neural network trained based on the holographic data of roadside units in step S2. The detection results of the holographic data perception detection model of the roadside unit are used as the annotation results of the perception data of driverless vehicles in the same time and space for supervised learning to train the perception model on the driverless vehicle side; the annotation boxes in the roadside unit fusion data also contain the data of driverless vehicles. Using the roadside unit annotation boxes as the ground truth, supervision information is provided for the data of driverless vehicles.

[0041] S6. Through the process of self-supervised learning of driverless vehicles with the holographic data of roadside units in different scenarios and at different times, automatic and efficient model iteration is achieved.

[0042] Specific application case:

[0043] On Shuijing Road in Suzhou High-Speed Railway New City, 12 roadside unit devices are deployed, covering a 1.4-km-long road. Through the data pre-fusion of the upper lidar, camera, and radar of the roadside unit, a batch of data is annotated. This data includes information such as the positions, headings, and categories of vehicles, pedestrians, and cyclists at different times and positions on this road. Using this data, the current mainstream pre-fusion deep learning network EPNet is trained. The trained EPNet can offline detect the continuous holographic data of roadside units and give detection results (BoundingBox, class, score, etc.). Align the lidar and camera data collected by the driverless vehicle at the same moment with the holographic data of the roadside unit, and use the detection results of EPNet as the annotation results of the vehicle-side data for supervised learning of the vehicle-side real-time online model.

[0044] In the case of a small amount of annotation, through the above vehicle-road collaboration and time-space alignment method, the detection results of holographic data are used to supervise the self-vehicle learning. Based on the automatically extracted massive high-precision supervision labels, the purpose of quickly iterating the detection model on the driverless vehicle is achieved.

[0045] In addition to the above embodiments, the present invention may have other implementation manners. Any technical solutions formed by equivalent replacement or equivalent transformation fall within the protection scope required by the present invention.

Claims

1. A self-supervised learning method for holographic data based on vehicle-road collaboration, characterized in that: It includes: S1. Obtain complete holographic data through roadside unit devices; S2. Collect holographic data of the road surface environment and a small amount of high-quality holographic data with manual annotation; S3. Use a small amount of high-quality annotated data with manual annotation to train a deep learning artificial neural network; S4. Through time synchronization, point cloud matching, and point cloud stitching methods, align the original data of the sensors of driverless vehicles on the same road surface at the same time with the holographic data of the roadside unit; S5. Through data alignment, the original sensor data perceived by the driverless vehicle in the same space-time can be matched with the perception detection results of the deep learning artificial neural network trained based on the roadside unit holographic data in step S2. Use the holographic data detection results of the roadside unit as the annotation results of the perception data of the driverless vehicle in the same space-time for supervised learning, and train the perception model at the driverless vehicle end; S6. Through the process of self-supervised learning of the detection results of the driverless vehicle and the roadside unit holographic data in different scenarios and at different times, achieve automatic and efficient model iteration.

2. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 1, characterized in that: In step S1, a roadside perception system is arranged on the roadside, and through time synchronization and sensor calibration, data from different devices and different types of sensors are fused at the data level to obtain holographic data information of the environment within the coverage range.

3. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 2, characterized in that: The roadside perception system includes a camera, a lidar, and a millimeter-wave radar.

4. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 2, characterized in that: Step S1 is specifically: Through GPS time synchronization, obtain the original data of each sensor at the same time. Since the frequencies of different sensors are different, taking 100 ms as a time window, the original data within the same time window is used as the original data at the same time; By using the original data at the same time, calibrate the sensors of different roadside units to obtain the spatial geometric relationship between different sensors; Through the calibration parameters, perform pre-fusion of the data of different sensors, and multiply the data volume through multiple roadside units.

5. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 4, characterized in that: The sensor data includes image information, point cloud information, and millimeter-wave information.

6. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 1, characterized in that: In step S2, annotate the pre-fusion data of a small number of roadside units to obtain annotation ground truths, and use these ground truths to train a detection model.

7. A self-supervised learning method for holographic data based on vehicle-road collaboration according to claim 6, characterized in that: The input of the detection model includes point cloud information, image information, and radar information.

8. A holographic data self-supervised learning method based on vehicle-road cooperation according to claim 1, characterized in that: The step S4 is specifically: Through GPS satellite timing, the sensor data on the driverless vehicle and the sensor data of the roadside unit are under the same clock source to ensure the unity of time; Taking the self-positioning information of the driverless vehicle as the initial value, the point cloud data on the driverless vehicle is aligned with the holographic point cloud data on the roadside unit.

9. A holographic data self-supervised learning method based on vehicle-road cooperation according to claim 8, characterized in that: Using point cloud registration technology, two pieces of point cloud data from different sources are stitched together in space.