Dynamic obstacle point cloud correction method and system under 4D truth value labeling

Abstract

The invention relates to the technical field of unmanned driving, and particularly discloses a dynamic obstacle point cloud correction method and system under 4D truth value labeling, and the method comprises the steps: obtaining point cloud 3D labeling information, vehicle body information ego, laser radar-to-vehicle body external parameters, a camera timestamp, and a timestamp of each point of the point cloud; performing time synchronization processing on the point cloud data; converting the point cloud data under the laser radar coordinate system into a vehicle body coordinate system, and further converting the point cloud data into a unified world coordinate system; after the original point cloud is subjected to distortion removal (the point cloud of the same frame is synchronized with the odo information according to time), detecting and tracking a dynamic obstacle; calculating a velocity vector of each dynamic target in a world coordinate system according to the associated point cloud data; calculating displacement according to the speed and the time difference of the dynamic obstacle, and performing displacement compensation on the point cloud of the dynamic obstacle; updating annotation information in the 4D value annotation system, and performing consistency check. According to the method, the point cloud of the dynamic obstacle is accurately corrected, and the unmanned 4D truth value marking precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a method and system for correcting a dynamic obstacle point cloud under 4D true value annotation. Background Art

[0002] In unmanned driving systems, environmental perception is one of the core links, among which point cloud data, as an important source of environmental information, is crucial for identifying obstacles such as roads, vehicles, pedestrians, etc. However, during the acquisition and processing of point cloud data, due to the inherent characteristics of scanning equipment (such as lidar) and changes in the dynamic environment, the timestamps of point cloud data are often inconsistent, especially when converted to the camera coordinate system or multi-sensor fusion, this time asynchrony can lead to significant deviations.

[0003] Traditional point cloud correction methods mainly perform preliminary correction based on the timestamp of each point and the position information of the vehicle, but this is still not accurate enough for dynamic obstacles, because dynamic obstacles themselves are also moving, and their positions change over time. Static correction alone cannot accurately reflect their true positions. Therefore, it is particularly important to develop a method that can effectively correct the point cloud of dynamic obstacles. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for correcting dynamic obstacle point clouds under 4D value annotation, which can achieve accurate correction of dynamic obstacle point clouds by comprehensively considering the motion information of dynamic obstacles and improve the accuracy of unmanned driving 4D true value annotation.

[0005] The present invention adopts the following technical solution: The method for correcting the dynamic obstacle point cloud under 4D true value annotation includes the following steps: S1, obtaining point cloud 3D annotation information, vehicle body information ego, laser radar to vehicle body external parameters, camera timestamp and timestamp of each point in the point cloud, wherein the step of obtaining point cloud 3D annotation information includes: the collected point cloud data is first pre-processed and then 3D annotated, the annotation information includes the attributes of the dynamic obstacle, and a unique identifier (ID) used to associate the same dynamic obstacle in consecutive frames, the attributes include position, shape, size, etc.; Point cloud preprocessing: The collected point cloud data (point cloud data driven by radar sensors) is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, the timestamps of point cloud data may be inconsistent due to differences in different radar sensors or scanning cycles. For static obstacles, preliminary timestamp compensation can be performed on each point cloud point based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis.

[0006] S2, converting the point cloud data in the laser radar coordinate system to the vehicle coordinate system, and further converting it to a unified world coordinate system; S3, tracking dynamic obstacles in the point cloud data processed by S2; S4. Calculate the velocity vector of each dynamic obstacle in the world coordinate system according to the annotation information; S5. Calculate the displacement according to the speed and time difference of the dynamic obstacle, and perform displacement compensation on the point cloud of the dynamic obstacle; S6. Update the annotation information in the 4D value annotation system and perform a consistency check.

[0007] As a further technical solution of the present invention, in step S1, obtaining the vehicle body information ego includes the following steps: The vehicle's real-time posture information, including position, speed, acceleration and direction, is obtained through on-board sensors. The real-time posture information will be used for subsequent point cloud coordinate conversion and displacement compensation. Obtaining the external parameters of the laser radar to the vehicle body: The relative position relationship / external parameters between the laser radar and the vehicle body are fixed, but need to be obtained through the calibration process. The external parameters include the rotation matrix and translation vector of the laser radar relative to the vehicle body, which are used to convert the point cloud data in the laser radar coordinate system to the vehicle body coordinate system; Get camera timestamp: When the camera collects image data, it will be accompanied by timestamp information; Get the timestamp of each point in the point cloud: When the lidar collects point cloud data, it also assigns a timestamp to each point.

[0008] As a further technical solution of the present invention, step S3 includes the following steps: After performing preliminary timestamp compensation on each point cloud point so that they can be compared on the same time basis, the point cloud deviation is processed according to the ego information of the self-vehicle to obtain new point cloud data, and then the new point cloud data will be 3D annotated. Since the annotation information itself has ID information, it can be directly associated with the ID data of the previous and next frames to match the targets of the previous and next frames.

[0009] As a further technical solution of the present invention, step S4 includes the following steps: According to the associated point cloud data, the annotation information is transferred from the radar coordinate system to the vehicle coordinate system and then to the world coordinate system. According to the time difference between the previous and next frames and the 3D position transformation of the world coordinate system, the speed of the target in the world coordinate system is calculated using the Kalman filter.

[0010] As a further technical solution of the present invention, step S5 includes the following steps: For dynamic obstacles, the displacement of the dynamic obstacle within the time difference is calculated based on the time difference between the original timestamp of each point and the current processing time, as well as the calculated speed of the dynamic obstacle. Then, the point cloud of each dynamic obstacle is compensated for displacement, that is, its original position vector is added with the calculated displacement to obtain the corrected point cloud position.

[0011] As a further technical solution of the present invention, step S6 includes the following steps: Update the corrected point cloud position to the original 4D value annotation system, where the 4D value includes the 3D space coordinates and timestamp; After updating the annotation information, a consistency check is performed to ensure the consistency of all annotation information in time and space. This can be done by comparing the projection of the corrected point cloud onto the image with the projection of the uncorrected point cloud onto the image to see if the dynamic obstacle point cloud is improved.

[0012] Another object of the present invention is to provide a dynamic obstacle point cloud correction system under 4D true value annotation, the system comprising: Point cloud preprocessing: The collected point cloud data is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, for static obstacles, preliminary timestamp compensation is performed on each point cloud point based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis. The coordinate conversion module is used to convert the point cloud data in the laser radar coordinate system to the vehicle coordinate system, and further to the unified world coordinate system; Dynamic obstacle tracking module, used to track dynamic obstacles in preprocessed point cloud data; The velocity calculation module calculates the velocity vector of each dynamic obstacle in the world coordinate system based on the annotation information; The displacement compensation module is used to calculate the displacement according to the speed and time difference of the dynamic obstacle and to perform displacement compensation on the point cloud of the dynamic obstacle; The annotation update module is used to update the annotation information in the 4D value annotation system and perform consistency checks.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. Significantly improve the positioning accuracy of dynamic obstacles: By comprehensively considering the motion information of dynamic obstacles (such as speed, acceleration, etc.), the present invention can make more accurate corrections to point cloud data. Traditional point cloud correction methods often only consider the changes in the vehicle's posture, while ignoring the movement of the dynamic obstacles themselves, resulting in large positioning errors in complex dynamic environments. The present invention effectively reduces this error by calculating the speed of the dynamic target in the world coordinate system and performing displacement compensation based on the time difference, significantly improving the positioning accuracy of dynamic obstacles.

[0014] 2. Enhance the environmental perception capability of unmanned driving systems: Unmanned driving systems need to perceive the surrounding environment in real time and accurately in order to make correct decisions and control. The present invention enables unmanned driving systems to more accurately identify and understand changes in dynamic environments, such as vehicle acceleration, deceleration, and steering, by accurately correcting dynamic obstacle point clouds, thereby enhancing the system's environmental perception capabilities.

[0015] 3. Improve the effect of multi-sensor fusion: In unmanned driving systems, the fusion of multiple sensors (such as lidar, camera, millimeter wave radar, etc.) is an important means to improve the accuracy of environmental perception. However, due to the differences in the working principles and data formats of different sensors, data mismatch problems often occur during the fusion process. The present invention uses a unified world coordinate system and precise displacement compensation to enable the dynamic obstacle point clouds collected by different sensors to correspond more accurately, thereby improving the effect of multi-sensor fusion.

[0016] 4. Improve the safety and reliability of unmanned driving systems: Accurate environmental perception is the basis of the safety and reliability of unmanned driving systems. Through the present invention, unmanned driving systems can more accurately identify dynamic obstacles and avoid dangerous situations such as collisions. At the same time, due to the precise correction of the point cloud of dynamic obstacles, the system's response speed to emergencies will also be improved, further enhancing the safety and reliability of unmanned driving systems.

[0017] 5. Promote the commercial application of driverless technology: With the continuous development of unmanned driving technology, its commercial application has become an inevitable trend. However, the commercial application of unmanned driving technology needs to meet strict safety standards and regulatory requirements. The present invention provides a more solid technical support for the commercial application of unmanned driving technology by improving the correction accuracy of dynamic obstacle point clouds, which helps to promote the rapid development and widespread application of unmanned driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is an overall flow chart of a method for correcting a dynamic obstacle point cloud under 4D value annotation in the present invention.

[0019] Figure 2 Schematic diagram of displacement compensation in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 to Figure 2 ,The dynamic obstacle point cloud correction method under 4D true value annotation includes the following steps: S1. Obtain point cloud 3D annotation information, vehicle body information ego, laser radar to vehicle body external parameters, camera timestamp and timestamp of each point in the point cloud. The steps of obtaining point cloud 3D annotation information include: using professional annotation tools or platforms, the collected point cloud data is first pre-processed and then 3D annotated. The annotation information includes the location, shape, size and other attributes of the dynamic obstacle, as well as a unique identifier (ID) used to associate the same dynamic obstacle in consecutive frames; dynamic obstacles refer to 3D targets that need to be sensed and detected, and the targets are in motion (such as walking people, moving vehicles, etc.).

[0022] Point cloud preprocessing: The collected point cloud data (point cloud data driven by radar sensors) is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, the timestamps of point cloud data may be inconsistent due to differences in different radar sensors or scanning cycles. For static obstacles, each point cloud point can be preliminarily compensated for the timestamp based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis. Although this step cannot completely solve the deviation problem of dynamic obstacles, it provides a more consistent data basis for subsequent processing. This usually involves steps such as interpolation, extrapolation, or resampling of timestamps to ensure the temporal consistency of all data.

[0023] Obtaining vehicle body information ego includes the following steps: obtaining the vehicle's real-time posture information, including position, speed, acceleration, and direction, through on-board sensors (such as IMU, GPS, etc.). The real-time posture information will be used for subsequent point cloud coordinate conversion and displacement compensation; Obtaining the external parameters of the laser radar to the vehicle body: The relative position relationship / external parameters between the laser radar and the vehicle body are fixed, but need to be obtained through the calibration process. The external parameters include the rotation matrix and translation vector of the laser radar relative to the vehicle body, which are used to convert the point cloud data in the laser radar coordinate system to the vehicle body coordinate system; Get camera timestamp: When the camera collects image data, it will be accompanied by timestamp information. These timestamps are used to synchronize camera images with lidar point cloud data for subsequent multi-sensor fusion and data processing; Get the timestamp of each point in the point cloud: When the lidar collects point cloud data, it will also assign a timestamp to each point. These timestamps are used to record the collection time of the point cloud data and are an important basis for subsequent time synchronization and displacement compensation.

[0024] S2. Convert the point cloud data in the laser radar coordinate system to the vehicle body coordinate system, and further convert it to a unified world coordinate system. This step requires the use of the laser radar body external parameters and vehicle body information ego obtained in the early stage for calculation.

[0025] S3, tracking dynamic obstacles in the point cloud data processed by S2; In the pre-processed point cloud data, we will perform a timestamp compensation on the point cloud (static obstacles). This is because when collecting point cloud data, due to differences in different sensors or scanning cycles, the timestamps of the point cloud data are inconsistent. We can perform preliminary timestamp compensation on each point cloud point based on the posture information and timestamp difference of the ego vehicle, so that they can be compared on the same time basis, and then process the point cloud deviation based on the ego information of the ego vehicle to obtain new point cloud data. Then, we will perform 3D annotation on the new point cloud data. Since the annotation information itself has id information, it can be directly associated with the id data of the previous and next frames to match the previous and next frame targets. S4. According to the annotation information, such as ID, the dynamic obstacles annotated in the previous and next frames can be associated, and the velocity vector of each dynamic obstacle (dynamic target) in the world coordinate system is calculated according to the annotation information; According to the associated dynamic obstacle information, the annotation information is transferred from the radar coordinate system to the vehicle coordinate system and then to the world coordinate system. According to the time difference between the previous and next frames and the 3D position transformation of the world coordinate system, the speed of the target in the world coordinate system is calculated using the Kalman filter.

[0026] S5. Calculate the displacement according to the speed and time difference of the dynamic obstacle, and perform displacement compensation on the point cloud of the dynamic obstacle; Specifically, for dynamic obstacles, the displacement of the dynamic target within the time difference is calculated based on the time difference between the original timestamp of each point and the current processing time, as well as the calculated dynamic target speed (dynamic obstacle speed). Then, displacement compensation is performed on the point cloud of each dynamic obstacle, that is, its original position vector is added with the calculated displacement to obtain the corrected point cloud position.

[0027] Figure 2Where obj is the annotated 3D frame information. At this time, the 3D frame information is the information in the lidar coordinate system (radar coordinate system). First, the obj position information needs to be converted to the ego coordinate system (vehicle body coordinate system) (calculated based on the selected translation external parameter information from lidar to vehicle body). Secondly, according to the current self-vehicle posture information, it is converted to the coordinate system Wx Wy in the figure, the world coordinate system. At this time, the (x, y, z) of obj in the world coordinate system (world coordinate system) can be obtained. According to the continuous annotation information, the speed of obj in the world coordinate system at that time can be estimated. The displacement of dynamic obstacles can be compensated according to the time difference. After the conversion is completed, it is necessary to convert from the world back to the ego coordinate system according to the self-vehicle posture, and then convert it to the lidar coordinate system through external parameters. This calculation needs to be done for each point of the dynamic obstacle, and finally the calculation result is given to the new point cloud file to complete the displacement compensation.

[0028] S6. Update the annotation information in the 4D value annotation system and perform a consistency check.

[0029] Specifically, it includes: updating the corrected point cloud position to the original 4D value annotation system. The 4D value includes three-dimensional space coordinates and timestamps, which is a commonly used environment description method in unmanned driving systems. The updated annotation information is equivalent to our perceived "true value", which provides a basis for subsequent perception result verification and quantification, and provides more accurate information for subsequent path planning, decision-making control and other modules; After updating the annotation information, a consistency check is performed to ensure the consistency of all annotation information in time and space. This can be done by comparing the projection of the corrected point cloud onto the image with the projection of the uncorrected point cloud onto the image to see if the dynamic obstacle point cloud is improved.

[0030] Another object of the present invention is to provide a dynamic obstacle point cloud correction system under 4D true value annotation, the system comprising: The data acquisition module is used to obtain the point cloud 3D annotation information, the vehicle body information ego, the laser radar to the vehicle body external parameters, the camera timestamp and the timestamp of each point in the point cloud. The steps of obtaining the point cloud 3D annotation information include: using a professional annotation tool or platform, the collected point cloud data is first pre-processed and then 3D annotated. The annotation information includes the location, shape, size and other attributes of the dynamic obstacle, and the unique identifier (ID) is used to associate the same dynamic obstacle in consecutive frames; Point cloud preprocessing: The collected point cloud data (point cloud data driven by radar sensors) is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, the timestamps of point cloud data may be inconsistent due to differences in different radar sensors or scanning cycles. For static obstacles, each point cloud point can be preliminarily compensated for the timestamp based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis. Although this step cannot completely solve the deviation problem of dynamic obstacles, it provides a more consistent data basis for subsequent processing. This usually involves steps such as interpolation, extrapolation, or resampling of timestamps to ensure the temporal consistency of all data.

[0031] The coordinate conversion module is used to convert the point cloud data in the laser radar coordinate system to the vehicle body coordinate system, and further to the unified world coordinate system; this step requires the use of the laser radar body external parameters and vehicle body information ego obtained in the early stage for calculation.

[0032] Dynamic obstacle tracking module, used to detect and track dynamic obstacles in preprocessed point cloud data; The speed calculation module is used to calculate the speed vector of each associated dynamic obstacle in the world coordinate system based on the 3D annotation information that will give an ID. The dynamic obstacles can be associated based on the same IDs in the previous and next frames. The displacement compensation module is used to calculate the displacement according to the speed and time difference of the dynamic obstacle and to perform displacement compensation on the point cloud of the dynamic obstacle; The annotation update module is used to update the annotation information in the 4D value annotation system and perform consistency checks.

[0033] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

[0034] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. The dynamic obstacle point cloud correction method under 4D true value annotation is characterized by: The following steps are involved: S1, obtaining point cloud 3D annotation information, vehicle body information ego, laser radar to vehicle body external parameters, camera timestamp and timestamp of each point in the point cloud, wherein the step of obtaining point cloud 3D annotation information includes: the collected point cloud data is first pre-processed and then 3D annotated, the annotation information includes the attributes of the dynamic obstacle, and a unique identifier used to associate the same dynamic obstacle in consecutive frames, the attributes include position, shape, size, etc.; Point cloud preprocessing: The collected point cloud data is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, for static obstacles, preliminary timestamp compensation is performed on each point cloud point based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis. S2, converting the point cloud data in the laser radar coordinate system to the vehicle coordinate system, and further converting it to a unified world coordinate system; S3, tracking dynamic obstacles in the point cloud data processed by S2; S4. Calculate the velocity vector of each dynamic obstacle in the world coordinate system according to the annotation information; S5. Calculate the displacement according to the speed and time difference of the dynamic obstacle, and perform displacement compensation on the point cloud of the dynamic obstacle; S6. Update the annotation information in the 4D value annotation system and perform a consistency check.

2. The method for correcting dynamic obstacle point cloud under 4D true value annotation according to claim 1, characterized in that: In step S1, obtaining vehicle body information ego includes the following steps: The vehicle's real-time posture information, including position, speed, acceleration and direction, is obtained through on-board sensors. The real-time posture information will be used for subsequent point cloud coordinate conversion and displacement compensation. Obtaining the external parameters of the laser radar to the vehicle body: The relative position relationship / external parameters between the laser radar and the vehicle body are fixed, but need to be obtained through the calibration process. The external parameters include the rotation matrix and translation vector of the laser radar relative to the vehicle body, which are used to convert the point cloud data in the laser radar coordinate system to the vehicle body coordinate system; Get camera timestamp: When the camera collects image data, it will be accompanied by timestamp information; Get the timestamp of each point in the point cloud: When the lidar collects point cloud data, it also assigns a timestamp to each point.

3. The method for correcting dynamic obstacle point cloud under 4D true value annotation according to claim 1, characterized in that: Step S3 includes the following steps: After performing preliminary timestamp compensation on each point cloud point so that they can be compared on the same time basis, the point cloud deviation is processed according to the ego information of the self-vehicle to obtain new point cloud data, and then the new point cloud data will be 3D annotated. Since the annotation information itself has ID information, it is directly associated with the ID data of the previous and next frames to match the targets of the previous and next frames.

4. The method for correcting dynamic obstacle point cloud under 4D true value annotation according to claim 3 is characterized in that: Step S4 includes the following steps: The annotation information is transferred from the radar coordinate system to the vehicle coordinate system and then to the world coordinate system. According to the time difference between the previous and next frames and the 3D position transformation of the world coordinate system, the speed of the target in the world coordinate system is calculated using the Kalman filter.

5. The method for correcting dynamic obstacle point cloud under 4D true value annotation according to claim 4, characterized in that: Step S5 includes the following steps: For dynamic obstacles, the displacement of the dynamic obstacle within the time difference is calculated based on the time difference between the original timestamp of each point and the current processing time, as well as the calculated speed of the dynamic obstacle. Then, the point cloud of each dynamic obstacle is compensated for displacement, that is, its original position vector is added with the calculated displacement to obtain the corrected point cloud position.

6. The method for correcting dynamic obstacle point cloud under 4D true value annotation according to claim 5, characterized in that: Step S6 includes the following steps: Update the corrected point cloud position to the original 4D value annotation system, where the 4D value includes the 3D space coordinates and timestamp; After updating the annotation information, a consistency check is performed to ensure the consistency of all annotation information in time and space. This is done by comparing the projection of the corrected point cloud onto the image with the projection of the uncorrected point cloud onto the image to see if the dynamic obstacle point cloud is improved. 7.4D true value annotation based dynamic obstacle point cloud correction system, characterized by: The system comprises: The data acquisition module is used to obtain the point cloud 3D annotation information, the vehicle body information ego, the laser radar to the vehicle body external parameters, the camera timestamp and the timestamp of each point in the point cloud. The step of obtaining the point cloud 3D annotation information includes: the collected point cloud data is first pre-processed and then 3D annotated; Point cloud preprocessing: The collected point cloud data is time-synchronized to ensure the temporal consistency of all data. When collecting point cloud data, for static obstacles, preliminary timestamp compensation is performed on each point cloud point based on the vehicle's posture information and timestamp differences, so that they can be compared and subsequently processed on the same time basis. The coordinate conversion module is used to convert the point cloud data in the laser radar coordinate system to the vehicle coordinate system, and further to the unified world coordinate system; Dynamic obstacle tracking module, used to track dynamic obstacles in preprocessed point cloud data; The velocity calculation module calculates the velocity vector of each dynamic obstacle in the world coordinate system based on the annotation information; The displacement compensation module is used to calculate the displacement according to the speed and time difference of the dynamic obstacle and to perform displacement compensation on the point cloud of the dynamic obstacle; The annotation update module is used to update the annotation information in the 4D value annotation system and perform consistency checks.

Images