An autonomous driving dataset construction system and method

Abstract

The application discloses an automatic driving data set construction system and method, and belongs to the technical field of automatic driving; the system comprises a hardware and a data layer for collecting relevant data of a target scene and clock aligning the data; a perception and planning layer connected with the output end of the hardware and the data layer, for processing the aligned data, and outputting physically credible three-dimensional geometric true values, kinematic true values and expert planning trajectories; and a semantic cognition layer connected with the output end of the perception and planning layer, for key frame screening, fact dictionary construction and counterfactual reasoning, for converting numerical true values into cognitive annotation data, and obtaining an automatic driving data set, wherein the system is combined with the corresponding method to establish a three-in-one multi-paradigm unified architecture, which not only provides a high-precision three-dimensional geometric annotation for a modular model, and a closed-loop planning trajectory for an end-to-end model, but also innovatively introduces a counterfactual reasoning and hallucination suppression mechanism.

Claims

1. A method for constructing an autonomous driving dataset, characterized in that, Includes the following steps: S1. Acquire raw data of the target scene through hardware and data layers and construct an unstructured raw image. Then, perform clock alignment on the raw data to obtain an aligned ROS2 standard format data stream. S2. The aligned ROS2 standard format data stream is input into the perception and planning layer. The perception and planning layer processes the data in S1 to obtain physically reliable 3D geometric truth values, kinematic truth values, and expert-planned trajectories. The process is as follows: S21. Perform point cloud motion compensation based on high-frequency odometry and extract the relative time offset of each laser point. By combining the PTP frame header time, each point is assigned an absolutely high-precision timestamp. Then, the point is retrieved from the double-ended queue indexed by timestamp. adjacent poses and Linear interpolation is performed on the translation vector generated during the translation of the drive-by-wire data acquisition vehicle, and spherical linear interpolation is performed on the rotation quaternion during rotation to construct... Instantaneous pose matrix of the vehicle at any given moment ,Pick pose at the end of the scan Based on this, the projection transformation is performed in parallel using CUDA kernel functions: ； in This represents the radar point cloud after projection transformation; S22. Convert all radar point clouds processed in S21 to the vehicle coordinate system, define a distance of several meters around the drive-by-wire data acquisition vehicle as the region of interest, divide the region of interest into several sub-grids, and construct a KD-Tree to perform radial distance-based analysis on the point clouds within the sub-grids. The segmented sampling process is as follows: ； After sampling is complete, three non-collinear points are randomly selected within each subgrid to calculate the initial plane parameters, as follows: ； in , , These represent the three orthogonal components of the local ground plane normal vector in the vehicle's three-dimensional coordinate system. The distance constant term in the plane equation is represented. This indicates the three-dimensional spatial coordinates of the lidar sampling point in the vehicle coordinate system; Iterate through the remaining points excluding the three selected points, calculate the orthogonal distance from each remaining point to the plane, and add points whose distance is less than a set local point threshold to the Inlier set. Perform several iterations, and take the set with the most Inliers as the local ground plane. Remove the corresponding points, extract the normal vectors of adjacent sub-grid planes, multiply them to calculate the angle, and if the angle is greater than... If the system determines that the "complex terrain" is causing the fitting to diverge, it will mark the coordinates and timestamp of the area and send it to the manual review queue for manual review. S23. After removing all ground planes, perform three-dimensional geometric truth fitting; S24, merging L2 Euclidean distance, BEV viewpoint intersection-over-interference ratio, category difference, and size difference based on the predicted trajectory from the previous frame. and the observation trajectory of the current frame Construct a two-dimensional cost matrix : ； in, , , and These represent the system's preset weight coefficients for each matching cost. This represents the crossover ratio from the BEV perspective. Indicates the cost of class differences. Indicates the cost of size differences; The Hungarian algorithm is used to reduce the row and column of the two-dimensional cost matrix to obtain the bipartite graph matching scheme with the minimum global total cost. Trajectories with more than 5 consecutive unmatched frames are destroyed to obtain successfully associated independent trajectories. S25. Using EKF to extract smooth kinematic features from successfully associated independent trajectories, the process is as follows: First, state space and prediction are performed. Prior states and prior covariance matrices are calculated using a constant velocity and constant angular velocity model combined with inter-frame time differences. Second, chassis kinematic compensation and observation updates are performed to compensate for relative speed errors caused by vehicle motion. Finally, Kalman gain and fusion are performed. The Kalman gain is calculated, and the residual is used for post-secondary state updates. When the residual increases sharply, the observation noise covariance matrix is ​​dynamically increased to reduce the single-frame observation weight, ultimately leading to convergence speed. With angular velocity Write the physical truth value into the Box property; S26. Determine the physically plausible future trajectory and extract expert intent, as follows: Extract the absolute coordinates of the corresponding timestamp from the global high-precision odometer queue, and perform matrix multiplication to map them uniformly to the vehicle's local coordinate system at the current moment. The matrix multiplication is as follows: ； in express The trajectory points at each moment are mapped to Relative spatial coordinates of the vehicle in its local coordinate system at any given time. Indicates the current position of the vehicle. Represents the absolute coordinates of the timestamp corresponding to the current moment; The original projected trajectory is smoothed and resampled using third-order splines. The continuity of trajectory position, velocity and acceleration is ensured through dynamic constraints and spline smoothing. After processing, the average displacement error of each expert trajectory is calculated. With final displacement error The process is as follows: ； ； in This represents the total number of discrete time steps contained in the future trajectory sequence. The trajectory is in the th order. Smoothly fitted coordinates at each time step, Indicates the trajectory at the corresponding number The actual physical observation coordinates at each time step; S3. Use the unstructured image obtained in S1 and the data obtained in S2 as input to filter keyframes with high game value; S4, the semantic cognition layer, processes the data from S2 based on the high game value keyframes selected from S3, and completes the dataset construction.

2. The method for constructing an autonomous driving dataset according to claim 1, characterized in that, The process of S1 is as follows: S11. Configure a heterogeneous redundant sensor array to install LiDAR, industrial camera and high-precision positioning module on the wire-controlled data acquisition vehicle. S12. Using the GNSS receiver as the sole absolute time reference for the entire vehicle, continuously output high-precision 1PPS hardware pulses to the on-board time synchronization box to achieve strict phase alignment, and simultaneously send TOD messages to provide absolute timestamps. S13. The time synchronization box completes system-level time parsing based on the absolute timestamp and distributes the time synchronization to subsequent sensor nodes. S14. Perform cross-modal hardware-level clock alignment on the heterogeneous redundant sensor array and encapsulate and transmit the data to obtain an aligned ROS2 standard format data stream.

3. The method for constructing an autonomous driving dataset according to claim 2, characterized in that, The process of cross-modal hardware-level clock alignment and data encapsulation and transmission in S14 is as follows: S141, The high-precision positioning module is configured to directly extract the aligned absolute timestamp and output the real-time vehicle status. The high-precision positioning module is strictly aligned by 1PPS and outputs the high-precision vehicle motion status with absolute GPS timestamp of the drive-by-wire data acquisition vehicle. The high-precision vehicle motion status is converted into a custom RTK message. S142. The built-in PTP clock module in the lidar ensures that the internal rotation scanning cycle is aligned with the GPS time. When monitoring and parsing UDP packets, the ROS2 driver in the IPC memory uses the PTP timestamp in the radar frame header to convert the entire frame point cloud into PointCloud2 format and uniformly assign a global time tag. S143. The drive-by-wire chassis VCU broadcasts CAN messages containing the motion status of the drive-by-wire data acquisition vehicle at a set frequency. The CAN interface card enables hardware-level timestamp marking, directly marking the CAN message with a microsecond-level tag the moment it arrives, completely eliminating system scheduling jitter. The S144 industrial camera video capture card integrates a PTP slave clock unit to generate a high-precision frame trigger signal to synchronously control the camera's exposure start time. This is used to strictly synchronize the exposure time of the industrial camera. The card captures the original image through the industrial camera and simultaneously assigns an exposure timestamp to the original image directly by the PTP. The image is then written to the IPC memory via the PCIe interface using direct memory access with zero copying.

4. The method for constructing an autonomous driving dataset according to claim 3, characterized in that, The process of S23 is as follows: S231. Traverse all non-ground point sets and calculate the radial distance for each selected seed point. Dynamically assign a unique search radius threshold to it: ； in This indicates that for a radial distance of The current seed point is dynamically assigned a dedicated search radius threshold. and These represent the minimum and maximum value functions, respectively. This indicates the upper limit of the system's preset search radius threshold. This indicates the lower limit of the system's preset search radius threshold. This represents the divergence compensation coefficient caused by radar angular resolution. This represents the basic search radius compensation constant; S232, Utilizing a pre-built 3DKD-Tree, in The nearest neighbor point within the radius is searched and added to the current cluster. This process is iterated and expanded until the queue is empty. For clusters whose volume exceeds a preset volume, density-based spatial clustering is called to traverse the points within the cluster. If obvious density faults are found in the neighborhood, the neighborhood is hard-split into multiple independent sensing clusters based on connectivity, and finally a 3D cluster is obtained. S233. Project the 3D cluster onto the XY top view plane, and set the rotation step size. Traverse the rotating point cloud within a preset range, and after each rotation, calculate the area of ​​the bounding box aligned with the axis. : ； in and These represent the 3D clusters projected onto the XY top view plane and rotated by a specific angle. After coordinate transformation, the two-dimensional orthogonal coordinates of each point within the cluster in the temporary rotated coordinate system; Take the smallest As This is the instantaneous yaw angle observation value of the target. Obtained through reverse mapping and , Obtained from the original z-axis range; S234. Record the short-term historical state of the target using a hash table, and apply exponential moving average filtering to the successfully associated consecutive frame targets, as follows: ； in This represents the final yaw angle output of the target in the current frame after being smoothed by an exponential moving average filter. This represents the system's preset historical state retention coefficient; Finally, we obtain the smoothed 7-tuple. Inheriting the hardware timestamp of the original point cloud, it is encapsulated as a standard custom BoundingBoxArray topic and published to the semantic cognition layer.

5. The method for constructing an autonomous driving dataset according to claim 4, characterized in that, The selection of high game value keyframes in S3 involves two steps: visual semantic clustering and dynamic feature clustering. The process is as follows: Visual semantic clustering: S31. Call the pre-trained CLIP encoder to map the original image into a 512-dimensional vector. ; S32. Calculate the sample covariance matrix. The process is as follows: ； in A 512-dimensional feature vector representing all N image samples. The average value; The first 64 principal components of the covariance matrix were extracted by eigenvalue decomposition to construct the projection matrix. The feature dimension was reduced to 64 dimensions. ; S33. Determine the number of clusters in the 64-dimensional subspace. Perform iterations, and after each iteration, calculate the distance from the in-cluster sample to the centroid. Euclidean distance Then, the top-5% of samples with the smallest distance are forcibly extracted and marked as a static cluster set; Dynamic feature clustering: S34. Extract the dynamic features of the next few seconds to construct a vector. Simultaneously, Z-Score standardization is performed to eliminate dimensional differences, where This represents the target's average translational velocity over a time window of several seconds. This represents the absolute value of the maximum acceleration within that time window. This represents the absolute value of the maximum yaw rate within that time window. This represents the maximum geometric curvature of the target's trajectory within that time window; S35. Set the neighborhood radius and core point threshold of DBSCAN, extract all points that cannot meet the conditions, mark them as dynamic outliers, and obtain the dynamic outlier cluster. S36. Combine the static cluster set and the dynamic outlier set to obtain the key frame with high game value.

6. The method for constructing an autonomous driving dataset according to claim 5, characterized in that, The process of S4 is as follows: S41. Extract the RTK latitude, longitude, and altitude of the first frame from the high game value keyframes. Using the origin as a reference, the curvature radii of the ramus and trochanter are calculated based on the WGS84 ellipsoid parameters, and the coordinates are then converted into local ENU offsets. Then use the current absolute heading angle By mapping to the vehicle coordinate system using a two-dimensional rotation matrix, we obtain... as follows: ； ； Then, the sliding time window center difference method is used in the first frame. Extracting translational velocity within the window ;exist Acceleration is obtained by performing second-order difference within the window. Indicates the current keyframe time. Centered on the target, sliding forward 0.5 seconds, the X-axis coordinates of the target in the vehicle's local coordinate system after rotation mapping are shown. This represents the X-axis coordinate of the target in the local coordinate system of the vehicle when sliding back 0.5 seconds to the history. calculate ,when When the angle is not less than 180°, let Then ordered Multiply by π / 180 to convert to a normalized angular velocity. ; S42. Through intention deduction using the two-dimensional cross product, the calculation process of the cross product is as follows: ； in and Let X and Y represent the X and Y components of the target's current trajectory tangent vector in the vehicle's local coordinate system, respectively. and These represent the X-axis and Y-axis components of the target's look-ahead vector in this coordinate system, respectively. Set steering threshold when If the value is greater than the threshold, write a left turn; when Negative numbers less than the threshold are written as right turns, and the rest are written as straight lines; S43, Extracting the absolute speed of obstacles and the azimuth angle relative to the vehicle Calculate the relative approach speed between the drive-by-wire data acquisition vehicle and the obstacle along the connecting line. ,in This represents the absolute speed of the drive-by-wire data acquisition vehicle, and the collision time is calculated based on the approach speed. ,Will The 3D coordinates of the obstacles are written into the fact dictionary; S44. Deploy a lightweight AI model at the edge and inject hard system prompts via API. After obtaining the JSON response, perform spatiotemporal consistency projection matching to completely filter out visual illusions. After S45 and S44 are completed, several strategies are simulated in the Frenet coordinate system, and the trajectory of each strategy is resampled and transformed back to the Cartesian coordinate system. S46. Assemble all policy information, inject it into the cloud-based large-parameter LLM through prompt word engineering, and output the autonomous driving dataset according to the prompt word template.

7. An autonomous driving dataset construction system, comprising an autonomous driving dataset construction method according to any one of claims 1-6, characterized in that: This includes the hardware and data layers used to collect relevant data from the target scene and clock-align the data; Connected to the output of the hardware and data layers, it is used to process the aligned data and output physically reliable 3D geometric truth values, kinematic truth values, and the perception and planning layer of expert-planned trajectories. Connected to the output of the perception and planning layers, it is used for keyframe filtering, fact dictionary construction, and counterfactual reasoning, transforming numerical truth values ​​into cognitive labeled data to obtain the semantic cognitive layer of the autonomous driving dataset.

8. The autonomous driving dataset construction system according to claim 7, characterized in that: The hardware and data layers include a wire-controlled data acquisition vehicle, a LiDAR, industrial cameras, and a high-precision positioning module. The LiDAR is deployed on the top of the wire-controlled data acquisition vehicle, and there are several industrial cameras deployed circumferentially on the wire-controlled data acquisition vehicle. The high-precision positioning module is also set on the wire-controlled data acquisition vehicle, which also has a wire-controlled chassis VCU.

9. The autonomous driving dataset construction system according to claim 8, characterized in that: The lidar has a built-in PTP slave clock module, the video capture card of the industrial camera integrates a PTP slave clock unit, and the GNSS receiver inside the high-precision positioning module continuously outputs 1PPS hardware pulses to provide a unified global spatiotemporal constraint for data fusion.

Images