Point cloud data processing method and device, vehicle and storage medium

Abstract

The embodiment of the invention provides a point cloud data processing method and device, a vehicle and a storage medium. The method comprises the following steps: extracting at least one single-frame point cloud containing a target object from an original point cloud sequence based on four-dimensional label information of the target object; and on the basis of the corresponding relationship between the vehicle pose information and the single-frame point cloud, performing registration operation on the at least one frame of single-frame point cloud to obtain the reconstructed point cloud data of the target object, thereby improving the integrity of the point cloud data, and improving the training precision of the autonomous vehicle perception model.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing technology, and in particular to a point cloud data processing method, a point cloud data processing device, a vehicle, and a computer-readable storage medium. Background Technology

[0002] Radar sensors can be used to determine the distance between a target and a sensor, such as lidar (LiDAR), which is a sensor that determines the distance to a target object by emitting a laser beam and measuring the time difference or phase difference of the reflected beam.

[0003] In actual road driving, radar may be blocked by other dynamic objects, or the performance of radar may be affected by various hardware and software factors. Point cloud data collected in a short time often has the following problems: Because the radar does not have time to fully scan the target, the collected point cloud data is relatively sparse, that is, there are few data points. In addition to the reflection points of the target vehicle, the radar may also receive cluttered reflected light from the surrounding environment. These cluttered reflected lights will generate noise in the point cloud data, affecting the quality of the data and making it impossible to fully express the true shape of the target. This sparsity characteristic will reduce the training accuracy of the perception model. Summary of the Invention

[0004] The present invention provides a point cloud data processing method, apparatus, vehicle, and computer-readable storage medium to overcome or at least partially solve the above-mentioned problems.

[0005] This invention discloses a point cloud data processing method, including:

[0006] Extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object.

[0007] Based on the correspondence between vehicle pose information and single-frame point cloud, a registration operation is performed on at least one single-frame point cloud to obtain the reconstructed point cloud data of the target object.

[0008] Optionally, the step of extracting at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object includes:

[0009] Obtain the original point cloud sequence for the target object during the driving process;

[0010] The four-dimensional label information of the target object for each frame is obtained through a pre-trained model; the four-dimensional label information includes the center position and size of the bounding box for the target object.

[0011] Extract at least one single-frame point cloud containing the target object based on the center position of the bounding box and the size of the bounding box.

[0012] Optionally, before the step of performing a registration operation on the at least one frame of point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain the reconstructed point cloud data of the target object, the method further includes:

[0013] Generate a set of single-frame point clouds for the single-frame point cloud; the set of single-frame point clouds includes tracking identifiers that have a unique correspondence with the target object;

[0014] Identify the same tracking identifier as the target tracking identifier;

[0015] Store the target tracking identifier and the corresponding frame information into a sequential container;

[0016] The sequential container is used to store the frame information of the target tracking identifier from the first frame to the last frame in sequence;

[0017] The frame information of the first frame is the earliest frame information of the target object corresponding to the target tracking identifier that was observed;

[0018] The frame information of the last frame is the frame information of the last time the target object corresponding to the target tracking identifier was observed.

[0019] Optionally, the step of performing a registration operation on at least one frame of point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain reconstructed point cloud data of the target object includes:

[0020] Obtain vehicle position and orientation information;

[0021] Traverse the sequence container to determine the data volume of the frame information, and construct a frame information sequence based on the data volume in descending order;

[0022] The starting frame is determined from the frame information sequence;

[0023] The point cloud of the starting frame is determined as the target point cloud, and the point clouds of other frames are determined as the source point clouds.

[0024] Based on the pose information, the source point cloud and the target point cloud are transformed from the vehicle center coordinate system to the world coordinate system, and the source point cloud is used to perform a registration operation on the target point cloud to generate a registration result.

[0025] The registration result is determined as the reconstructed point cloud data of the target object.

[0026] Optionally, the step of registering the source point cloud with the target point cloud to generate a registration result includes:

[0027] The frames adjacent to the target point cloud are determined from the frame information sequence;

[0028] Generate first estimated pose information between the target point cloud and the point clouds of the adjacent frames;

[0029] Obtain historical registration results, and generate second estimated pose information between the target point cloud and other source point clouds besides the point clouds of the adjacent frames based on the historical registration results;

[0030] A registration result is generated based on the first estimated pose information and / or the second estimated pose information.

[0031] Optionally, it also includes:

[0032] Determine the robust function and the error threshold corresponding to the tracking identifier at the corresponding point;

[0033] Based on the robust function and the error threshold, the generated registration result is subjected to robustness enhancement and noise reduction operations.

[0034] Optionally, it also includes:

[0035] When the registration score of the registration result is greater than a preset threshold, the registration result is determined as the target point cloud;

[0036] When it is determined that all single-frame point clouds have been traversed, the target point cloud is stored.

[0037] This invention also discloses a point cloud data processing device, comprising:

[0038] The single-frame point cloud extraction module is used to extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object.

[0039] The point cloud data construction module is used to perform a registration operation on at least one single-frame point cloud based on the correspondence between vehicle pose information and single-frame point cloud, so as to obtain the reconstructed point cloud data of the target object.

[0040] This invention also discloses a vehicle, comprising:

[0041] One or more processors;

[0042] And one or more machine-readable media thereon storing instructions, which, when executed by the one or more processors, cause the vehicle to perform one or more methods as described above.

[0043] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0044] The embodiments of the present invention have the following advantages:

[0045] In this embodiment of the invention, at least one single-frame point cloud containing the target object is extracted from the original point cloud sequence based on the four-dimensional label information of the target object; based on the correspondence between the vehicle pose information and the single-frame point cloud, a registration operation is performed on the at least one single-frame point cloud to obtain the reconstructed point cloud data of the target object, thereby improving the integrity of the point cloud data and thus improving the training accuracy of the perception model of autonomous driving vehicles. Attached Figure Description

[0046] Figure 1 This is a flowchart of the steps of a point cloud data processing method provided in an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating a point cloud data processing method provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a vehicle's running trajectory provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of another vehicle trajectory provided in an embodiment of the present invention;

[0050] Figure 5 This is a structural block diagram of a point cloud data processing device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1 The diagram illustrates a flowchart of a point cloud data processing method provided in an embodiment of the present invention, which may specifically include the following steps:

[0053] Step 101: Extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object.

[0054] Step 102: Based on the correspondence between vehicle pose information and single-frame point cloud, perform a registration operation on the at least one single-frame point cloud to obtain the reconstructed point cloud data of the target object.

[0055] In practical applications, a point cloud is a dataset of spatial points obtained from LiDAR scanning. Each point contains three-dimensional coordinates (X, Y, Z) and other information, such as reflection intensity and timestamp. These points are arranged and combined in space to form a point cloud.

[0056] Four-dimensional labels (4D-labels) are used in the field of autonomous driving, especially in object detection tasks, to describe the state and motion information of a target object in three-dimensional space. They typically include the following dimensions:

[0057] 3D bounding box: describes the position, size, and orientation of a target object in three-dimensional space;

[0058] Confidence: Indicates the degree of certainty the model has about the detection results;

[0059] Category: Indicates the category of the target object, such as vehicle, pedestrian, bicycle, etc.

[0060] Motion information: including the target object's velocity, acceleration, etc.

[0061] In practical applications, the original point cloud sequence consists of multiple single-frame point clouds acquired by the vehicle during its operation, representing the target object. For example, the target object can be another vehicle besides the vehicle itself, and the four-dimensional label information is the four-dimensional label information of the target object. The single-frame point clouds constituting the original point cloud sequence suffer from problems such as sparse arrangement and high noise levels, resulting in an inability to fully represent the true shape of the target object. These problems reduce the training accuracy of the perception model.

[0062] In a specific implementation, embodiments of the present invention can extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object.

[0063] For example, suppose that during the vehicle's operation, a raw point cloud sequence consisting of one frame per second of point cloud was collected by a LiDAR. The desired outcome is to extract a single frame of point cloud at the nth second. The four-dimensional label information of the target object can then be obtained, containing information such as the vehicle's position, size, and orientation. Based on the information in the four-dimensional label information, a bounding box is created in three-dimensional space, which perfectly encloses the target vehicle. The point cloud frame at the 5th second is then traversed, and the points falling within the bounding box are extracted to form a new point cloud set. This new point cloud set can represent the single frame of point cloud of the target object at the 5th second.

[0064] This invention extracts at least one frame of points containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object, which can accurately extract the point cloud of the target object and reduce the influence of background noise.

[0065] After extracting a single-frame point cloud, embodiments of the present invention can also perform a registration operation on at least one single-frame point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain reconstructed point cloud data of the target object.

[0066] In practical applications, for large objects or complex scenes, scanning from multiple perspectives is often required to obtain complete data. Registration allows these local point cloud data to be stitched together into a complete 3D model. When point cloud data is acquired from multiple sensors, they need to be fused together to obtain richer and more accurate scene information. Registration allows comparison of point cloud data at different times, enabling object recognition and tracking. For example, SLAM technology can continuously register newly acquired point cloud data with existing maps, enabling robots to localize and build maps in unknown environments. SLAM stands for Simultaneous Localization and Mapping. It is a technology that allows robots to autonomously localize in unknown environments and simultaneously build environmental maps.

[0067] In practical implementation, vehicle pose information can describe the position and attitude of the vehicle relative to a reference coordinate system (such as the world coordinate system) at a specific moment, and can include translation and rotation information. As can be seen from the above, a single frame point cloud can represent the point cloud data obtained by radar scanning of the vehicle at a certain moment. These point cloud data are relative to the vehicle coordinate system. The correspondence between vehicle pose information and single frame point cloud refers to how to transform the point cloud data in the vehicle coordinate system to the reference coordinate system so that different point cloud data can be unified into the same coordinate system for registration and fusion.

[0068] The embodiments of the present invention can perform registration operations based on the correspondence between vehicle pose information and single-frame point cloud, which can unify multiple point cloud data acquired from different perspectives or at different times into the same coordinate system, forming a complete reconstructed point cloud data for representing the target object, thereby avoiding the problem of sparse point cloud.

[0069] In this embodiment of the invention, at least one single-frame point cloud containing the target object is extracted from the original point cloud sequence based on the four-dimensional label information of the target object; based on the correspondence between the vehicle pose information and the single-frame point cloud, a registration operation is performed on the at least one single-frame point cloud to obtain the reconstructed point cloud data of the target object, thereby improving the integrity of the point cloud data and thus improving the training accuracy of the perception model of autonomous driving vehicles.

[0070] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0071] refer to Figure 2 , Figure 2 This is a flowchart illustrating a point cloud data processing method provided in an embodiment of the present invention;

[0072] In an optional embodiment of the present invention, the step of extracting at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object includes:

[0073] Obtain the original point cloud sequence for the target object during the driving process;

[0074] The four-dimensional label information of the target object for each frame is obtained through a pre-trained model; the four-dimensional label information includes the center position and size of the bounding box for the target object.

[0075] Extract at least one single-frame point cloud containing the target object based on the center position of the bounding box and the size of the bounding box.

[0076] The embodiments of the present invention can acquire the original point cloud sequence collected during vehicle driving, and the four-dimensional label information of the dynamic vehicle in each frame predicted by the pre-trained model. The four-dimensional label information may include, but is not limited to, the bounding box size of the vehicle's length, width and height l*w*h, the center position of the bounding box in the coordinate system of the rear axle of the vehicle [x,y,z], the velocity vector in the coordinate system of the rear axle of the vehicle [v_x,v_y,v_z], the vehicle tracking ID, etc.

[0077] In this embodiment of the invention, after obtaining the four-dimensional label information and the original single-frame point cloud sequence, the center position and size of the vehicle bounding box provided by the four-dimensional label information can be used to extract the single-frame point cloud of the moving vehicle from the original point cloud sequence. Based on the four-dimensional label information, the single-frame point cloud can be extracted from the original single-frame point cloud sequence, which can accurately segment the point cloud part corresponding to the target object from the point cloud, remove background interference, and at the same time, through segmentation, the target object can be separated from other objects, which is convenient for subsequent tracking, recognition and analysis.

[0078] In an optional embodiment of the present invention, before the step of performing a registration operation on the at least one frame of point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain reconstructed point cloud data of the target object, the method further includes:

[0079] Generate a set of single-frame point clouds for the single-frame point cloud; the set of single-frame point clouds includes tracking identifiers that have a unique correspondence with the target object;

[0080] Identify the same tracking identifier as the target tracking identifier;

[0081] Store the target tracking identifier and the corresponding frame information into a sequential container;

[0082] The sequential container is used to store the frame information of the target tracking identifier from the first frame to the last frame in sequence;

[0083] The frame information of the first frame is the earliest frame information of the target object corresponding to the target tracking identifier that was observed;

[0084] The frame information of the last frame is the frame information of the last time the target object corresponding to the target tracking identifier was observed.

[0085] In practical applications, the data representation method of a single-frame point cloud can be determined first. In this embodiment of the invention, after obtaining the four-dimensional label information and the original single-frame point cloud sequence, the vehicle bounding box center position and bounding box size provided by the four-dimensional label information can be used to extract the moving vehicle single-frame point cloud from the original point cloud sequence, denoted as Li_i_j = {p1, p2...pn}; where...

[0086] L: Represents a point cloud set, that is, a set containing multiple points.

[0087] i: Represents a tracking identifier that uniquely corresponds to the target object. In practical applications, this can be the track_id of a moving vehicle, which is the vehicle's unique identifier. In multi-target tracking, each tracked vehicle is assigned a unique ID to distinguish between different vehicles.

[0088] j: Represents the single frame ID, which is the sequence number of the image or LiDAR scan frame corresponding to the current point cloud data.

[0089] {p1,p2,...pn}: Represents all points belonging to the vehicle in the point cloud of frame j. Each point p is a three-dimensional coordinate, representing a point on the vehicle surface. Points that correspond to the tracking identifier track_id can be called corresponding points.

[0090] The set Li_j represents the set of point clouds extracted from the original point cloud sequence that belong to a specific vehicle at a specific time. By constructing and utilizing such a set, accurate tracking, analysis, and understanding of moving vehicles can be achieved, facilitating subsequent analysis processes and thus improving the efficiency of point cloud data reconstruction.

[0091] In practical applications, to further improve the reconstruction efficiency of point cloud data and achieve concurrent operations, a specific storage method can be used to enable each thread to process a separate set of tracking identifiers in order to improve efficiency.

[0092] In a specific implementation, the embodiments of the present invention can identify the same tracking identifier as the target tracking identifier, and then store the target tracking identifier and the frame information corresponding to the target tracking identifier into a sequential container. Specifically, the sequential container can be used to save the frame information of the target tracking identifier from the first frame to the last frame in sequence; the frame information of the first frame is the earliest frame information of the target object corresponding to the target tracking identifier; the frame information of the last frame is the frame information of the last time the target object corresponding to the target tracking identifier was observed.

[0093] For example, a hash table is used to group vehicles with the same track_id together.

[0094] Key: In data structures, a key is typically used to uniquely identify a data entry. It's like an index in a dictionary; the corresponding value can be quickly found using the key.

[0095] value: The data corresponding to the key.

[0096] Sequential containers: A type of container that stores elements in a specific order, such as vector and list.

[0097] A hash table can be used to group vehicles with the same track_id together; where the key is track_id and the value is a sequential container, which stores all frame information corresponding to that track_id, including single-frame point cloud, 4D label information, vehicle pose information, etc.

[0098] In this embodiment of the invention, the same tracking identifier track_id can be identified as the target tracking identifier. The frame information of the target tracking identifier from the first frame to the last frame can be stored in the sequential container. The first frame stores the frame information of the earliest time when the vehicle with track_id was observed, and the last frame stores the frame information of the last time when the vehicle with track_id was observed.

[0099] For example, as shown in Table 1, Table 1 illustrates the frame information contained in multiple sequential containers.

[0100]

[0101] In this embodiment of the invention, identical tracking identifiers are identified as target tracking identifiers; the target tracking identifiers and their corresponding frame information are stored in a sequential container; the sequential container is used to sequentially save the frame information of the target tracking identifier from the first frame to the last frame; the frame information of the first frame is the earliest observed frame information of the target tracking identifier; the frame information of the last frame is the frame information of the target tracking identifier at the last time it was observed. This enables concurrent operations on point cloud data for multiple targets, that is, it allows each thread to process a set of track_id data independently, thereby further improving the reconstruction efficiency of point cloud data.

[0102] In an optional embodiment of the present invention, the step of performing a registration operation on at least one frame of single-frame point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain reconstructed point cloud data of the target object includes:

[0103] Obtain vehicle position and orientation information;

[0104] Traverse the sequence container to determine the data volume of the frame information, and construct a frame information sequence based on the data volume in descending order;

[0105] The starting frame is determined from the frame information sequence;

[0106] The point cloud of the starting frame is determined as the target point cloud, and the point clouds of other frames are determined as the source point clouds.

[0107] Based on the pose information, the source point cloud and the target point cloud are transformed from the vehicle center coordinate system to the world coordinate system, and the source point cloud is used to perform a registration operation on the target point cloud to generate a registration result.

[0108] The registration result is determined as the reconstructed point cloud data of the target object.

[0109] In image processing and computer vision, box information typically refers to the position and size of a rectangular region within an image. This rectangular region can be represented by four values:

[0110] x-coordinate: The horizontal coordinate of the top-left corner of the rectangle.

[0111] y-coordinate: The vertical coordinate of the top left corner of the rectangle.

[0112] Width: The width of the rectangle.

[0113] Height: The height of the rectangle.

[0114] For example, since each group of data currently stored in the sequential container has the same track_id, assuming that the above data has a total of n frames, the number of point clouds of the tracked vehicle and the box information in the four-dimensional label information are counted in the n frames. The n frames of data are sorted in descending order of point cloud size, and the frame with a larger number of point clouds and smaller changes in box information between the preceding and following frames is selected as the starting frame.

[0115] After determining the starting frame, this embodiment of the invention can determine the point cloud of the starting frame as the target point cloud, determine the point clouds of other frames as the source point cloud, and also obtain vehicle pose information, and transform the source point cloud and target point cloud from the vehicle center coordinate system to the world coordinate system based on the pose information.

[0116] In fields such as autonomous driving and robotics, coordinate system transformation is a fundamental and crucial concept. Different sensors and algorithms often employ different coordinate systems.

[0117] The vehicle center coordinate system is a coordinate system established with the center of the rear axle of the vehicle as the origin. It is usually used to describe the state of the vehicle itself, such as speed and acceleration.

[0118] The world coordinate system is a fixed, global coordinate system used to describe objects and vehicles in the environment.

[0119] The main purposes of converting the rear axle centerline of the vehicle to the global system are as follows:

[0120] Multi-sensor fusion: Data acquired by different sensors (such as LiDAR and cameras) are often in different coordinate systems. In order to fuse these data, they need to be unified into the same coordinate system.

[0121] Map matching: Matching the vehicle's position in the world coordinate system with a high-precision map to achieve positioning and navigation.

[0122] Path planning: Plan the vehicle's trajectory in the world coordinate system and convert it into control commands in the vehicle's center coordinate system.

[0123] Achieving global positioning: By transforming the position information in the vehicle's center coordinate system to the world coordinate system, the vehicle's accurate position in the global coordinate system can be determined, thus achieving global positioning of the vehicle.

[0124] Facilitates perception and decision-making: By unifying the coordinates of all objects (including the vehicle itself and objects in the environment) to the world coordinate system, it is possible to easily perform perception tasks such as target detection, tracking, and obstacle avoidance, as well as high-level tasks such as path planning and decision-making.

[0125] Performing kinematic and dynamic analysis: Transforming the vehicle's motion state from the vehicle's center coordinate system to the world coordinate system allows for convenient kinematic and dynamic analysis of the vehicle, providing a basis for vehicle control.

[0126] LiDAR (Light Detection and Ranging)

[0127] Principle: By emitting a laser beam into the surrounding environment and measuring the time it takes for the laser to return, the distance to the target object is calculated, thereby obtaining 3D point cloud data of the surrounding environment.

[0128] Advantages: High accuracy, unaffected by lighting conditions, capable of acquiring high-resolution point cloud data, suitable for high-precision map building and object recognition.

[0129] Applications: High-precision map building, object detection, autonomous driving, etc.

[0130] RTK (Real-time Dynamic Differential GPS):

[0131] Principle: By using the difference between the base station and the mobile station, the error of the GPS signal is eliminated, and centimeter-level positioning accuracy is achieved.

[0132] Advantages: High positioning accuracy and good real-time performance, suitable for scenarios with high positioning accuracy requirements.

[0133] Applications: High-precision map surveying, autonomous driving, drone positioning, etc.

[0134] Wheel speed odometer:

[0135] Principle: By measuring the wheel speed, calculating the vehicle's travel distance, and combining this with the vehicle's steering angle, the vehicle's position and attitude can be estimated.

[0136] Advantages: Low cost, easy to integrate, and can provide continuous location information.

[0137] Disadvantages: It is easily affected by factors such as roller skating and uneven ground, resulting in large cumulative errors over a long period of time.

[0138] Application: As an auxiliary positioning sensor, it can be fused with other sensors to improve positioning accuracy.

[0139] For example, coordinate system transformation can be implemented in the following way.

[0140] By fusing sensor data from LiDAR, RTK, and wheel speed odometers, and processing it using SLAM algorithms, the vehicle's position and attitude in a world coordinate system (such as the UTM coordinate system) can be calculated in real time. This position and attitude are represented by a coordinate transformation matrix T_w_veh, which transforms the source point cloud and target point cloud in the vehicle's center coordinate system to the world coordinate system.

[0141] SLAM algorithms can simultaneously solve the problems of localization and mapping. By processing sensor data, SLAM algorithms can estimate the vehicle's pose and construct an environmental map. Different sensors have different characteristics, advantages, and disadvantages. LiDAR provides high-precision local information, RTK provides high-precision global information, and wheel speed-oddscopy provides continuous motion information. By fusing these sensors, the accuracy and robustness of localization can be improved.

[0142] The T_w_veh matrix contains the vehicle's position (translation) and attitude (rotation) information.

[0143] This matrix allows you to transform the coordinates of any point on the vehicle from the vehicle coordinate system to the world coordinate system.

[0144] Optionally, the step of registering the source point cloud with the target point cloud to generate a registration result includes:

[0145] The frames adjacent to the target point cloud are determined from the frame information sequence;

[0146] Generate first estimated pose information between the target point cloud and the point clouds of the adjacent frames;

[0147] Obtain historical registration results, and generate second estimated pose information between the target point cloud and other source point clouds besides the point clouds of the adjacent frames based on the historical registration results;

[0148] A registration result is generated based on the first estimated pose information and / or the second estimated pose information.

[0149] For example, to ensure no loss of generality, assume the starting frame ID is i, and register the point cloud of the starting frame as the target point cloud and the point clouds of other frames as the source point clouds in the order of [i, i-1, i+1, i-2, i+2...] from near to far.

[0150] The specific method is as follows:

[0151] refer to Figure 3 , Figure 3 This is a schematic diagram of a vehicle running trajectory provided in an embodiment of the present invention; in order to reduce the impact of the unstable 4d_label results of the model pre-brushing, the entire registration is performed in the world frame, so it is necessary to use the vehicle pose T_w_veh to convert the point cloud to the world frame.

[0152] The matching method uses Generalized-ICP (Iterative Closest Point), and registration can include an initial registration stage and a non-initial registration stage.

[0153] For the initial registration, i.e. the registration of frames i-1, i+1 and i, considering that the displacement of adjacent frames is small, the first estimated pose information of the initial registration can be the unit matrix or predicted from the 4d_label result. At the same time, the maximum distance threshold between the matching points of the target point cloud and the source point cloud can be appropriately increased according to the preset adjustment range to improve the initial matching accuracy.

[0154] refer to Figure 4 , Figure 4 This is a schematic diagram of another vehicle trajectory provided in an embodiment of the present invention;

[0155] For non-initial registration, taking frame i+2 as an example, the second estimated pose information from i+2 to i can be predicted by building a kinematic model for the vehicle using historical registration results (from i-1 to i, from i+1 to i, and from i-2 to i). Using the predicted pose can make frame i+2 close to frame i.

[0156] This invention can generate a registration result based on the first and second predicted pose information. After successful registration, the transformation matrix obtained by registration makes the i+2 frame and the i frame completely overlap. The above transformation matrix is ​​superimposed with the transformation matrix of the predicted pose and saved to establish a kinematic model.

[0157] In an optional embodiment of the present invention, it further includes:

[0158] Determine the robust function and the error threshold corresponding to the tracking identifier at the corresponding point;

[0159] Based on the robust function and the error threshold, the generated registration result is subjected to robustness enhancement and noise reduction operations.

[0160] For example, during vehicle tracking, due to the vehicle's potentially large distance from the driver or severe occlusion, some frames may have fewer point clouds or more noise, leading to significant errors in the pose estimation results obtained through matching, or even matching failure. To improve matching accuracy, a robust kernel function with an adaptive threshold is introduced into the cost function in ICP. The formula is as follows:

[0161]

[0162] The above formula is used in point cloud registration, especially in the ICP algorithm, to find the optimal transformation matrix that minimizes the distance between corresponding points in two point clouds.

[0163] arg max: indicates finding the x that makes the following expression take the maximum value.

[0164] x: The parameter to be optimized, which usually represents the transformation matrix between two point clouds.

[0165] f_i(x): represents the error of the i-th pair of corresponding points under the current transformation x.

[0166] ρ_i: is a robust function used to reduce the impact of outliers on the optimization results.

[0167] ||f_i(x)|| 2 : represents the square of the Euclidean distance.

[0168] The purpose of the formula is to find an optimal transformation matrix x that minimizes the weighted sum of squared errors between all corresponding point pairs. The weighting refers to assigning different weights to different point pairs using a robust function ρ_i, thereby mitigating the influence of outliers.

[0169] In point cloud registration, the goal is usually to find a transformation matrix that optimally aligns two point clouds. This formula describes this optimization problem.

[0170] f_i(x): In point cloud registration, it usually represents the distance between corresponding points in two point clouds.

[0171] ρ_i: Since point clouds may contain noise and outliers, directly minimizing the sum of squared distances between all corresponding point pairs may make the algorithm sensitive to this noise. Therefore, a robust function ρ_i is introduced to reduce the penalty for outlier pairs. Commonly used robust functions include the Huber function and the Tukey function.

[0172] In an optional embodiment of the present invention, it further includes:

[0173] When the registration score of the registration result is greater than a preset threshold, the registration result is determined as the target point cloud;

[0174] When it is determined that all single-frame point clouds have been traversed, the target point cloud is stored.

[0175] In practical applications, there may be situations where the same vehicle is tracked for multiple frames. In order to save computation and improve accuracy, a method of extracting key frames is introduced. The registration process will only be added when the dynamic vehicle's position and direction relative to the vehicle exceeds a threshold, or when the vehicle's movement distance or direction exceeds a threshold.

[0176] The registration results are scored based on the nearest neighbor search. If the registration fails, the reconstruction process is stopped and a single frame result is used.

[0177] If not all single-frame point clouds have been traversed, the predicted pose for this registration should be recalculated using the historical registration results. When it is determined that all single-frame point clouds have been traversed, the target point cloud can be stored to complete the reconstruction of the point cloud data.

[0178] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0179] Reference Figure 5 The diagram illustrates a structural block diagram of a point cloud data processing device provided in an embodiment of the present invention, which may specifically include the following modules:

[0180] The single-frame point cloud extraction module 501 is used to extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object.

[0181] The point cloud data construction module 502 is used to perform a registration operation on at least one single-frame point cloud based on the correspondence between vehicle pose information and single-frame point cloud, so as to obtain the reconstructed point cloud data of the target object.

[0182] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0183] This invention also provides a vehicle, comprising:

[0184] One or more processors; and

[0185] One or more machine-readable media storing instructions thereon, when executed by the one or more processors, cause the vehicle to perform the methods described in embodiments of the present invention.

[0186] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the point cloud data processing method embodiments described above, achieving the same technical effects. To avoid repetition, these details will not be repeated here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0187] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0188] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0189] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0190] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0193] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A point cloud data processing method, characterized in that, include: Extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object. Based on the correspondence between vehicle pose information and single-frame point cloud, a registration operation is performed on at least one single-frame point cloud to obtain the reconstructed point cloud data of the target object.

2. The method according to claim 1, characterized in that, The step of extracting at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object includes: Obtain the original point cloud sequence for the target object during the driving process; The four-dimensional label information of the target object for each frame is obtained through a pre-trained model; the four-dimensional label information includes the center position and size of the bounding box for the target object. Extract at least one single-frame point cloud containing the target object based on the center position of the bounding box and the size of the bounding box.

3. The method according to claim 1 or 2, characterized in that, Before the step of performing a registration operation on at least one frame of point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain the reconstructed point cloud data of the target object, the method further includes: Generate a set of single-frame point clouds for the single-frame point cloud; the set of single-frame point clouds includes tracking identifiers that have a unique correspondence with the target object; Identify the same tracking identifier as the target tracking identifier; Store the target tracking identifier and the corresponding frame information into a sequential container; The sequential container is used to store the frame information of the target tracking identifier from the first frame to the last frame in sequence; The frame information of the first frame is the earliest frame information of the target object corresponding to the target tracking identifier that was observed; The frame information of the last frame is the frame information of the last time the target object corresponding to the target tracking identifier was observed.

4. The method according to claim 3, characterized in that, The step of performing a registration operation on at least one frame of point cloud based on the correspondence between vehicle pose information and single-frame point cloud to obtain the reconstructed point cloud data of the target object includes: Obtain vehicle position and orientation information; Traverse the sequence container to determine the data volume of the frame information, and construct a frame information sequence based on the data volume in descending order; The starting frame is determined from the frame information sequence; The point cloud of the starting frame is determined as the target point cloud, and the point clouds of other frames are determined as the source point clouds. Based on the pose information, the source point cloud and the target point cloud are transformed from the vehicle center coordinate system to the world coordinate system, and the source point cloud is used to perform a registration operation on the target point cloud to generate a registration result. The registration result is determined as the reconstructed point cloud data of the target object.

5. The method according to claim 4, characterized in that, The step of registering the source point cloud with the target point cloud to generate a registration result includes: The frames adjacent to the target point cloud are determined from the frame information sequence; Generate first estimated pose information between the target point cloud and the point clouds of the adjacent frames; Obtain historical registration results, and generate second estimated pose information between the target point cloud and other source point clouds besides the point clouds of the adjacent frames based on the historical registration results; A registration result is generated based on the first estimated pose information and / or the second estimated pose information.

6. The method according to claim 5, characterized in that, Also includes: Determine the robust function and the error threshold corresponding to the tracking identifier at the corresponding point; Based on the robust function and the error threshold, the generated registration result is subjected to robustness enhancement and noise reduction operations.

7. The method according to claim 5, characterized in that, Also includes: When the registration score of the registration result is greater than a preset threshold, the registration result is determined as the target point cloud; When it is determined that all single-frame point clouds have been traversed, the target point cloud is stored.

8. A point cloud data processing device, characterized in that, include: The single-frame point cloud extraction module is used to extract at least one single-frame point cloud containing the target object from the original point cloud sequence based on the four-dimensional label information of the target object. The point cloud data construction module is used to perform a registration operation on at least one single-frame point cloud based on the correspondence between vehicle pose information and single-frame point cloud, so as to obtain the reconstructed point cloud data of the target object.

9. A vehicle, characterized in that, include: One or more processors; and One or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the vehicle to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images