Semantic scene flow based autonomous driving planning method and verification system training method

Abstract

A semantic scene flow-based automatic driving planning method and a verification system training method are disclosed. Point cloud data corresponding to multiple continuous time points is input into a pre-trained unsupervised joint network, a dynamic area in point cloud data corresponding to each time point is obtained through a coarse-level preprocessing module, and all instances in the point cloud data corresponding to each time point are obtained; global geometric features and local geometric features of the point cloud data corresponding to each time point are obtained through a space-time feature extraction module; a semantic scene flow is obtained through a double-head task decoding module in combination with all instances corresponding to each point cloud data; based on a double-path planning mechanism, a first path planning result and / or a second path planning result is obtained according to the semantic scene flow, and then an automatic driving planning result is obtained. Based on the double-head task decoder and the double-path planning mechanism, high-precision perception and safe decision-making of a dynamic environment are realized, and the controllability and safety of path planning in a complex environment are effectively improved through a verification system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, specifically to an autonomous driving planning method and a verification system training method based on semantic scene flow. Background Technology

[0002] With the rapid iteration of autonomous driving technology, vehicles have acquired basic environmental perception and path planning capabilities in conventional scenarios such as urban roads and highways. However, the "long tail problem" brought about by complex and dynamic traffic environments has not yet been effectively solved. Highly uncertain and random scenarios such as pedestrians crossing the road, irregular lane changes by non-motorized vehicles, and sudden braking by vehicles in front are not only key factors causing traffic accidents and system failures, but also "long tail problems" that existing technologies cannot effectively cover.

[0003] Current mainstream solutions mostly adopt a phased processing mode of "object detection + trajectory tracking". However, it only relies on object bounding box information and ignores the overall motion features of the point cloud, resulting in incomplete dynamic understanding. Moreover, perception and motion estimation lack tight coupling, making it difficult to achieve globally consistent scene modeling. In addition, it is highly dependent on high-precision maps and external positioning, and its stability is insufficient in dynamic and changing environments. While scene flow can provide fine-grained information on point-by-point 3D motion of the point cloud, offering new possibilities for understanding dynamic environments, existing methods mostly focus on improving estimation accuracy and optimizing registration algorithms, lacking a systematic integration design with semantic labeling, path planning, and scene generation, making it difficult to directly support the robust operation of autonomous driving in complex dynamic scenarios.

[0004] Therefore, there is an urgent need to design a safety planning method for autonomous vehicles in complex dynamic environments, as well as a verification system that integrates point cloud perception, dynamic information extraction, planning decision-making, and simulation verification. Summary of the Invention

[0005] The main technical problem addressed by this invention is how to design a safety planning method for autonomous vehicles in complex dynamic environments, as well as a verification system that integrates point cloud perception, dynamic information extraction, planning decision-making, and simulation verification.

[0006] According to the first aspect, one embodiment provides an autonomous driving planning method based on semantic scene flow, comprising:

[0007] Acquire point cloud data corresponding to multiple consecutive time points;

[0008] The point cloud data corresponding to multiple consecutive time points are input into a pre-trained unsupervised joint network to obtain a semantic scene stream. In the unsupervised joint network, a coarse-level preprocessing module obtains the dynamic regions in the point cloud data corresponding to each time point, and performs preliminary instance segmentation on the dynamic regions to obtain all instances in the point cloud data corresponding to each time point. A spatiotemporal feature extraction module obtains the global and local geometric features of the point cloud data corresponding to each time point. A dual-head task decoding module obtains the semantic scene stream based on all instances, global geometric features, and local geometric features corresponding to each point cloud data.

[0009] Based on the dual-path planning mechanism, the first path planning result and / or the second path planning result are obtained according to the semantic scene flow;

[0010] The autonomous driving planning result is obtained based on the first path planning result and / or the second path planning result.

[0011] According to the second aspect, one embodiment provides a training method for an autonomous driving planning and verification system, comprising:

[0012] Acquire multimodal data at multiple consecutive moments in different scenarios, wherein the multimodal data includes a first original viewpoint image and point cloud data;

[0013] For any given scenario: input the point cloud data from all multimodal data corresponding to the scenario into an unsupervised joint network to obtain the semantic scene stream at each time step; input the semantic scene streams at all time steps into a diffusion model to initialize the Gaussian model and obtain the initial Gaussian model corresponding to the scenario.

[0014] Obtain the time series corresponding to each time moment, which includes the time moment and multiple historical time moments before the time moment; obtain the first original trajectory corresponding to each time moment and the lane / speed change information corresponding to all time moments in the time series corresponding to each time moment; generate a new trajectory corresponding to the time moment based on the first original trajectory corresponding to the time moment and all multimodal data and lane / speed change information corresponding to the time series corresponding to the time moment, and obtain the first new perspective image, the second original trajectory, and the second original perspective image corresponding to the time moment;

[0015] Based on the semantic scene flow corresponding to each time step in the current scene, the initial Gaussian model corresponding to the current scene is iteratively optimized. Then, in any training epoch:

[0016] For any given moment: input the semantic scene stream corresponding to that moment into the Gaussian model corresponding to the current scene, so as to render the second new perspective image corresponding to that moment according to the new trajectory, and render the third original perspective image corresponding to that moment according to the first original trajectory;

[0017] A trajectory consistency loss is obtained based on the first original trajectory and the second original trajectory corresponding to each time step; a motion consistency loss is obtained based on the semantic scene flow and lane / speed change information corresponding to each time step; a comprehensive rendering loss is obtained based on the first new view image, the second new view image, and the third original view image corresponding to each time step; a first reconstruction loss is obtained based on the first original view image and the second original view image corresponding to each time step; a second reconstruction loss is obtained based on the first original view image and the third original view image corresponding to each time step; and a total loss function is obtained based on the trajectory consistency loss, the motion consistency loss, the comprehensive rendering loss, the first reconstruction loss, and the second reconstruction loss.

[0018] The Gaussian model corresponding to the current scene is optimized based on the total loss function until the total loss function converges, thus obtaining the Gaussian verification model corresponding to the current scene.

[0019] According to a third aspect, one embodiment provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the above-described training method for the autonomous driving planning method or the autonomous driving planning verification system based on semantic scene flow.

[0020] According to the above embodiment, an autonomous driving planning method based on semantic scene flow firstly outputs scene flow information and semantic label information of each point cloud based on the joint output of the dual-head task decoder in the unsupervised joint network, to obtain the semantic scene flow corresponding to each time moment, which is used to evaluate potential conflicts in the dynamic environment in real time (such as the intersection of object trajectories). Then, the semantic scene flow is integrated into a dual-path planning mechanism. The point-based planner directly uses the point-level scene flow to perform fine-grained trajectory reasoning to avoid local collisions. The bounding box-based planner generates a global trajectory with object-level constraints to ensure that the vehicle avoids risks in advance, thereby improving the safety and real-time performance of the decision. The path planning results output by different planners are selected for output, or the weights of different planners are dynamically assigned according to the scene complexity and task requirements, so that the final generated motion trajectory can not only cope with local dynamic changes, but also meet the overall traffic safety and efficiency requirements, thereby achieving safe and reliable path planning in dynamic scenarios.

[0021] According to the training method of the autonomous driving planning verification system in the above embodiment, a Gaussian point cloud rendering environment is constructed using a real dataset. Based on the first original trajectory, lane / speed change information and multimodal data under different scenarios, a new trajectory is predicted by a world model, and a first new perspective image conforming to physical laws, a regenerated second original image, and a second original trajectory are output. Then, the prediction results of the world model are further input into the Gaussian reconstruction module to optimize the geometric and motion details of the dynamic scene. During the training process, the total loss function is constructed by combining the second new perspective image rendered by the Gaussian model with the third original perspective image, thereby ensuring the accuracy and stability of the final planning result. Then, combined with semantic scene flow information, motion references for autonomous driving decision-making and trajectory planning are generated to achieve closed-loop simulation verification, effectively improving the controllability and safety of planning in complex environments. Attached Figure Description

[0022] Figure 1 This is a flowchart of an autonomous driving planning method based on semantic scene flow;

[0023] Figure 2 This is a schematic diagram of network modules in an unsupervised joint network;

[0024] Figure 3 This is a training flowchart for a training method of an autonomous driving planning and verification system. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0026] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the order of the steps or actions in the method description can be changed or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0027] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0028] The phased processing mode of "target detection + trajectory tracking" first detects dynamic targets such as vehicles and pedestrians in point clouds or images through the perception module, then predicts the motion trajectory of the targets through time-series tracking, and finally generates the driving path of the vehicle by the planning module.

[0029] Traditional vision- or point cloud-based mapping and localization methods are poorly adapted to dynamic environments. Abnormal data introduced by the motion of moving objects can severely interfere with sensor data matching, leading to a decrease in the accuracy of pose estimation. Although the robustness of the algorithm can be improved by detecting and removing dynamic objects, such as by using random sampling consensus and its improved methods, this "detect first, process later" strategy often reduces computational efficiency due to complex iterative calculations. Furthermore, the possibility of false rejections and false rejections when identifying dynamic features can reduce accuracy, ultimately affecting the overall performance of the mapping and localization algorithm.

[0030] While some existing planning algorithms use semantic information to identify and eliminate object categories considered "potentially dynamic," they are essentially coarse-grained filtering based on static labels. They are limited to the extraction and classification of static features and fail to fully integrate the motion-related information of the targets. This makes it difficult for the algorithms to accurately capture the motion intentions and behavioral evolution trends of targets in complex scenes. For example, although they can identify the categories (static attributes) of dynamic targets such as "pedestrians" and "vehicles" through semantic labels, they cannot obtain dynamic information such as the motion state (such as speed and direction), trajectory trends (such as whether they are about to cross the road), and behavioral intentions (such as whether non-motorized vehicles are preparing to change lanes). This limits the effectiveness of the algorithms in making decisions in complex dynamic scenes.

[0031] Current mainstream scene flow estimation frameworks can be divided into two main categories based on the granularity of point cloud processing, and these two categories differ significantly in computational efficiency: one category is estimation methods based on sparse point cloud samples, which select key feature points or local region point clouds for motion calculation, balancing efficiency but potentially sacrificing detail; the other category is estimation methods based on full point cloud samples, which perform point-by-point traversal processing on the input point cloud, preserving more complete motion information but incurring higher computational costs. However, due to factors such as measurement inconsistencies between different frames, object occlusion, and low point cloud density, relying solely on raw point cloud data is insufficient to achieve robust scene flow estimation; therefore, additional information is needed to resist these interferences, thereby providing effective support for accurate matching of point clouds across different frames, and ultimately improving the accuracy and stability of scene flow estimation.

[0032] Furthermore, while simply combining existing methods for scene flow estimation, semantic labeling, path planning, and scene generation can achieve a preliminary technical loop, it can lead to a series of deep coupling problems: First, the independent operation of each module can cause errors to accumulate and amplify during time-series propagation, resulting in inconsistencies in spatiotemporal modeling; second, the lack of a collaborative optimization mechanism between tasks prevents the full utilization of the complementarity of each module, ultimately leading to suboptimal overall system performance; third, the serial or parallel operation of multiple modules can cause competition for computing resources, making it difficult to meet the core requirement of real-time response for autonomous driving; fourth, this loose combination lacks targeted design for the uncertainties of long-tail scenarios, has weak adaptability to sudden situations in complex dynamic environments, and is difficult to support the robust operation of the system.

[0033] This invention first constructs a semantic scene flow estimation module through unsupervised learning to achieve joint modeling of point clouds, semantics, and 3D motion fields. Based on this, it automatically separates static backgrounds and dynamic targets to predict the motion trends of dynamic obstacles. In the path planning stage, dynamic constraints are provided to the dual-path planner based on the semantic scene flow. The point-based planner directly utilizes the point-level scene flow for fine-grained trajectory reasoning, avoiding local collisions. The bounding box-based planner generates a global trajectory with object-level constraints, enabling the vehicle to avoid potential conflicts in advance and improving the real-time performance and safety of decision-making. In the verification stage, the verification system generates a controllable dynamic traffic scene using a Gaussian point cloud environment. Through four core stages—environment, perception, planning, and verification—it achieves closed-loop testing and robustness evaluation of the planning algorithm, thereby realizing safe planning and closed-loop verification of autonomous vehicles in complex dynamic environments.

[0034] Please refer to Figure 1 Some embodiments provide an autonomous driving planning method based on semantic scene flow, which includes the following steps:

[0035] Step S100: Obtain point cloud data corresponding to multiple consecutive time points.

[0036] LiDAR sensors, such as vehicle-mounted LiDAR, are used to collect point cloud data of the real road environment in real time in autonomous driving scenarios, with continuous time frames.

[0037] For example, point cloud data can be obtained at three consecutive time points, such as point cloud data P corresponding to time points t-1, t, and t+1. t-1 P t P t+1 This data is then used as input to a subsequent unsupervised joint network to obtain a semantic scene stream. The autonomous driving scene consists of various dynamic and static objects such as vehicles, pedestrians, trees, and buildings. Each point in the resulting 3D point set (point cloud data) contains its corresponding spatial coordinates and auxiliary attributes (such as reflection intensity).

[0038] Step S110: Input the point cloud data corresponding to multiple consecutive time points into the pre-trained unsupervised joint network to obtain the semantic scene stream.

[0039] In dynamic environments, static detection and target tracking alone are insufficient to fully reflect the motion trends of multiple targets. This embodiment introduces semantic scene flow modeling, combining motion and semantic information from point cloud data to form the motion vector and category label for each point or rigid body object. This joint representation reflects both the local dynamic characteristics of individuals and the overall category attributes.

[0040] Then, the average velocity and direction of motion of each rigid object are estimated based on the point cloud sequence (i.e., the overall motion vector of the rigid object), and semantic labels are used to distinguish different types of traffic participants. Subsequently, the path planner based on scene flow / dynamic objects uses this information to generate constrained trajectories, so that the autonomous driving system can predict potential risks in advance and take avoidance measures in highly dynamic scenarios, ensuring the safe and efficient operation of the vehicle.

[0041] The unsupervised joint network in this embodiment includes a coarse-level preprocessing module 111, a spatiotemporal feature extraction module 112, and a dual-head task decoding module 113. The network takes point cloud data corresponding to three consecutive time frames as input and simultaneously outputs point-level scene flow estimation results and point-level semantic labels. Through self-supervised reconstruction consistency constraints and time cycle constraints, the network can learn the motion and semantic features of point cloud data without manual annotation, thereby obtaining motion and category information simultaneously in a dynamic environment.

[0042] Among them, the reconstruction consistency constraint refers to the consistency between the source frame data and the real target frame data after mapping the source frame data to the target frame based on the estimated scene flow. For example, for time t-1, the scene flow estimation result based on time t-1 corresponds to the point cloud data P at time t-1. t-1 After mapping, the point cloud data P corresponding to time t t Maintaining consistency; the core logic of the time cycle consistency constraint is to utilize the motion continuity and cyclic mapping consistency of multi-frame point clouds to further constrain the accuracy of scene flow estimation, thereby allowing the network to learn continuous motion features that conform to physical laws, rather than isolated motion between two frames. For example, for the point cloud data P corresponding to times t-1, t, and t+1 t-1 P t P t+1 First, calculate the scene flow from time t-1 to time t and from time t to time t+1, then superimpose the two scene flows, which is P. t-1 After P t Indirect mapping to P t+1 Simultaneously calculate P directly. t-1Direct mapping to P t+1 If the scene flow is accurately estimated, then the superimposed direct mapping should be consistent with the direct mapping, and P t+1 Map back to P via reverse scene flow t-1 Corresponding to the original P t-1 The overlap ensures that scene flow estimation not only satisfies the geometric alignment of adjacent frames, but also conforms to the physical rationality of long-term temporal sequence.

[0043] In the unsupervised joint network of this embodiment, the dynamic region in the point cloud data corresponding to each time step is first obtained by the coarse-level preprocessing module 111, and preliminary instance segmentation is performed based on the obtained dynamic region to obtain all instances in the point cloud data corresponding to each time step.

[0044] For example, for point cloud data corresponding to any one of the times t-1, t, and t+1, an unsupervised method is used to first roughly separate the static and dynamic points in the point cloud data through ray-casting, and then cluster the dynamic points, such as the DBSCAN clustering algorithm, to initially segment different instances, thereby obtaining all instances in the point cloud data corresponding to that time.

[0045] The spatiotemporal feature extraction module 112 obtains the global and local geometric features of the point cloud data corresponding to each time step. In this embodiment, for any point cloud data corresponding to any time step, the global geometric features of the point cloud data are obtained. For example, the global geometric features in the point cloud data are extracted by a unified encoder. The unified encoder is essentially a deep learning backbone network with strong generalization ability (such as a feature extraction architecture based on Transformer), which performs end-to-end feature encoding on the input 3D point cloud data. Through operations such as spatial convolution, attention mechanism or local aggregation, the spatial topology, target geometry and global scene association information of the point cloud are mined, and the final output is a global geometric feature containing multi-dimensional general representation.

[0046] In this embodiment, the unified encoder is used as the shared backbone for the scene flow estimation task of the subsequent scene flow decoder and the instance segmentation task of the strength segmentation decoder. Through the feature reuse mechanism, the two tasks share the same set of encoder feature extraction process and output results. Only in the subsequent process, the targeted mapping of features and the solution of task objectives are completed through their respective dedicated task heads. This not only ensures the global consistency of feature representation, but also improves the model running efficiency by avoiding redundant calculations.

[0047] To further optimize features, this embodiment employs an iterative module based on GRU (Gated Recurrent Unit) to update voxel-level feature representations and point-level feature vectors. By aggregating all point features within each voxel, the voxel-level feature representation is refined; then, the refined voxel-level feature representation is used to update the point-level feature vector, thereby achieving efficient cross-scale information propagation while preserving spatial details.

[0048] The point cloud data is divided into voxels to obtain multiple voxels. Based on the feature vectors of all points within each voxel, the voxel-level feature representation and the point-level feature vector are iteratively updated for a preset number of rounds to obtain the local geometric features of the point cloud data. Specifically, in any iteration: the initial feature vector corresponding to each point in the point cloud data in the current round is obtained; the initial feature vectors of each point within any voxel are aggregated to obtain the voxel-level feature representation corresponding to that voxel; the feature vector of each point within that voxel is updated based on the voxel-level feature representation, resulting in the updated feature vector of each point in the current round; if the current round is not the last round, the updated feature vector of each point in the current round is used as the initial feature vector for that point in the next round; if the current round is the last round, the updated feature vector of each point in the current round is the local geometric feature of the point cloud data.

[0049] For example, for point cloud data at any given time: First, the point cloud data is voxelized, that is, the point cloud data is divided into uniform grids based on voxel resolution, and multi-scale representation is achieved through voxel resolution; where the voxel resolution can be set to 0.05m.

[0050] The initial multilayer perceptron (MLP) extracts the feature vector of each point in the point cloud data. Then, average pooling is used to aggregate and refine the features of each point within a voxel, resulting in a voxel-level feature representation for each voxel. For example, element-wise averaging is performed on the feature vectors of points within each voxel. Assuming the feature vectors of three points within a voxel are [1.0, 2.0], [1.5, 2.5], and [2.0, 3.0], the refined voxel-level feature representation would be [(1.0 + 1.5 + 2.0) / 3, (2.0 + 2.5 + 3.0) / 3] = [1.5, 2.5]. Alternatively, point-wise aggregation can be performed using the gating mechanism of the GRU (update gate and reset gate), calculating a weighted average or gated sum, i.e., voxel-level feature representation = Σ(GRU). gate ×point i ) / N, where N is the number of contained points within the voxel, using the gating weights output by the GRU. gate For the local features of all points (point) iAfter dynamic weighted filtering, a voxel-level feature representation is generated, which can retain the key information of the point cloud data and suppress the interference of redundant / noise points, thereby capturing local geometry and spatiotemporal dependence and avoiding the loss of details due to simple averaging.

[0051] Then, a weighted fusion mechanism is used to backpropagate the voxel-level feature representation to each point within the voxel, and this process is iterated for 2-3 rounds to enhance cross-scale information propagation, ensuring the spatiotemporal consistency and robustness of the feature representation. This implementation is compatible with frameworks such as PointNet++, facilitating efficient computation. Specifically, in any iteration update process, the expression for updating the feature vector corresponding to each point within the voxel based on the voxel-level feature representation is:

[0052] new i =α·orig i +(1-α)·voxel

[0053] Among them, new i otig represents the initial feature vector corresponding to the i-th point within the voxel in the current round; i This represents the feature vector of the i-th point within the voxel after the update in the current round; α is a learnable parameter, typically 0.5; voxel represents the voxel-level feature representation corresponding to the voxel; alternatively, the voxel-level feature representation can be used as the initial hidden state of the GRU, and the voxel information can be fused using the feature vector corresponding to the point as input through the recursive update formula of the GRU, i.e.:

[0054]

[0055] Among them, h t The final hidden state represents the updated feature vector corresponding to that point; h t-1 This represents the initial hidden state (corresponding to voxel-level feature representation) in the current iteration, z. t To update the door, For candidate activation, the corresponding initial feature vector of the point is used to achieve iterative refinement of point-level features. Through 2-3 iterations, spatiotemporal consistency is ensured, and sparse point cloud processing is compatible, thereby improving the feature robustness in dynamic scenes.

[0056] The semantic scene stream is obtained by the dual-head task decoding module 113 based on all instances, global geometric features and local geometric features corresponding to each point cloud data.

[0057] Point cloud data corresponding to every two adjacent time points in a series of consecutive time points are considered as a set of point cloud data; the dual-head task decoding module in this embodiment includes a scene stream decoder and an instance segmentation decoder, and the resulting semantic scene stream includes the scene stream estimation result corresponding to each set of point cloud data and point-level semantic labels; wherein:

[0058] The global and local geometric features corresponding to each point cloud data in each set are input into the scene flow decoder for scene flow estimation, thus obtaining the scene flow estimation result for that set of point cloud data. For example, for a set of point cloud data formed at time t-1 and time t, the global and local geometric features corresponding to this set of point cloud data are input into the scene flow decoder to obtain the scene flow estimation result at time t-1. This scene flow estimation result at time t-1 is the scene flow estimation result for that set of point cloud data.

[0059] The scene flow estimation result predicted by the scene flow decoder in the unsupervised joint network includes two parts: scene flow residual and vehicle motion. The scene flow residual is directly predicted by the scene flow decoder and represents the local motion of each point in the dynamic environment relative to the vehicle when the vehicle is stationary. That is, the relative motion vector of each point in the dynamic environment relative to the vehicle. The vehicle motion represents the motion vector of the vehicle, which is calculated by coarsely segmenting the background point cloud in the set of point cloud data. Then, the vehicle motion and the scene flow residual are added to obtain the scene flow estimation result corresponding to the set of point cloud data. At this time, each point in the point cloud data corresponding to time t-1 corresponds to a three-dimensional displacement vector (three-dimensional motion vector).

[0060] All instances corresponding to each point cloud data in each group are input into the instance segmentation decoder for fine-grained instance segmentation to obtain the point-level semantic label corresponding to the group of point cloud data. Specifically, the instance segmentation decoder obtains the probability that each point in the point cloud data corresponding to the earliest time in each group of point cloud data belongs to different instance categories. For any point in the point cloud data corresponding to the earliest time: the instance category corresponding to the highest probability among all probabilities of the point is taken as the semantic label of the point, and the point-level semantic label of the point cloud data is taken as the point-level semantic label of the group of point cloud data.

[0061] For example, for a set of point cloud data formed at time t-1 and time t, after inputting the instances corresponding to all point cloud data in the set of point cloud data into the instance segmentation decoder, a mask classification vector for each point in the point cloud data at time t-1 is obtained. The mask classification vector of each point contains the unnormalized probability that the point belongs to different instances. The maximum probability is selected from these normalized probabilities, and the instance number corresponding to the maximum probability is used as the instance category label corresponding to the point. Among them, the instance label corresponding to the background static points is 0, and the category label of each dynamic point is set according to the instance number (such as 1, 2, ..., C-1) corresponding to its instance. Thus, the category label (point-level semantic label) corresponding to each point in the point cloud data at time t-1 is obtained, and the point-level semantic label of the point cloud data at time t-1 is used as the point-level semantic label of the set of point cloud data.

[0062] like Figure 2 As shown, the point cloud data P corresponding to consecutive times t-1, t, and t+1 are... t-1 P t With P t+1 The input is fed into an unsupervised joint network based on the point cloud data P corresponding to time t-1 and t. t-1 P t The point-level semantic labels and scene flow estimation results obtained from the set of point cloud data at time t-1 constitute the semantic scene flow M at time t-1. t-1 ; and based on the point cloud data P corresponding to times t and t+1 t P t+1 The point-level semantic labels and scene flow estimation results obtained from the generated set of point cloud data at time t constitute the semantic scene flow M at time t. t .

[0063] It should be noted that the refined instance segmentation in this embodiment is based on motion and instance grouping, and does not depend on the specific object type of each point in reality. That is, the instance category label in this embodiment does not correspond to a specific semantic category (such as whether the instance belongs to a vehicle or a pedestrian).

[0064] Step S120: Based on the dual-path planning mechanism, obtain the first path planning result and / or the second path planning result according to the semantic scene flow.

[0065] The dual-path planning mechanism in this embodiment includes a point-based planner and a bounding box-based planner to obtain a first path planning result and / or a second path planning result. By combining the point-based planner and the bounding box-based planner, high-precision perception and safe decision-making of the dynamic environment can be achieved.

[0066] Considering the diversity of sensor outputs and data formats in practical applications, this embodiment also incorporates an input adaptation mechanism. When the input data contains semantic labels and scene flow information, a bounding box-based planner fuses the semantic labels and motion vectors to generate a targeted trajectory. When the input data contains only scene flow data without semantic information, a point-based planner can directly rely on motion vector information to generate the trajectory, ensuring normal operation even with missing semantic information. By uniformly processing data from different sources, including point clouds, semantic labels, and motion fields (scene flows) for multimodal fusion, unified modeling and constraints on multiple categories of dynamic targets are achieved. This ensures flexibility and universality in path planning for multi-source inputs, thereby enhancing its adaptability in complex traffic environments.

[0067] The first path planning result is obtained by using the point cloud data corresponding to each time step and the scene flow estimation result corresponding to each set of point cloud data as input through a point-based planner. The point-based planner uses points as the smallest modeling unit to directly model dynamic changes in the environment and infer trajectories. Through point-by-point analysis and prediction, it captures local details and small motion changes in the scene and can have high adaptability and response accuracy for complex dynamic scenes (such as densely populated areas or multi-object interaction areas).

[0068] And / or, a second path planning result is obtained through a bounding box-based planner. This bounding box-based planner uses rigid body objects as modeling units and focuses on the stability analysis of macroscopic structure and overall motion. It is suitable for regular road environments and traffic scenarios with multiple vehicle interactions. By predicting the overall speed and position of the target object, it can effectively plan the global path and reduce the impact of local noise on decision-making.

[0069] This embodiment first performs rigid body object partitioning based on the point-level semantic labels corresponding to each set of point cloud data, obtaining the bounding box of each rigid body object. After obtaining the scene flow estimation results and point-level semantic labels corresponding to a set of point cloud data, this embodiment takes a set of point cloud data corresponding to time t-1 and time t as an example. At this time, each point in the point cloud data corresponding to time t-1 corresponds to a motion vector and a semantic label. Then, based on the semantic labels and spatial connectivity, the point cloud data corresponding to time t-1 is rigidly partitioned, thereby dividing the point cloud data corresponding to this time into several rigid body objects, and then obtaining the bounding box of each rigid body object.

[0070] Then, based on the scene flow estimation results corresponding to the point cloud data, the overall motion vector of each rigid body object is obtained; wherein, the overall motion vector of the entire object can be obtained based on the motion vector of the points inside each rigid body object. For example, the motion vector corresponding to each point is obtained based on the scene flow estimation results corresponding to the point cloud data; for any rigid body object: obtain the object center point of the rigid body object, such as obtaining the geometric center point by calculating the average coordinates of all points within the rigid body object as the object center point of the rigid body object; then, determine the confidence weight of each point in the rigid body object based on the distance between each point in the rigid body object and the object center point. In this embodiment, a normalized weighting strategy of "the closer the distance, the greater the weight" is used to determine the confidence weight corresponding to each point within the rigid body object. Then, the confidence weight corresponding to the j-th point can be expressed as:

[0071]

[0072] Where B represents the current rigid body object; w j d represents the confidence weight corresponding to the j-th point in the current rigid body object; j d represents the distance between the j-th point in the current rigid body object and the center point of the object; max This represents the maximum distance between all points in the current rigid body object and the object's center point; d k This represents the distance between the k-th point in the current rigid body object and the center point of the object.

[0073] The overall motion vector of the rigid body object is obtained by multiplying the confidence weight of each point in the rigid body object with its motion vector. This is done by multiplying the confidence weight of each point in the rigid body object with its motion vector, and summing the results of multiplying all points together to obtain the overall motion vector of the rigid body object. This allows points closer to the center and with more stable motion to dominate the overall motion state, effectively suppressing edge noise and local deformation, and improving the accuracy and robustness of rigid body motion estimation.

[0074] Finally, the bounding box of each rigid body object and the overall motion vector are input into the bounding box-based planner to obtain the second path planning result.

[0075] Step S130: Obtain the autonomous driving planning result based on the first path planning result and / or the second path planning result.

[0076] In the dual-path planning mechanism, the two types of planners work collaboratively in a unified scheduling module to achieve complementary decision-making: the point-based planner ensures the detail and accuracy of trajectory reasoning, while the bounding box-based planner ensures the global consistency and robustness of path planning. The scheduling module can either directly select one of the planning results output by the two planners as the final automatic planning result, i.e., use either the first path planning result or the second path planning result as the autonomous driving planning result; or it can dynamically assign weights to the first and second path planning results according to the scenario complexity and task requirements, and then obtain the final autonomous driving planning result based on the first and second path planning results and their corresponding weights. This ensures that the final generated trajectory can cope with local dynamic changes while meeting the overall traffic safety and efficiency requirements.

[0077] Furthermore, the dual-path planning mechanism in this embodiment can directly output vehicle control commands to guide the generation of the autonomous vehicle's trajectory, thereby achieving safe and efficient autonomous navigation.

[0078] This embodiment first uses a dual-head task decoder in an unsupervised joint network to jointly output scene flow information and semantic label information from the point cloud, obtaining the semantic scene flow corresponding to each time step. This is used to evaluate potential conflicts in the dynamic environment (such as the intersection of object trajectories) in real time. Then, the semantic scene flow is integrated into a dual-path planning mechanism. The point-based planner directly uses the point-level scene flow to perform fine-grained trajectory reasoning, avoiding local collisions. The bounding box-based planner generates a global trajectory with object-level constraints, ensuring that vehicles avoid risks in advance, thereby improving the safety and real-time performance of decision-making. The path planning results output by different planners are selected for output, or weights are dynamically assigned to different planners according to scene complexity and task requirements. The final generated motion trajectory can not only cope with local dynamic changes, but also meet the overall traffic safety and efficiency requirements, thus achieving safe and reliable path planning in dynamic scenarios.

[0079] Please refer to Figure 3 Some embodiments provide a training method for an autonomous driving planning and verification system, which includes the following steps:

[0080] The verification system in this embodiment includes four core components: environment, perception, planner, and closed-loop testing. Through multimodal information fusion and semantic modeling, it effectively realizes a complete closed loop from point cloud perception and dynamic information extraction to planning decision-making and simulation verification.

[0081] Step S200: Acquire multimodal data at multiple consecutive moments in different scenes, wherein the multimodal data includes the first original view image and point cloud data.

[0082] Existing testing frameworks often rely on rule-defined virtual scenarios, lacking coverage of complex interactions and extreme situations. This embodiment verifies the planning effect through a scene generation and rendering mechanism based on Gaussian Splatting. Using Gaussian point cloud modeling, discrete point cloud data is mapped into a continuous Gaussian distribution scene, constructing a realistic and scalable dynamic simulation environment. In this environment, scene flow and semantic labels are injected in real time, forming a 3D scene representation that includes both structural information and motion features. New trajectories obtained from the planner receive continuously updated input in this environment, achieving a complete closed-loop operation of "perception-planning-feedback". In this way, the robustness, stability, and decision boundaries of the autonomous driving system under different dynamic conditions can be systematically verified. Furthermore, long-tail or extreme scenarios can be generated through scene perturbation injection, thereby comprehensively testing the reliability and generalization ability of the planning strategy.

[0083] First, in the environmental phase, videos corresponding to different scenes (such as tunnels, urban roads, highways, etc.) are acquired through sensors, thereby obtaining multimodal data at multiple consecutive time points. The multimodal data corresponding to each time point includes the original view image and point cloud data corresponding to that time point. In this embodiment, the original view image acquired by the sensors is referred to as the first original view image, and the first original view image corresponding to the t-th time point is denoted as Y. t1 .

[0084] Step S210: For any scene: Input the point cloud data from all multimodal data corresponding to the scene into the unsupervised joint network to obtain the semantic scene stream corresponding to each time step, and then obtain the initial Gaussian model corresponding to the current scene through the diffusion model.

[0085] In the perception phase, point cloud data from all time points in the current scene are input into an unsupervised joint network. After passing through a coarse-level preprocessing module, a spatiotemporal feature extraction module, and a dual-head task decoding module, the semantic scene stream corresponding to each time point is obtained. For example, for the multimodal data corresponding to time points 1 to t+1, the point cloud data corresponding to every three adjacent time points are input into the unsupervised joint network for processing to obtain the semantic scene stream corresponding to time points 1 to t. The network structure and data processing of the unsupervised joint network have been described in step S110 of the above-mentioned autonomous driving planning method based on semantic scene streams, and will not be repeated here.

[0086] In this embodiment, a Gaussian model is set up for different scenarios. For the current scenario, the semantic scene stream corresponding to all times in the current scenario is input into the diffusion model to initialize the Gaussian model and obtain the initial Gaussian model corresponding to the scenario. Thus, the diffusion model and the Gaussian model work together to drive the prediction and high-fidelity reconstruction of future scenarios.

[0087] Step S220: Obtain the time series corresponding to each time moment, the first original trajectory, and the lane / speed change information corresponding to each time moment in the time series corresponding to that time moment. Combine all the multimodal data corresponding to the time series corresponding to that time moment to generate the new trajectory corresponding to that time moment, and obtain the first new perspective image, the second original trajectory, and the second original perspective image corresponding to that time moment.

[0088] In this embodiment, during the planner stage, a new trajectory, a first new perspective image, a second original trajectory, and a second original perspective image are obtained at any given moment in the current scene based on the world model. The world model is an intelligent system in the field of artificial intelligence that can learn, abstract, and model the core laws (geometric structure, physical rules, causal relationships, and temporal dynamics) of the external environment from multimodal observation data (such as sensor data). Its core is to construct a "virtual mapping" of the real world and, based on this mapping, to predict and extrapolate future environmental states and the consequences of actions in accordance with physical laws, ultimately assisting intelligent agents (such as autonomous vehicles) in making more forward-looking and robust decisions. It can both preserve the continuity of the current perspective and generate scene representations from new perspectives to expand the observation range, while simulating possible future states of the vehicle and the environment. Subsequently, the predicted new trajectory is further input into the Gaussian reconstruction module to optimize the geometric and motion details of the dynamic scene, thereby ensuring the accuracy and stability of the planning results.

[0089] First, obtain the time series corresponding to each moment. This time series contains the current moment and multiple historical moments before the current moment. The number of historical moments included in the time series can be set by the user. For example, if the current moment is t+1 and the time series needs to include two historical moments before the current moment, then the time series corresponding to the current moment will contain the moments t-1, t, and t+1.

[0090] Then, the first original trajectory corresponding to each moment is obtained. In this embodiment, the first original trajectory refers to the real position sequence of the vehicle from a certain historical moment to the current moment. At this time, the historical moment can be the earliest moment in the time sequence corresponding to the current moment, or it can be the starting moment of the entire movement. It can be set according to actual needs.

[0091] The first original trajectory corresponding to the given moment, lane / speed change information for all moments in the time series corresponding to that moment, and multimodal data are input into the world model. A new perspective image is generated, and the original perspective image and the first original trajectory are regenerated, resulting in the new trajectory, the first new perspective image, the second original perspective image, and the second original trajectory corresponding to that moment. The new trajectory corresponding to that moment can be generated using a point-based planner and / or a bounding box-based planner in a dual-path planning mechanism. Let Ft be the first new perspective image and the second original perspective image corresponding to the t-th moment. t1 Y t2 .

[0092] Step S230: Iteratively optimize the initial Gaussian model corresponding to the current scene based on the semantic scene flow corresponding to each time step in the current scene to obtain the Gaussian verification model corresponding to the current scene.

[0093] The total loss function is constructed based on the reference data generated by the world model and the rendering results of the Gaussian model in the Gaussian environment reconstruction stage. The Gaussian parameters are continuously optimized using a self-supervised method to achieve high-fidelity reconstruction of the environment.

[0094] In any training epoch, for any given time: the semantic scene stream corresponding to that time is input into the Gaussian model corresponding to the current scene for Gaussian environment reconstruction, so as to render the second new perspective image corresponding to that time based on the obtained new trajectory, and render the third original perspective image corresponding to that time based on the first original trajectory; where, let F be the second new perspective image and the third original perspective image corresponding to the t-th time. t2 Y t3 .

[0095] It should be noted that this embodiment also employs an online repair mechanism to supervise Gaussian generation through scene flow, thereby eliminating artifacts (such as floating points and holes) that may occur in the reconstruction of the new trajectory and ensuring the geometric and motion consistency between the original trajectory and the newly added trajectory in the Gaussian representation space. Furthermore, although the scene reconstructed by Gaussian is a continuous high-fidelity dynamic environment generated driven by real point clouds and semantic scene flow, the new position point cloud obtained after simulating vehicle motion through this environment is not the original real data, but rather a virtual observation predicted by the Gaussian model.

[0096] Based on the first and second original trajectories corresponding to each time step, a trajectory consistency loss is obtained to constrain the accuracy of vehicle motion prediction. The trajectory consistency loss can be expressed as:

[0097]

[0098] Among them, L traj P represents the trajectory consistency loss, where T represents the total number of time points in the current scenario;orig (t) represents the first original trajectory corresponding to the t-th time; P gen (t) represents the second original trajectory at time t, output by the world model; This represents the L2 distance.

[0099] Motion consistency loss is derived from the semantic scene flow and lane / speed change information at each time step to ensure the consistent motion trend of dynamic targets. The motion consistency loss can then be expressed as:

[0100]

[0101] Among them, L flow This represents the motion consistency loss, where T represents the total number of moments in the current scene. Let represent the semantic scene stream corresponding to time t; x represents the x-th point in the semantic scene stream; v(x) represents the motion vector corresponding to the x-th point; Δx represents the actual displacement corresponding to the x-th point obtained from lane / speed change information, which can be calculated by continuous frame point cloud registration or Gaussian prediction displacement; ||·||1 represents the L1 distance.

[0102] The comprehensive rendering loss is obtained based on the first new perspective image, the second new perspective image, and the third original perspective image corresponding to each time step. Specifically, photometric loss is used: a first rendering loss is constructed based on the second new perspective image rendered by the Gaussian model and the first new perspective image generated by the world model at each time step; a second rendering loss is constructed based on the second new perspective image reconstructed by the Gaussian model and the third original perspective image to evaluate the geometric matching effect between the original trajectory and the newly added trajectory under different viewing angles. The comprehensive rendering loss is obtained based on the first and second rendering losses, thereby constraining the consistency of the Gaussian model under different perspectives. The comprehensive rendering loss can then be expressed as:

[0103]

[0104] Among them, L render To account for rendering losses, L photo Indicates photometric loss; F t1 F represents the first new perspective image corresponding to time t; t2 Y represents the second new perspective image corresponding to time t; t3 L represents the third original view image corresponding to time t; photo (F) t1 F t2 ) represents the first rendering loss, L photo (F) t2 Y t3) represents the second rendering loss; λ1 and λ2 are the weights corresponding to the first rendering loss and the second rendering loss, respectively, which can be set by yourself.

[0105] A first reconstruction loss is obtained based on the first original viewpoint image and the second original viewpoint image corresponding to each time step; a second reconstruction loss is obtained based on the first original viewpoint image and the third original viewpoint image corresponding to each time step; wherein, for example, the mean square error loss corresponding to the pixel value of the pixel at the corresponding position in the first original viewpoint image and the second original viewpoint image corresponding to each time step can be constructed to obtain the first reconstruction loss; the mean square error loss corresponding to the pixel value of the pixel at the corresponding position in the first original viewpoint image and the third original viewpoint image corresponding to each time step can be constructed to obtain the second reconstruction loss.

[0106] The total loss function is obtained by considering trajectory consistency loss, motion consistency loss, comprehensive rendering loss, first reconstruction loss, and second reconstruction loss. For example, each loss function is assigned a corresponding weight, and the total loss function is obtained based on each loss function and its corresponding weight.

[0107] The Gaussian model corresponding to the current scene is optimized based on the total loss function until the total loss function converges, thus obtaining the Gaussian validation model corresponding to the current scene.

[0108] After obtaining the new trajectory, the entire verification system performs closed-loop verification based on the acquired trajectory. First, the verification system simulates driving according to the new trajectory in its corresponding scenario, acquiring new point cloud data and rendering a second new perspective image using the Gaussian verification model corresponding to that scenario. The acquired new point cloud data and the rendered second new perspective image are then input into the verification system as new multimodal data to continue acquiring new trajectories. Point cloud acquisition and motion feedback are then performed using the acquired new perspective, and the results are fed back to the perception and planning stages, thus achieving verification within closed-loop simulation and effectively improving the controllability and safety of planning in complex environments.

[0109] In addition, to enhance multi-view consistency, the verification system in this embodiment evaluates the geometric matching effect of the original trajectory and the newly added trajectory under different viewing angles through Chamfer distance and Pearson correlation; the system can also deal with dynamic occlusion problems, such as pedestrians occluding vehicles, and improve the coverage of the far field region by predicting the initial Gaussian distribution.

[0110] This embodiment utilizes a real dataset to construct a Gaussian point cloud rendering environment. Based on the first original trajectory, lane / speed change information, and multimodal data under different scenarios, it outputs a new trajectory prediction result that conforms to physical laws, along with a first new perspective image, a regenerated second original image, and a second original trajectory, all based on a world model. The prediction results of the world model are then further input into the Gaussian reconstruction module to optimize the geometric and motion details of the dynamic scene, thereby ensuring the accuracy and stability of the final planning result. Combined with semantic scene flow information, a motion reference for autonomous driving decision-making and trajectory planning is generated and verified in closed-loop simulation, effectively improving the controllability and safety of planning in complex environments.

[0111] Some embodiments of this application also disclose an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the above-described training method for the autonomous driving planning method or the autonomous driving planning verification system based on semantic scene flow.

[0112] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0113] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A training method for an autonomous driving planning and verification system, characterized in that, include: Acquire multimodal data at multiple consecutive moments in different scenarios, wherein the multimodal data includes a first original viewpoint image and point cloud data; For any given scenario: the point cloud data from all multimodal data corresponding to the scenario is input into an unsupervised joint network to obtain the semantic scene stream at each time step. In the unsupervised joint network, a coarse-level preprocessing module is used to obtain the dynamic regions in the point cloud data at each time step, and preliminary instance segmentation is performed on the dynamic regions to obtain all instances in the point cloud data at each time step. A spatiotemporal feature extraction module is used to obtain the global and local geometric features of the point cloud data at each time step. A dual-head task decoding module is used to obtain the semantic scene stream based on all instances, global geometric features, and local geometric features corresponding to each point cloud data. The semantic scene streams at all time steps are input into a diffusion model for Gaussian model initialization to obtain the initial Gaussian model corresponding to the scenario. The process involves: acquiring the time series corresponding to each moment, which includes the current moment and multiple historical moments preceding it; acquiring the first original trajectory corresponding to each moment and the lane / speed change information corresponding to all moments in the time series corresponding to each moment; generating a new trajectory corresponding to the current moment based on the first original trajectory and all multimodal data and lane / speed change information corresponding to the time series corresponding to the current moment, which is obtained through a dual-path planning mechanism; and obtaining the first new perspective image, the second original trajectory, and the second original perspective image corresponding to the current moment. Based on the semantic scene flow corresponding to each time step in the current scene, the initial Gaussian model corresponding to the current scene is iteratively optimized. Then, in any training epoch: For any given moment: input the semantic scene stream corresponding to that moment into the Gaussian model corresponding to the current scene, so as to render the second new perspective image corresponding to that moment according to the new trajectory, and render the third original perspective image corresponding to that moment according to the first original trajectory; A trajectory consistency loss is obtained based on the first original trajectory and the second original trajectory corresponding to each time step; a motion consistency loss is obtained based on the semantic scene flow and lane / speed change information corresponding to each time step; a comprehensive rendering loss is obtained based on the first new view image, the second new view image, and the third original view image corresponding to each time step; a first reconstruction loss is obtained based on the first original view image and the second original view image corresponding to each time step; a second reconstruction loss is obtained based on the first original view image and the third original view image corresponding to each time step; and a total loss function is obtained based on the trajectory consistency loss, the motion consistency loss, the comprehensive rendering loss, the first reconstruction loss, and the second reconstruction loss. The Gaussian model corresponding to the current scene is optimized based on the total loss function until the total loss function converges, thus obtaining the Gaussian verification model corresponding to the current scene.

2. The training method for the autonomous driving planning and verification system as described in claim 1, characterized in that, Also includes: After obtaining the new trajectory, the verification system performs closed-loop verification based on the new trajectory; wherein, the verification system simulates driving in its corresponding scenario according to the new trajectory, and obtains new point cloud data and renders a second new perspective image through the Gaussian verification model corresponding to the scenario; the new point cloud data and the rendered second new perspective image are input into the verification system as new multimodal data to continue obtaining new trajectories.

3. An autonomous driving planning method based on semantic scene flow, characterized in that, include: Acquire point cloud data corresponding to multiple consecutive time points; The point cloud data corresponding to multiple consecutive time points are input into a pre-trained unsupervised joint network to obtain a semantic scene stream. In the unsupervised joint network, a coarse-level preprocessing module obtains the dynamic regions in the point cloud data corresponding to each time point, and performs preliminary instance segmentation on the dynamic regions to obtain all instances in the point cloud data corresponding to each time point. A spatiotemporal feature extraction module obtains the global and local geometric features of the point cloud data corresponding to each time point. A dual-head task decoding module obtains the semantic scene stream based on all instances, global geometric features, and local geometric features corresponding to each point cloud data. Based on the dual-path planning mechanism, the first path planning result and / or the second path planning result are obtained according to the semantic scene flow; The autonomous driving planning result is obtained based on the first path planning result and / or the second path planning result, and the autonomous driving planning result is used as the new trajectory as described in any one of claims 1-2.

4. The autonomous driving planning method based on semantic scene flow as described in claim 3, characterized in that, The step of obtaining the global and local geometric features of the point cloud data at each time step through the spatiotemporal feature extraction module includes: For point cloud data at any given time: obtain the global geometric features of the point cloud data; The point cloud data is divided into voxels to obtain multiple voxels. Based on the feature vectors of all points within each voxel, the voxel-level feature representation and the point-level feature vector are iteratively updated for a predetermined number of rounds to obtain the local geometric features of the point cloud data. Specifically, in any given iteration: Obtain the initial feature vector corresponding to each point in the point cloud data in the current round; aggregate the initial feature vector of each point in any voxel to obtain the voxel-level feature representation corresponding to that voxel; The initial feature vector of each point in the voxel is updated according to the voxel-level feature representation corresponding to the voxel, so that the feature vector of each point is updated in the current round. If the current round is not the last round, then the feature vector of each point after being updated in the current round will be used as the initial feature vector of that point in the next round. If the current round is the last round, then the feature vector of each point after the update in the current round is the local geometric feature of the point cloud data.

5. The autonomous driving planning method based on semantic scene flow as described in claim 3, characterized in that, The semantic scene stream is obtained through the dual-head task decoding module based on all instances, global geometric features, and local geometric features corresponding to each point cloud data, including: The point cloud data corresponding to each two adjacent time points in a series of consecutive time points are taken as a group of point cloud data; the semantic scene stream includes the scene stream estimation result and point-level semantic labels corresponding to each group of point cloud data. The dual-head task decoding module includes a scene stream decoder and an instance segmentation decoder, wherein: The global and local geometric features corresponding to each point cloud data in each group of point cloud data are input into the scene flow decoder to perform scene flow estimation, and the scene flow estimation result corresponding to the group of point cloud data is obtained. All instances corresponding to each point cloud data in each group of point cloud data are input into the instance segmentation decoder for fine-grained instance segmentation to obtain the point-level semantic labels corresponding to the group of point cloud data.

6. The autonomous driving planning method based on semantic scene flow as described in claim 5, characterized in that, The step of inputting all instances corresponding to each point cloud data in each group of point cloud data into the instance segmentation decoder for fine-grained instance segmentation to obtain the point-level semantic labels corresponding to the group of point cloud data includes: The instance segmentation decoder is used to obtain the probability that each point in the point cloud data corresponding to the earliest time in each group of point cloud data belongs to a different instance category; for any point in the point cloud data corresponding to the earliest time: the instance category corresponding to the highest probability among all probabilities of the point is used as the semantic label of the point, and the point-level semantic label of the point cloud data is obtained. The point-level semantic label corresponding to the point cloud data is used as the point-level semantic label corresponding to the group of point cloud data.

7. The autonomous driving planning method based on semantic scene flow as described in claim 5, characterized in that, The dual-path planning mechanism, which obtains the first path planning result and / or the second path planning result based on the semantic scene flow, includes: The dual-path planning mechanism includes a point-based planner and a bounding box-based planner, wherein: The point cloud data corresponding to each time step and the scene flow estimation result corresponding to each set of point cloud data are input into the point-based planner to obtain the first path planning result; and / or, Rigid body objects are partitioned based on the point-level semantic labels corresponding to each set of point cloud data to obtain the bounding box of each rigid body object; the overall motion vector of each rigid body object is obtained based on the scene flow estimation results corresponding to the set of point cloud data; the bounding box of each rigid body object and the overall motion vector are input into the bounding box-based planner to obtain the second path planning result.

8. The autonomous driving planning method based on semantic scene flow as described in claim 7, characterized in that, The step of obtaining the overall motion vector of each rigid body object based on the scene flow estimation results corresponding to the set of point cloud data includes: The motion vector corresponding to each point is obtained based on the scene flow estimation results corresponding to this set of point cloud data; For any rigid body object: obtain the object center point of the rigid body object; determine the confidence weight of each point in the rigid body object based on the distance between each point and the object center point; obtain the overall motion vector of the rigid body object based on the confidence weight of each point in the rigid body object and the motion vector.

9. The autonomous driving planning method based on semantic scene flow as described in claim 3, characterized in that, The step of obtaining the autonomous driving planning result based on the first path planning result and / or the second path planning result includes: The first path planning result or the second path planning result shall be used as the autonomous driving planning result; or, The first path planning result and the second path planning result are dynamically weighted according to the complexity of the scenario and the requirements of the task. The autonomous driving planning result is obtained based on the first path planning result, the second path planning result and their corresponding weights.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the autonomous driving planning method based on semantic scene flow as described in any one of claims 3-9 or the training method of the autonomous driving planning verification system as described in any one of claims 1-2.

Images