A dynamic obstacle trajectory prediction planning method and system based on time sequence occupation grid

Abstract

The application relates to the technical field of automatic driving, in particular to a dynamic obstacle trajectory prediction planning method and system based on a time sequence occupation grid, a backup input channel independent of a main perception model is constructed at a planning level, and a single-point perception failure problem is fundamentally solved. Confidence evaluation is performed on a structured perception result output by a BEVTransformer model to determine whether the current perception result is reliable. When the current perception result is unreliable, a dynamic obstacle trajectory prediction planning scheme of the backup input channel is entered. The application completely bypasses the dependence on obstacle detection, classification and tracking, directly perceives and predicts motion from a geometric level, and is particularly effective when the BEV fails. Static and dynamic elements are expressed by a unified occupation grid, and an input interface of a planner is simplified. By predicting future occupation conditions, more forward-looking and safer decisions can be made, for example, early deceleration and yielding.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically a dynamic obstacle trajectory prediction and planning method and system based on temporal occupancy grids. Background Technology

[0002] Modern autonomous driving systems rely heavily on the output of upstream perception modules for their planning modules. Currently, the BEVTransformer-based perception architecture has become mainstream, capable of outputting rich structured scene information (lane lines, obstacle bounding boxes, etc.). The spatiotemporal joint planner then optimizes the trajectory based on this structured information.

[0003] However, this serial architecture carries the risk of a single point of failure. When the BEV perception model fails completely or outputs serious errors due to corner cases such as sudden changes in lighting, severe weather, or external objects, the downstream planner will be unable to generate a safe trajectory due to abnormal or missing inputs, which may lead to a vehicle collision or entering a minimum risk state, seriously threatening driving safety.

[0004] While some existing technologies use occupied grids as a supplement to BEV perception and improve perception accuracy through feature-level fusion, this post-fusion approach does not change the planner's reliance on a single perception backbone. Once the BEV backbone network fails, the fused features are also unreliable and cannot provide effective backup input for the planner. Furthermore, real-world road environments contain numerous dynamic obstacles (such as vehicles and pedestrians). Existing instance-based prediction modules (such as VectorNet) rely on obstacle recognition and tracking results output by BEV perception. In a "safe mode" where BEV perception fails, this prediction process also collapses. Therefore, how to handle dynamic obstacles in a "safe mode" remains an unresolved challenge. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a dynamic obstacle trajectory prediction and planning method and system based on temporal occupancy grids to solve the problems in the background art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] This application presents a dynamic obstacle trajectory prediction and planning method based on a temporally occupied grid, comprising the following steps:

[0008] The structured perception results output from the vehicle BEV Transformer model are obtained, along with image sequences captured by multiple cameras of the vehicle and point cloud data collected by LiDAR. The image sequences include multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points prior to the current time point.

[0009] A sensor data sequence is constructed based on the image sequence and point cloud data, and a temporal occupancy grid is generated based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points.

[0010] The confidence level of the structured perception results is evaluated to obtain the confidence level of the perception results;

[0011] When the confidence level of the perception result is less than a preset confidence level threshold, path planning is performed based on the time-series occupancy grid to obtain the planned path.

[0012] In one embodiment of this application, path planning is performed based on the time-series occupancy grid to obtain a planned path, including:

[0013] The temporal occupancy grid is input into a pre-constructed scene flow estimation function to obtain the scene flow field, wherein the scene flow field includes the displacement vector of each voxel in the time dimension;

[0014] Extract the current occupied grid at the current time point from the temporal occupied grid, and perform prediction based on the current occupied grid and the scene flow field to obtain a dynamic occupied probability field, wherein the dynamic occupied probability field includes the occupied probability of each voxel at multiple future time points;

[0015] An optimization problem is constructed based on the dynamic occupancy probability field, and a planner is used to solve the optimization problem to obtain a planned path. The objective of the optimization problem is to minimize the comprehensive cost, which includes obstacle avoidance cost and path smoothness cost.

[0016] In one embodiment of this application, a prediction is performed based on the currently occupied grid and the scene flow field to obtain a dynamic occupancy probability field, including:

[0017] For the target voxel in the currently occupied grid, a basic prediction formula is constructed, wherein the mathematical expression of the basic prediction formula is:

[0018]

[0019] In the formula, Indicates the current moment. For time steps, Let the coordinates of the target voxel be... Indicates the unit of time step. Indicates the target voxel in the future The probability of being occupied after the next step. Indicates in Occupy the grid at any time Represents the flow field of the scene. Indicates the target voxel from time point At the appointed time The displacement;

[0020] The basic prediction formula is solved using a recursive method to calculate the occupancy probability of the target voxel at multiple time points; and a dynamic occupancy probability field is constructed based on the occupancy probability of multiple voxels at multiple time points.

[0021] In one embodiment of this application, an optimization problem is constructed based on the dynamic occupancy probability field, including:

[0022] Define the coordinates of the planned trajectory at multiple time points to obtain the parametric trajectory. , wherein the parameterized trajectory The mathematical expression is:

[0023]

[0024] In the formula, Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in The position at that moment;

[0025] An optimization objective is constructed based on the dynamic occupancy probability field, wherein the mathematical expression of the optimization objective is:

[0026]

[0027]

[0028]

[0029] In the formula, For the overall cost, , and All are weighting coefficients. Indicates the smoothing cost. This indicates the cost of obstacle avoidance. Indicates position The neighborhood set of voxels, Indicates the time at which the planning begins. Predicted future moments voxels The probability of being occupied.

[0030] In one embodiment of this application, the structured perception result includes dynamic obstacle trajectories, static scene elements, drivable area boundaries, and semantic categories. A confidence assessment is performed on the structured perception result to obtain a perception result confidence score, including:

[0031] Based on the semantic categories, a classification deterministic assessment is performed to obtain the semantic recognition confidence level. ;

[0032] The boundary determinism of the dynamic obstacle trajectory and the drivable area boundary is evaluated to obtain the instance segmentation confidence. ;

[0033] The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. ;

[0034] A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. ;

[0035] Based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence level Constructing confidence scores for perception results The confidence level of the perception result The mathematical expression is:

[0036]

[0037] In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight.

[0038] In one embodiment of this application, a classification deterministic assessment is performed based on the semantic category to obtain the semantic recognition confidence level. ,include:

[0039] For each object in the semantic category, extract the predicted probabilities of all predicted categories for each object. ,in, Indicates the first The predicted probabilities of each category;

[0040] Predicted probability based on each prediction category Calculate the entropy value of each object. The entropy value The mathematical expression is:

[0041]

[0042] In the formula, Indicates the number of predicted categories;

[0043] The semantic recognition confidence score is obtained by taking the maximum value from the entropy values ​​of multiple objects. .

[0044] In one embodiment of this application, a boundary determinism assessment is performed on the trajectory of the dynamic obstacle and the boundary of the drivable area to obtain the instance segmentation confidence level. ,include:

[0045] Extract the masking coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ;

[0046] Based on the number of connected components corresponding to the instance segmentation mask Calculate connectivity score The mathematical expression for the connectivity is:

[0047]

[0048] Based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is mentioned above. The mathematical expression is:

[0049]

[0050] Based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is:

[0051]

[0052] In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

[0053] In one embodiment of this application, the temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain a temporal consistency confidence level. ,include:

[0054] Get the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number;

[0055] Based on Kalman filtering and the motion parameters of all objects in the previous frame image Predicted motion parameters of all objects in the next frame image ;

[0056] Calculate motion parameters in each frame of the image With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image;

[0057] Calculate the average rate of change of motion for each object to obtain the temporal consistency confidence score. .

[0058] In one embodiment of this application, a consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain a fusion confidence level. ,include:

[0059] Extract the first position from the camera from the structured perception results. and the second position from the lidar ;

[0060] Calculate the first position of the same object With the second position European distance ;

[0061] Calculate the Euclidean distance between multiple objects average distance ;

[0062] For the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is:

[0063]

[0064] In the formula, This is the maximum reference distance set.

[0065] This application also provides a dynamic obstacle trajectory prediction and planning system based on a temporal occupancy grid, including:

[0066] The acquisition module is used to acquire the structured perception results output from the vehicle BEV Transformer model, and to acquire image sequences collected by multiple cameras of the vehicle and point cloud data collected by LiDAR. The image sequence includes multiple frames of images collected by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point.

[0067] The meshing module is used to construct a sensor data sequence based on the image sequence and point cloud data, and to generate a temporal occupancy grid based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points.

[0068] The confidence assessment module is used to assess the confidence of the structured perception results and obtain the confidence of the perception results.

[0069] The replanning module is used to perform path planning based on the time-series occupancy grid to obtain a first planned path when the confidence level of the perception result is less than a preset confidence level threshold.

[0070] The beneficial effects of this application are as follows: This application provides a dynamic obstacle trajectory prediction and planning method and system based on a temporal occupancy grid. It constructs an alternative input path independent of the main perception model at the planning level, fundamentally solving the problem of single-point perception failure. By evaluating the confidence level of the structured perception results output by the BEVTransformer model, the reliability of the current perception result is determined. When unreliable, the dynamic obstacle trajectory prediction and planning scheme of the alternative input path is adopted. This application completely bypasses the dependence on obstacle detection, classification, and tracking, directly perceiving and predicting motion at the geometric level, which is particularly effective when BEV fails. Using a unified occupancy grid to represent static and dynamic elements simplifies the planner's input interface. By predicting future occupancy situations, more forward-looking and safer decisions can be made, such as slowing down and yielding in advance. Attached Figure Description

[0071] The present application will be further described below with reference to the accompanying drawings and embodiments:

[0072] Figure 1 This is a flowchart illustrating a dynamic obstacle trajectory prediction and planning method based on a temporal occupancy grid in one embodiment of this application;

[0073] Figure 2 This is a flowchart illustrating path planning based on an occupied grid in one embodiment of this application;

[0074] Figure 3This is a structural diagram of a dynamic obstacle trajectory prediction and planning system based on a temporal occupancy grid, as shown in one embodiment of this application. Detailed Implementation

[0075] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0076] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the layers related to this application and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0077] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of this application; however, it will be apparent to those skilled in the art that embodiments of this application may be practiced without these specific details.

[0078] Figure 1 This is a flowchart illustrating a dynamic obstacle trajectory prediction and planning method based on a temporal occupancy grid in one embodiment of this application, as shown below. Figure 1 As shown, this embodiment of a dynamic obstacle trajectory prediction and planning method based on a temporal occupancy grid includes the following steps:

[0079] S110, acquire the structured perception results output from the vehicle BEV Transformer model, and acquire the image sequence collected by multiple cameras of the vehicle and the point cloud data collected by the LiDAR, wherein the image sequence includes multiple frames of images collected by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point;

[0080] The structured perception result is the first type of perception result in this application; the first type of perception result is a structured description containing the semantics and instances of objects in the scene, such as the dynamic obstacle trajectory, static scene elements, drivable area boundaries and semantic categories output by the BEV Transformer model.

[0081] S120, a sensor data sequence is constructed based on the image sequence and point cloud data, and a temporal occupancy grid is generated based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points.

[0082] The three-dimensional occupancy grid is the second type of perception result output by this application. The second type of perception result is a dense geometric occupancy description of the three-dimensional space of the scene, such as an occupancy grid, whose output is the occupancy probability of each voxel.

[0083] S130, perform a confidence assessment on the structured perception results to obtain the confidence level of the perception results;

[0084] This application aims to construct a safe passage independent of mainstream BEV perception. By directly introducing occupancy grid information at the planning level, perception redundancy is achieved, ensuring that the system can still output a safe drivable trajectory based on geometric occupancy information when BEV perception fails, thereby greatly improving the functional safety and robustness of the autonomous driving system.

[0085] Specifically, the process of evaluating the confidence level of the structured perception results in this application to obtain the confidence level of the perception results includes:

[0086] S131, perform a classification deterministic assessment based on the semantic category to obtain the semantic recognition confidence level. Semantic recognition confidence The extraction methods include:

[0087] S1311, Extract the prediction probabilities of all predicted categories for each object from the semantic categories. ,in, Indicates the first The predicted probabilities of each category;

[0088] S1312, Based on the predicted probability of each predicted category Calculate the entropy value of each object. The entropy value The mathematical expression is:

[0089]

[0090] In the formula, Indicates the number of predicted categories;

[0091] S1313: Take the maximum value from the entropy values ​​of multiple objects to obtain the semantic recognition confidence score. .

[0092] In this application, semantic confidence is the entropy value of the recognition probability distribution of each object. The lower the entropy, the more certain the recognition (the more concentrated the probability distribution), for example:

[0093] The entropy of the probability distribution [0.8, 0.15, 0.05] is:

[0094] -0.8log(0.8)-0.15log(0.15)-0.05log(0.05)≈0.45

[0095] The entropy of the probability distribution [0.95, 0.03, 0.02] is: -0.95log(0.95) - 0.03log(0.03) - 0.02log(0.02) ≈ 0.22

[0096] Therefore, the smaller the entropy value (0.22 < 0.45), the more certain the identification.

[0097] Finally, to ensure the certainty of recognition and the safety of vehicle operation, the maximum entropy value among multiple objects is taken as the semantic recognition confidence score. ,Right now:

[0098]

[0099] S132, perform boundary deterministic evaluation on the dynamic obstacle trajectory and the drivable area boundary to obtain the instance segmentation confidence score. Specifically, the instance segmentation confidence score is constructed based on features such as the connectivity and boundary clarity of the segmentation mask. The calculation methods include:

[0100] S1321, Extract the mask coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ;

[0101] Specifically, the BEV Transformer model outputs mask coefficients for each detected object. These coefficients represent the segmentation confidence of the object and are typically values ​​between -1 and 1. In the BEV Transformer, these coefficients are output through the model's segmentation branch.

[0102] Number of connected regions This indicates the segmentation of connectivity. If the segmentation result of a vehicle is divided into multiple unconnected regions, the system may mistakenly identify them as multiple vehicles, leading to planning errors.

[0103] variance of boundary gradient This is used to assess the clarity of boundaries. If the boundaries of pedestrians or vehicles are unclear, the system may not be able to accurately determine the size and position of the object, affecting obstacle avoidance decisions.

[0104] S1322, Based on the number of connected regions corresponding to the instance segmentation mask. Calculate connectivity score The mathematical expression for the connectivity is:

[0105]

[0106] Example 1: Ideal segmentation case

[0107] Original segmentation mask: a complete car outline.

[0108] Connectivity analysis result: 1 connected region.

[0109] Connectivity score = 1 / 1 = 1.0.

[0110] Example 2: Segmentation error

[0111] Original segmentation mask: The car is incorrectly segmented into three unconnected regions.

[0112] Connectivity analysis results: 3 connected regions.

[0113] Connectivity score = 1 / 3 ≈ 0.33.

[0114] S1323, based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is mentioned above. The mathematical expression is:

[0115]

[0116] For example:

[0117] Example 1: Clear boundary case

[0118] Original segmentation mask: The car outline has smooth and sharp boundaries.

[0119] Boundary gradient calculation: The gradient magnitude varies greatly and the variance is high on the boundary.

[0120] Gradient variance = 15.2.

[0121] Boundary clarity score = 1 - 1 / (1 + 15.2) ≈ 0.94.

[0122] Example 2: Fuzzy boundary cases

[0123] Original segmentation mask: The outline of the car is blurred and discontinuous.

[0124] Boundary gradient calculation: The gradient magnitude changes little and the variance is low on the boundary.

[0125] Gradient variance = 2.3.

[0126] Boundary clarity score = 1 - 1 / (1 + 2.3) ≈ 0.70.

[0127] S224, based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is:

[0128]

[0129] In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

[0130] Finally, the dimensionless normalized values ​​are weighted and the confidence levels are divided. .

[0131] S133, Evaluate the temporal stability of the dynamic obstacle trajectory and the static scene elements to obtain the temporal consistency confidence level. ;

[0132] In this application, for each detected object, the differences in its position, velocity, orientation, and other attributes between consecutive frames are calculated, and the stability of the perception results is measured based on these differences, specifically, the temporal consistency confidence level. The calculation methods include:

[0133] S1331, Obtain the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number;

[0134] S1332, based on Kalman filtering and the motion parameters of all objects in the previous frame image. Predicted motion parameters of all objects in the next frame image ;

[0135] S1333, calculate the motion parameters for each frame of the image. With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image;

[0136] S1334, calculate the average rate of change of motion for each object to obtain the temporal consistency confidence level. .

[0137] In this application, Kalman filtering is used to predict the possible position of the object in the next frame, and the error between the actual detected position and the predicted position is calculated. The smaller the error, the higher the temporal consistency.

[0138] For ease of calculation, the rates of change calculated in this application are all dimensionless parameters. For example, when the motion parameter is velocity, the rate of change is:

[0139]

[0140] When the motion parameter is position, the rate of change is:

[0141]

[0142] In the formula, This represents the distance between the predicted location and the actual location. Typical distance;

[0143] When the motion parameter is in direction, the rate of change is:

[0144]

[0145] in, This indicates the angle between the predicted direction and the actual direction. Indicates a typical included angle.

[0146] S134, A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the LiDAR to obtain the fusion confidence score. ;

[0147] This application calculates the fusion confidence level by comparing the consistency of perception results of the same object from different sensors (such as cameras and radar). The specific process includes:

[0148] S1341, Extract the first position from the camera from the structured perception result. and the second position from the lidar ;

[0149] S1342, Calculate the first position of the same object. With the second position European distance ;

[0150] S1343, Calculate the Euclidean distance between multiple objects. average distance ;

[0151] S1344, regarding the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is:

[0152]

[0153] In the formula, This is the maximum reference distance set.

[0154] Specifically, in the above process, the perception results of different sensors are projected onto the same coordinate system (such as the BEV coordinate system); the distance between the detection results of each sensor for the same object is calculated.

[0155] For example: If the camera detects a vehicle at position (10, 5) and the radar detects the same vehicle at position (9.8, 5.2), then the distance = (10 - 9.8). 2 +(5-5.2) 2 =0.28m; if the maximum allowable difference is 0.5m, then the consistency score = 1 - 0.28 / 0.5 = 0.44.

[0156] S250, based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence level Constructing confidence scores for perception results The confidence level of the perception result The mathematical expression is:

[0157]

[0158] In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight.

[0159] Finally, the confidence scores of the first type of perception results are weighted to obtain the comprehensive confidence score. The threshold for the comprehensive confidence score can be dynamically adjusted according to the environmental complexity, for example:

[0160] Sunny weather: Threshold set to 0.85.

[0161] Rainy / foggy weather: The threshold is automatically reduced to 0.75.

[0162] Low light at night: The threshold was further reduced to 0.7.

[0163] To avoid frequent mode switching caused by sudden threshold changes, a sliding window average is used to calculate the current environmental conditions and smoothly adjust the threshold.

[0164] The bimodal planner monitors the confidence score of the first sensing pathway in real time.

[0165] If the confidence level is higher than the threshold, it is in normal mode, and the planner uses the first type of perception results to perform high-precision and comfortable trajectory planning.

[0166] If the confidence level is below the threshold, it indicates that BEV perception may have failed, and the system immediately switches to safe mode. At this time, the planner ignores the first type of perception results and instead performs planning based on the second type of perception results (occupied grid). Figure 2 Here is a flowchart of path planning based on an occupied grid in one embodiment of this application. Please refer to [link / reference]. Figure 2 Based on the understanding, path planning is performed according to the occupied grid as follows.

[0167] S140, when the confidence level of the perception result is less than a preset confidence level threshold, path planning is performed based on the time-series occupancy grid to obtain the planned path.

[0168] In this application, when BEV perception may have failed, path planning is performed through the following process:

[0169] S141, The temporal occupancy grid is input into a pre-constructed scene flow estimation function to obtain the scene flow field, wherein the scene flow field includes the displacement vector of each voxel in the time dimension;

[0170] In the preceding text, the generated temporal occupancy grid was represented as... ,in, They represent from Time's up The grid is occupied at any given moment;

[0171] Then, the existing scene flow estimation function is used to generate the scene flow field, which is represented as follows:

[0172]

[0173] Indicates from time Time Scene flow, Voxel representation In time Time The displacement vector.

[0174] S142, extract the current occupied grid at the current time point from the temporal occupied grid, and perform prediction based on the current occupied grid and the scene flow field to obtain a dynamic occupied probability field, wherein the dynamic occupied probability field includes the occupied probability of each voxel at multiple future time points;

[0175] After obtaining the temporal occupancy grid, based on the current occupancy grid and the scene flow, the occupancy grid for future time steps is predicted, resulting in a dynamic occupancy probability field. The process includes:

[0176] S1421, For the target voxel in the currently occupied grid, construct a basic prediction formula, wherein the mathematical expression of the basic prediction formula is:

[0177]

[0178] In the formula, Indicates the current moment. For time steps, Let the coordinates of the target voxel be... Indicates the unit of time step. Indicates the target voxel in the future The probability of being occupied after the next step. Indicates in Occupy the grid at any time Represents the flow field of the scene. Indicates the target voxel from time point At the appointed time The displacement;

[0179] The principle of the basic prediction formula is that the probability of occupancy moves with the object and satisfies the law of conservation of mass.

[0180] To predict whether a location will be occupied in the future, it is necessary to trace "which object is currently in which location will move to that location";

[0181] Consider the superposition of displacements in multi-step motion and adapt to acceleration / turning motion.

[0182] S1422, Solve the basic prediction formula based on the recursive method to calculate the occupancy probability of the target voxel at multiple time points; and construct a dynamic occupancy probability field based on the occupancy probability of multiple voxels at multiple time points.

[0183] Specifically, in practical implementation, the dynamic occupancy probability field can be calculated recursively, including:

[0184]

[0185]

[0186] in, An interpolation function representing the voxel position, used to predict the occupancy state at the next time step based on the scene flow.

[0187] A recursive approach is used to solve the problem, avoiding repeated calculations of predictions at the same time steps, while naturally propagating the uncertainty of motion estimation during the recursive process.

[0188] S143, construct an optimization problem based on the dynamic occupancy probability field, and use a planner to solve the optimization problem to obtain a planned path. The objective of the optimization problem is to minimize the comprehensive cost, which includes obstacle avoidance cost and path smoothness cost.

[0189] After obtaining the dynamic occupancy probability field, we can construct the following optimization problem and use a planner to solve it:

[0190] S1431 defines the coordinates of the planned trajectory at multiple time points, resulting in the parametric trajectory. , wherein the parameterized trajectory The mathematical expression is:

[0191]

[0192] In the formula, Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in The position at that moment;

[0193] S1432, Construct an optimization objective based on the dynamic occupancy probability field, wherein the mathematical expression of the optimization objective is:

[0194]

[0195]

[0196]

[0197] In the formula, For the overall cost, , and All are weighting coefficients. Indicates the smoothing cost. This indicates the cost of obstacle avoidance. Indicates position The neighborhood set of voxels, Indicates the time at which the planning begins. Predicted future moments voxels The probability of being occupied.

[0198] In the above optimization problem, the overall cost By balancing smoothness and safety through weight λ, obstacle avoidance costs are reduced. By using an exponentially decaying potential field, obstacles at close range incur high costs, while having minimal impact at long ranges.

[0199] This application is implemented based on the following overall steps:

[0200] Temporal Occupancy Mesh Generation: Based on sensor data from multiple consecutive time steps, a 3D occupancy mesh corresponding to each time step is generated, forming a temporal occupancy mesh sequence.

[0201] Scene flow estimation: The temporal occupancy grid sequence is processed to estimate the motion displacement of each voxel between consecutive frames, i.e., to calculate the 3D scene flow field.

[0202] Dynamic occupancy probability field prediction: Based on the occupancy grid at the current moment and the estimated scene flow field, the occupancy grid for multiple future time steps is extrapolated to form a dynamic occupancy probability field.

[0203] Time-varying obstacle avoidance planning: In safe mode, the bimodal planner uses the predicted dynamic occupancy probability field for planning. The obstacle distance term in its cost function becomes a time-varying function, ensuring that the planned trajectory avoids the occupancy area predicted to be reached not only at the current moment but also over a period of time in the future.

[0204] The core value of this application lies in providing deterministic safety guarantees under perceptual uncertainty. Instead of attempting to restore perception capabilities when perception fails, it bypasses perception and directly uses raw data for planning. This roundabout strategy is often more feasible and reliable in practical engineering than directly addressing the perception problem. As autonomous driving evolves from L2 to L3 / L4, such degraded safety mechanisms will become a regulatory requirement and a necessary condition for commercial success. This solution provides a specific technical implementation path for this, possessing significant theoretical and practical value.

[0205] like Figure 3 As shown, this application also provides a dynamic obstacle trajectory prediction and planning system based on a temporal occupancy grid, comprising:

[0206] The acquisition module is used to acquire the structured perception results output from the vehicle BEV Transformer model, and to acquire image sequences collected by multiple cameras of the vehicle and point cloud data collected by LiDAR. The image sequence includes multiple frames of images collected by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point.

[0207] The meshing module is used to construct a sensor data sequence based on the image sequence and point cloud data, and to generate a temporal occupancy grid based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points.

[0208] The confidence assessment module is used to assess the confidence of the structured perception results and obtain the confidence of the perception results.

[0209] The replanning module is used to perform path planning based on the time-series occupancy grid to obtain a first planned path when the confidence level of the perception result is less than a preset confidence level threshold.

[0210] This application presents a dynamic obstacle trajectory prediction and planning method and system based on a temporal occupancy grid. It constructs an alternative input path independent of the main perception model at the planning level, fundamentally solving the problem of single-point perception failure. The reliability of the current perception result is determined by evaluating the confidence level of the structured perception results output by the BEVTransformer model. If the result is unreliable, the dynamic obstacle trajectory prediction and planning scheme of the alternative input path is adopted. This application completely bypasses the dependence on obstacle detection, classification, and tracking, directly perceiving and predicting motion at the geometric level, which is particularly effective when BEV fails. A unified occupancy grid is used to represent static and dynamic elements, simplifying the planner's input interface. By predicting future occupancy situations, more forward-looking and safer decisions can be made, such as slowing down and yielding in advance.

[0211] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0212] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory so that the terminal performs any of the methods in this embodiment.

[0213] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0214] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0215] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0216] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0217] In the above embodiments, although the present application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. The embodiments of the present application are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0218] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A dynamic obstacle trajectory prediction and planning method based on temporal occupancy grids, characterized in that, Including the following steps: The structured perception results output from the vehicle BEV Transformer model are obtained, along with image sequences captured by multiple cameras of the vehicle and point cloud data collected by LiDAR. The image sequences include multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points prior to the current time point. A sensor data sequence is constructed based on the image sequence and point cloud data, and a temporal occupancy grid is generated based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points. The structured perception results are evaluated for confidence to obtain a perception result confidence score. The structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries, and semantic categories. The confidence evaluation of the structured perception results to obtain the perception result confidence score includes: performing a classification deterministic evaluation based on the semantic categories to obtain a semantic recognition confidence score. The dynamic obstacle trajectory and the drivable area boundary are evaluated for boundary determinism to obtain the instance segmentation confidence score. The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. Based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence Constructing confidence of perception results The confidence level of the perception result The mathematical expression is: In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight; When the confidence level of the perception result is less than a preset confidence threshold, path planning is performed based on the temporal occupancy grid to obtain a planned path. This process includes: inputting the temporal occupancy grid into a pre-constructed scene flow estimation function to obtain a scene flow field, where the scene flow field includes the displacement vector of each voxel in the time dimension; extracting the current occupancy grid at the current time point from the temporal occupancy grid, and performing prediction based on the current occupancy grid and the scene flow field to obtain a dynamic occupancy probability field, where the dynamic occupancy probability field includes the occupancy probability of each voxel at multiple future time points; constructing an optimization problem based on the dynamic occupancy probability field, and solving the optimization problem using a planner to obtain a planned path, where the objective of the optimization problem is to minimize the overall cost, which includes obstacle avoidance cost and path smoothness cost.

2. The dynamic obstacle trajectory prediction and planning method based on temporal occupancy grid according to claim 1, characterized in that, Based on the currently occupied grid and the scene flow field, a prediction is performed to obtain a dynamic occupancy probability field, including: For the target voxel in the currently occupied grid, a basic prediction formula is constructed, wherein the mathematical expression of the basic prediction formula is: In the formula, Indicates the current moment. For time steps, Let the coordinates of the target voxel be... Indicates the unit of time step. Indicates the target voxel in the future The probability of being occupied after the next step. Indicates in Occupy the grid at any time Represents the flow field of the scene. Indicates the target voxel from time point At the appointed time The displacement; The basic prediction formula is solved using a recursive method to calculate the occupancy probability of the target voxel at multiple time points; and a dynamic occupancy probability field is constructed based on the occupancy probability of multiple voxels at multiple time points.

3. The dynamic obstacle trajectory prediction and planning method based on a temporal occupancy grid according to claim 2, characterized in that, The optimization problem is constructed based on the dynamic occupancy probability field, including: Define the coordinates of the planned trajectory at multiple time points to obtain the parametric trajectory. , wherein the parameterized trajectory The mathematical expression is: In the formula, Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in Location at any given moment Indicates that the vehicle is in The position at that moment; An optimization objective is constructed based on the dynamic occupancy probability field, wherein the mathematical expression of the optimization objective is: In the formula, For the overall cost, , and All are weighting coefficients. Indicates the smoothing cost. This indicates the cost of obstacle avoidance. Indicates position The neighborhood voxel set, Indicates the time at which the planning begins Predicted future moments voxels The probability of being occupied.

4. The dynamic obstacle trajectory prediction and planning method based on temporal occupancy grid according to claim 1, characterized in that, Based on the semantic categories, a classification deterministic assessment is performed to obtain the semantic recognition confidence level. ,include: For each object in the semantic category, extract the predicted probabilities of all predicted categories for each object. ,in, Indicates the first The predicted probabilities of each category; Predicted probability based on each prediction category Calculate the entropy value of each object. The entropy value The mathematical expression is: In the formula, Indicates the number of predicted categories; The semantic recognition confidence score is obtained by taking the maximum value from the entropy values ​​of multiple objects. .

5. The dynamic obstacle trajectory prediction and planning method based on temporal occupancy grid according to claim 1, characterized in that, The boundary determinism of the dynamic obstacle trajectory and the drivable area boundary is evaluated to obtain the instance segmentation confidence. ,include: Extract the masking coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ; Based on the number of connected components corresponding to the instance segmentation mask Calculate connectivity score The mathematical expression for the connectivity is: Based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is... The mathematical expression is: Based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is: In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

6. The dynamic obstacle trajectory prediction and planning method based on temporal occupancy grid according to claim 1, characterized in that, The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. ,include: Get the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number; Based on Kalman filtering and the motion parameters of all objects in the previous frame image Predicted motion parameters of all objects in the next frame image ; Calculate motion parameters in each frame of the image With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image; Calculate the average rate of change of motion for each object to obtain the temporal consistency confidence score. .

7. The dynamic obstacle trajectory prediction and planning method based on temporal occupancy grid according to claim 1, characterized in that, A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. ,include: Extract the first position from the camera from the structured perception results. and the second position from the lidar ; Calculate the first position of the same object With the second position European distance ; Calculate the Euclidean distance between multiple objects average distance ; For the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is: In the formula, This is the maximum reference distance set.

8. A dynamic obstacle trajectory prediction and planning system based on temporal occupancy grids, characterized in that, include: The acquisition module is used to acquire the structured perception results output from the vehicle BEV Transformer model, and to acquire image sequences collected by multiple cameras of the vehicle and point cloud data collected by LiDAR. The image sequence includes multiple frames of images collected by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point. The meshing module is used to construct a sensor data sequence based on the image sequence and point cloud data, and to generate a temporal occupancy grid based on a pre-constructed occupancy grid generation model, wherein the temporal occupancy grid includes the occupancy probability of multiple voxels at multiple time points. The confidence assessment module is used to assess the confidence of the structured perception results to obtain a perception result confidence score. The structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries, and semantic categories. Assessing the confidence of the structured perception results to obtain the perception result confidence score includes: performing a classification deterministic assessment based on the semantic categories to obtain a semantic recognition confidence score. The dynamic obstacle trajectory and the drivable area boundary are evaluated for boundary determinism to obtain the instance segmentation confidence score. The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. Based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence Constructing confidence of perception results The confidence level of the perception result The mathematical expression is: In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight; A replanning module is used to perform path planning based on the temporal occupancy grid to obtain a first planned path when the confidence level of the perception result is less than a preset confidence level threshold; and to perform path planning based on the temporal occupancy grid to obtain a planned path, including: inputting the temporal occupancy grid into a pre-constructed scene flow estimation function to obtain a scene flow field, wherein the scene flow field includes the displacement vector of each voxel in the time dimension; extracting the current occupancy grid at the current time point from the temporal occupancy grid, and performing prediction based on the current occupancy grid and the scene flow field to obtain a dynamic occupancy probability field, wherein the dynamic occupancy probability field includes the occupancy probability of each voxel at multiple future time points; constructing an optimization problem based on the dynamic occupancy probability field, and solving the optimization problem using a planner to obtain a planned path, wherein the objective of the optimization problem is to minimize the comprehensive cost, wherein the comprehensive cost includes obstacle avoidance cost and path smoothness cost.

Images