Robust planning system and method for autonomous driving based on dual-lane perception

Abstract

The application relates to the technical field of automatic driving, in particular to an automatic driving robust planning system and method based on double-path perception, the automatic driving robust planning system and method based on double-path perception, and the application constructs a backup input path independent of a main perception model at a planning level, fundamentally solving a single-point failure problem of perception. Automatic and seamless mode switching is realized through confidence monitoring, and high performance under normal conditions and high safety under abnormal conditions are considered. The design target of a second perception path (an occupancy grid) is more single (geometric occupancy), and the second perception path can be made lighter and more robust, and is complementary to the first path. The architecture in the application meets the requirements of redundancy and safety architecture in the functional safety standard (such as ISO 26262) of the automobile industry, and helps to improve the overall safety level of the automatic driving system.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a robust planning system and method for autonomous driving based on dual-path perception. Background Technology

[0002] Modern autonomous driving systems rely heavily on the output of upstream perception modules for their planning modules. Currently, the BEVTransformer-based perception architecture has become mainstream, capable of outputting rich structured scene information (lane lines, obstacle bounding boxes, etc.). The spatiotemporal joint planner then optimizes the trajectory based on this structured information.

[0003] However, this serial architecture carries the risk of a single point of failure. When the BEV perception model fails completely or outputs serious errors due to corner cases such as sudden changes in lighting, severe weather, or external objects, the downstream planner will be unable to generate a safe trajectory due to abnormal or missing inputs, which may lead to a vehicle collision or entering a minimum risk state, seriously threatening driving safety.

[0004] While some existing technologies use occupied grids as a supplement to BEV perception and improve perception accuracy through feature-level fusion, this post-fusion approach does not change the planner's reliance on a single perception backbone. Once the BEV backbone network fails, the fused features are also unreliable and cannot provide effective backup input for the planner. Therefore, there is an urgent need in this field for a redundant architecture that can maintain basic planning functions even when the perception backbone fails. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a robust planning system and method for autonomous driving based on dual-path perception, so as to solve the problems in the background art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] The robust planning system for autonomous driving based on dual-path perception in this application includes:

[0008] The acquisition module is used to acquire image sequences captured by multiple cameras of the vehicle and point cloud data acquired by LiDAR. The image sequence includes multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point.

[0009] The first perception pathway module is used to input the image sequence and the point cloud data into a pre-built BEVTransformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries and semantic categories.

[0010] The second perception pathway module is used to input the image sequence and the point cloud data into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels.

[0011] The bimodal planning module is used to evaluate the confidence level of the structured perception results to obtain the confidence level of the perception results. When the confidence level of the perception results is greater than or equal to a preset confidence level threshold, the structured perception results are output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level of the perception results is less than the preset confidence level threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

[0012] In one embodiment of this application, the confidence level of the structured perception result is evaluated to obtain the confidence level of the perception result, including:

[0013] Based on the semantic categories, a classification deterministic assessment is performed to obtain the semantic recognition confidence level. ;

[0014] The boundary determinism of the dynamic obstacle trajectory and the drivable area boundary is evaluated to obtain the instance segmentation confidence. ;

[0015] The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. ;

[0016] A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. ;

[0017] Based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence Constructing confidence of perception results The confidence level of the perception result The mathematical expression is:

[0018]

[0019] In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight.

[0020] In one embodiment of this application, a classification deterministic assessment is performed based on the semantic category to obtain the semantic recognition confidence level. ,include:

[0021] For each object in the semantic category, extract the predicted probabilities of all predicted categories for each object. ,in, Indicates the first The predicted probabilities of each category;

[0022] Predicted probability based on each prediction category Calculate the entropy value of each object. The entropy value The mathematical expression is:

[0023]

[0024] In the formula, Indicates the number of predicted categories;

[0025] The semantic recognition confidence score is obtained by taking the maximum value from the entropy values ​​of multiple objects. .

[0026] In one embodiment of this application, a boundary determinism assessment is performed on the trajectory of the dynamic obstacle and the boundary of the drivable area to obtain the instance segmentation confidence level. ,include:

[0027] Extract the masking coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ;

[0028] Based on the number of connected components corresponding to the instance segmentation mask Calculate connectivity score The mathematical expression for the connectivity is:

[0029]

[0030] Based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is mentioned above. The mathematical expression is:

[0031]

[0032] Based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is:

[0033]

[0034] In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

[0035] In one embodiment of this application, the temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain a temporal consistency confidence level. ,include:

[0036] Get the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number;

[0037] Based on Kalman filtering and the motion parameters of all objects in the previous frame image Predicted motion parameters of all objects in the next frame image ;

[0038] Calculate motion parameters in each frame of the image With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image;

[0039] Calculate the average rate of change of motion for each object to obtain the temporal consistency confidence score. .

[0040] In one embodiment of this application, a consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain a fusion confidence level. ,include:

[0041] Extract the first position from the camera from the structured perception results. and the second position from the lidar ;

[0042] Calculate the first position of the same object With the second position European distance ;

[0043] Calculate the Euclidean distance between multiple objects average distance ;

[0044] For the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is:

[0045]

[0046] In the formula, This is the maximum reference distance set.

[0047] In one embodiment of this application, path planning is performed based on the three-dimensional occupancy grid to obtain a second planned path, including:

[0048] Extract target voxels whose occupied probability is higher than a preset probability threshold from the three-dimensional occupied grid;

[0049] Construct a cost function, wherein the cost function includes path smoothing cost, velocity cost, obstacle cost and orientation cost, and the obstacle cost is determined based on the nearest distance between the path node and the target voxel;

[0050] Based on the cost function and the three-dimensional occupied grid, the RRT* fast random tree algorithm is executed to obtain the second planned path.

[0051] In one embodiment of this application, the mathematical expression of the cost function is:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] In the formula, , , and All are weighting coefficients. Represents the cost of smoothness. This indicates a typical value representing the degree of path curvature. Indicates the cost of speed. Indicates the cost of obstacles. Indicates the cost of direction. This represents the position vector of the path in space. For arc length parameters, For vehicle speed, To limit the speed, This represents the typical speed value. Representing path points The closest distance to the target voxel. Represents the path dimension element. The path direction vector. The target direction vector.

[0058] This application also provides a robust planning method for autonomous driving based on dual-path perception, including the following steps:

[0059] The system acquires image sequences captured by multiple cameras of the vehicle and point cloud data collected by LiDAR, wherein the image sequences include multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points prior to the current time point;

[0060] The image sequence and the point cloud data are input into a pre-built BEV Transformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries, and semantic categories.

[0061] The image sequence and the point cloud data are input into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels;

[0062] The structured perception results are evaluated for confidence to obtain a confidence level. When the confidence level is greater than or equal to a preset confidence threshold, the structured perception results are output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level is less than the preset confidence threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

[0063] The beneficial effects of this application are as follows: The robust planning system and method for autonomous driving based on dual-path perception constructs an alternative input path independent of the main perception model at the planning level, fundamentally solving the problem of single-point perception failure. Automatic and seamless mode switching is achieved through confidence monitoring, while simultaneously ensuring high performance under normal conditions and high safety under abnormal conditions. The design goal of the second perception path (occupancy grid) is more singular (geometric occupancy), allowing it to be made lighter and more robust, complementing the first path. The architecture in this application complies with the requirements for redundancy and safety architecture in automotive industry functional safety standards (such as ISO 26262), contributing to improving the overall safety level of the autonomous driving system. Attached Figure Description

[0064] The present application will be further described below with reference to the accompanying drawings and embodiments:

[0065] Figure 1This is a structural diagram of an autonomous driving robust planning system based on dual-path perception, as shown in one embodiment of this application.

[0066] Figure 2 This is a flowchart illustrating a robust planning method for autonomous driving based on dual-path perception, as shown in one embodiment of this application. Detailed Implementation

[0067] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0068] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the layers related to this application and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0069] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of this application; however, it will be apparent to those skilled in the art that embodiments of this application may be practiced without these specific details.

[0070] Figure 1 This is a structural diagram of a robust autonomous driving planning system based on dual-path perception, as shown in one embodiment of this application. Figure 1 As shown, the autonomous driving robust planning system based on dual-path perception in this embodiment includes:

[0071] The acquisition module is used to acquire image sequences captured by multiple cameras of the vehicle and point cloud data acquired by LiDAR. The image sequence includes multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point.

[0072] The first perception pathway module is used to input the image sequence and the point cloud data into a pre-built BEVTransformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries and semantic categories.

[0073] The structured perception result is the first type of perception result in this application; the first type of perception result is a structured description containing the semantics and instances of objects in the scene, such as the dynamic obstacle trajectory, static scene elements, drivable area boundaries and semantic categories output by the BEV Transformer model.

[0074] The second perception pathway module is used to input the image sequence and the point cloud data into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels.

[0075] The three-dimensional occupancy grid is the second type of perception result output by this application. The second type of perception result is a dense geometric occupancy description of the three-dimensional space of the scene, such as an occupancy grid, whose output is the occupancy probability of each voxel.

[0076] The bimodal planning module is used to evaluate the confidence level of the structured perception results to obtain the confidence level of the perception results. When the confidence level of the perception results is greater than or equal to a preset confidence level threshold, the structured perception results are output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level of the perception results is less than the preset confidence level threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

[0077] This application aims to construct a safe passage independent of mainstream BEV perception. By directly introducing occupancy grid information at the planning level, perception redundancy is achieved, ensuring that the system can still output a safe drivable trajectory based on geometric occupancy information when BEV perception fails, thereby greatly improving the functional safety and robustness of the autonomous driving system.

[0078] Specifically, the process of evaluating the confidence level of the structured perception results in this application to obtain the confidence level of the perception results includes:

[0079] S210, perform a classification deterministic assessment based on the semantic category to obtain the semantic recognition confidence level. Semantic recognition confidence The extraction methods include:

[0080] S211, extract the prediction probabilities of all predicted categories for each object from the semantic categories. ,in, Indicates the first The predicted probabilities of each category;

[0081] S212, based on the predicted probability for each predicted category Calculate the entropy value of each object. The entropy value The mathematical expression is:

[0082]

[0083] In the formula, Indicates the number of predicted categories;

[0084] S213, take the maximum value from the entropy values ​​of multiple objects to obtain the semantic recognition confidence score. .

[0085] In this application, semantic confidence is the entropy value of the recognition probability distribution of each object. The lower the entropy, the more certain the recognition (the more concentrated the probability distribution), for example:

[0086] The entropy of the probability distribution [0.8, 0.15, 0.05] is:

[0087] -0.8log(0.8)-0.15log(0.15)-0.05log(0.05)≈0.45.

[0088] The entropy of the probability distribution [0.95, 0.03, 0.02] is:

[0089] -0.95log(0.95)-0.03log(0.03)-0.02log(0.02)≈0.22.

[0090] Therefore, the smaller the entropy value (0.22 < 0.45), the more certain the identification.

[0091] Finally, to ensure the certainty of recognition and the safety of vehicle operation, the maximum entropy value among multiple objects is taken as the semantic recognition confidence score. ,Right now:

[0092]

[0093] S220, perform boundary deterministic evaluation on the trajectory of the dynamic obstacle and the boundary of the drivable area to obtain the instance segmentation confidence score. Specifically, the instance segmentation confidence score is constructed based on features such as the connectivity and boundary clarity of the segmentation mask. The calculation methods include:

[0094] S221, extract the mask coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ;

[0095] Specifically, the BEV Transformer model outputs mask coefficients for each detected object. These coefficients represent the segmentation confidence of the object and are typically values ​​between -1 and 1. In the BEV Transformer, these coefficients are output through the model's segmentation branch.

[0096] Number of connected regions This indicates the segmentation of connectivity. If the segmentation result of a vehicle is divided into multiple unconnected regions, the system may mistakenly identify them as multiple vehicles, leading to planning errors.

[0097] variance of boundary gradient This is used to assess the clarity of boundaries. If the boundaries of pedestrians or vehicles are unclear, the system may not be able to accurately determine the size and position of the object, affecting obstacle avoidance decisions.

[0098] S222, Based on the number of connected regions corresponding to the instance segmentation mask. Calculate connectivity score The mathematical expression for the connectivity is:

[0099]

[0100] Example 1: Ideal segmentation case

[0101] Original segmentation mask: a complete car outline.

[0102] Connectivity analysis result: 1 connected region.

[0103] Connectivity score = 1 / 1 = 1.0.

[0104] Example 2: Segmentation error

[0105] Original segmentation mask: The car is incorrectly segmented into three unconnected regions.

[0106] Connectivity analysis results: 3 connected regions.

[0107] Connectivity score = 1 / 3 ≈ 0.33.

[0108] S223, based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is... The mathematical expression is:

[0109]

[0110] For example:

[0111] Example 1: Clear boundary case

[0112] Original segmentation mask: The car outline has smooth and sharp boundaries.

[0113] Boundary gradient calculation: The gradient magnitude varies greatly and the variance is high on the boundary.

[0114] Gradient variance = 15.2.

[0115] Boundary clarity score = 1 - 1 / (1 + 15.2) ≈ 0.94.

[0116] Example 2: Fuzzy boundary cases

[0117] Original segmentation mask: The outline of the car is blurred and discontinuous.

[0118] Boundary gradient calculation: The gradient magnitude changes little and the variance is low on the boundary.

[0119] Gradient variance = 2.3.

[0120] Boundary clarity score = 1 - 1 / (1 + 2.3) ≈ 0.70.

[0121] S224, based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is:

[0122]

[0123] In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

[0124] Finally, the dimensionless normalized values ​​are weighted and the confidence levels are divided. .

[0125] S230, evaluate the temporal stability of the dynamic obstacle trajectory and the static scene elements to obtain the temporal consistency confidence level. ;

[0126] In this application, for each detected object, the differences in its position, velocity, orientation, and other attributes between consecutive frames are calculated, and the stability of the perception results is measured based on these differences, specifically, the temporal consistency confidence level. The calculation methods include:

[0127] S231, Obtain the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number;

[0128] S232, based on Kalman filtering and the motion parameters of all objects in the previous frame image. Predicted motion parameters of all objects in the next frame image ;

[0129] S233, calculate the motion parameters for each frame of the image. With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image;

[0130] S234, calculate the average rate of change of motion for each object to obtain the temporal consistency confidence level. .

[0131] In this application, Kalman filtering is used to predict the possible position of the object in the next frame, and the error between the actual detected position and the predicted position is calculated. The smaller the error, the higher the temporal consistency.

[0132] For ease of calculation, the rates of change calculated in this application are all dimensionless parameters. For example, when the motion parameter is velocity, the rate of change is:

[0133]

[0134] When the motion parameter is position, the rate of change is:

[0135]

[0136] In the formula, This represents the distance between the predicted location and the actual location. Typical distance;

[0137] When the motion parameter is in direction, the rate of change is:

[0138]

[0139] in, This indicates the angle between the predicted direction and the actual direction. Indicates a typical included angle.

[0140] S240, performs a consistency assessment on the structured perception results from the camera and the structured perception results from the LiDAR, and obtains the fusion confidence score. ;

[0141] This application calculates the fusion confidence level by comparing the consistency of perception results of the same object from different sensors (such as cameras and radar). The specific process includes:

[0142] S241, Extract the first position from the camera from the structured perception result. and the second position from the lidar ;

[0143] S242, Calculate the first position of the same object. With the second position European distance ;

[0144] S243, Calculate the Euclidean distance between multiple objects. average distance ;

[0145] S244, regarding the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is:

[0146]

[0147] In the formula, This is the maximum reference distance set.

[0148] Specifically, in the above process, the perception results of different sensors are projected onto the same coordinate system (such as the BEV coordinate system); the distance between the detection results of each sensor for the same object is calculated.

[0149] For example: If the camera detects a vehicle at position (10, 5) and the radar detects the same vehicle at position (9.8, 5.2), then the distance = (10 - 9.8). 2 +(5-5.2) 2 =0.28m; if the maximum allowable difference is 0.5m, then the consistency score = 1 - 0.28 / 0.5 = 0.44.

[0150] S250, based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence Constructing confidence of perception results The confidence level of the perception result The mathematical expression is:

[0151]

[0152] In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight.

[0153] Finally, the confidence scores of the first type of perception results are weighted to obtain the comprehensive confidence score. The threshold for the comprehensive confidence score can be dynamically adjusted according to the environmental complexity, for example:

[0154] Sunny weather: Threshold set to 0.85.

[0155] Rainy / foggy weather: The threshold is automatically reduced to 0.75.

[0156] Low light at night: The threshold was further reduced to 0.7.

[0157] To avoid frequent mode switching caused by sudden threshold changes, a sliding window average is used to calculate the current environmental conditions and smoothly adjust the threshold.

[0158] The bimodal planner monitors the confidence score of the first sensing pathway in real time.

[0159] If the confidence level is higher than the threshold, it is in normal mode, and the planner uses the first type of perception results to perform high-precision and comfortable trajectory planning.

[0160] If the confidence level falls below a threshold, it indicates that BEV perception may have failed, and the system immediately switches to safety mode. In this mode, the planner ignores the first type of perception results and instead performs planning based on the second type of perception results (occupied grid). The planning objective simplifies to: finding a collision-free path from the current position to the target position within the geometric space described by the occupied grid. The specific process includes:

[0161] (1) Extract target voxels from the three-dimensional occupied grid whose occupied probability is higher than a preset probability threshold;

[0162] (2) Construct a cost function, wherein the cost function includes path smoothing cost, velocity cost, obstacle cost and direction cost, and the obstacle cost is determined based on the nearest distance between the path node and the target voxel;

[0163] The mathematical expression for the cost function is:

[0164]

[0165]

[0166]

[0167]

[0168]

[0169] In the formula, , , and All are weighting coefficients. Represents the cost of smoothness. This indicates a typical value representing the degree of path curvature. Indicates the cost of speed. Indicates the cost of obstacles. Indicates the cost of direction. This represents the position vector of the path in space. For arc length parameters, For vehicle speed, To limit the speed, This represents the typical speed value. Representing path points The closest distance to the target voxel. Represents the path dimension element. The path direction vector. The target direction vector.

[0170] (3) Based on the cost function and the three-dimensional occupied grid, execute the RRT* fast random tree algorithm to obtain the second planning path.

[0171] This application utilizes the RRT* fast random tree traversal algorithm for path planning, and the core of the RRT* fast random tree traversal algorithm is the cost function. In this application, the path smoothing cost, speed cost, obstacle cost, and direction cost of the driving path are considered to plan a safe path. Among these, the obstacle cost is given primary emphasis, therefore the weight parameters... A higher value can be selected, such as 0.8.

[0172] The main features and beneficial effects of the above content include:

[0173] Redundant architecture, safety first: For the first time, a backup input path independent of the main perception model was built at the planning level, fundamentally solving the problem of single point of failure in perception.

[0174] Adaptive mode: Automatic and seamless mode switching is achieved through confidence monitoring, while taking into account both high performance under normal conditions and high security under abnormal conditions.

[0175] Decoupled design: The design goal of the second sensing path (occupation grid) is more singular (geometric occupancy), which can be made lighter and more robust, complementing the first path.

[0176] Improved System Rating: This architecture meets the requirements for redundancy and safety architecture in automotive industry functional safety standards (such as ISO 26262), which helps to improve the overall safety level of autonomous driving systems.

[0177] like Figure 2 As shown, this application also provides a robust planning method for autonomous driving based on dual-path perception, including the following steps:

[0178] S210, acquire image sequences captured by multiple cameras of the vehicle and point cloud data captured by LiDAR, wherein the image sequence includes multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point;

[0179] S220, the image sequence and the point cloud data are input into the pre-built BEV Transformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries and semantic categories;

[0180] S230, the image sequence and the point cloud data are input into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels;

[0181] S240, the confidence level of the structured perception result is evaluated to obtain the confidence level of the perception result. When the confidence level of the perception result is greater than or equal to a preset confidence level threshold, the structured perception result is output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level of the perception result is less than the preset confidence level threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

[0182] This application presents a robust planning system and method for autonomous driving based on dual-path perception. At the planning level, this application constructs a backup input path independent of the primary perception model, fundamentally solving the problem of single-point perception failure. Automatic and seamless mode switching is achieved through confidence monitoring, while simultaneously ensuring high performance under normal conditions and high safety under abnormal conditions. The design goal of the second perception path (occupancy grid) is more singular (geometric occupancy), allowing it to be made lighter and more robust, complementing the first path. The architecture in this application complies with the requirements for redundancy and safety architecture in automotive industry functional safety standards (such as ISO 26262), contributing to improving the overall safety level of the autonomous driving system.

[0183] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0184] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the terminal to perform any of the methods in this embodiment.

[0185] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0186] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0187] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0188] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0189] In the above embodiments, although the present application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. The embodiments of the present application are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0190] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A robust planning system for autonomous driving based on dual-path perception, characterized in that, include: The acquisition module is used to acquire image sequences captured by multiple cameras of the vehicle and point cloud data acquired by LiDAR. The image sequence includes multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points before the current time point. The first perception pathway module is used to input the image sequence and the point cloud data into a pre-built BEVTransformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries and semantic categories. The second perception pathway module is used to input the image sequence and the point cloud data into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels. The bimodal planning module is used to evaluate the confidence level of the structured perception results to obtain the confidence level of the perception results. When the confidence level of the perception results is greater than or equal to a preset confidence level threshold, the structured perception results are output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level of the perception results is less than the preset confidence level threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

2. The robust planning system for autonomous driving based on dual-path perception according to claim 1, characterized in that, The confidence level of the structured perception results is evaluated to obtain the confidence level of the perception results, including: Based on the semantic categories, a classification deterministic assessment is performed to obtain the semantic recognition confidence level. ; The boundary determinism of the dynamic obstacle trajectory and the drivable area boundary is evaluated to obtain the instance segmentation confidence. ; The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. ; A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. ; Based on the semantic recognition confidence level The instance segmentation confidence level The aforementioned time-series consistency confidence level and the fusion confidence level Constructing confidence scores for perception results The confidence level of the perception result The mathematical expression is: In the formula, As the first weight, As the second weight, As the third weight, It is the fourth weight.

3. The robust planning system for autonomous driving based on dual-path perception according to claim 2, characterized in that, Based on the semantic categories, a classification deterministic assessment is performed to obtain the semantic recognition confidence level. ,include: For each object in the semantic category, extract the predicted probabilities of all predicted categories for each object. ,in, Indicates the first The predicted probabilities of each category; Predicted probability based on each prediction category Calculate the entropy value of each object. The entropy value The mathematical expression is: In the formula, Indicates the number of predicted categories; The semantic recognition confidence score is obtained by taking the maximum value from the entropy values ​​of multiple objects. .

4. The robust planning system for autonomous driving based on dual-path perception according to claim 2, characterized in that, The boundary determinism of the dynamic obstacle trajectory and the drivable area boundary is evaluated to obtain the instance segmentation confidence. ,include: Extract the masking coefficients of the dynamic obstacle trajectory and the drivable area boundary. Number of connected components corresponding to the instance segmentation mask variance of boundary gradient ; Based on the number of connected components corresponding to the instance segmentation mask Calculate connectivity score The mathematical expression for the connectivity is: Based on the variance of the boundary gradient Calculate the boundary sharpness score The boundary sharpness score is... The mathematical expression is: Based on the mask coefficients The connectivity score and the boundary sharpness score Calculate instance segment confidence Among them, instance segmentation confidence The mathematical expression is: In the formula, As the fifth weight, It is the sixth weight. It is the seventh weight.

5. The robust planning system for autonomous driving based on dual-path perception according to claim 2, characterized in that, The temporal stability of the dynamic obstacle trajectory and the static scene elements is evaluated to obtain the temporal consistency confidence level. ,include: Get the motion parameters of all objects in the previous frame image. The motion parameters include position, velocity, and direction. Indicates the object's serial number; Based on Kalman filtering and the motion parameters of all objects in the previous frame image Predicted motion parameters of all objects in the next frame image ; Calculate motion parameters in each frame of the image With predicted motion parameters The dimensionless difference is used to obtain the rate of motion change of each object in each frame of the image; Calculate the average rate of change of motion for each object to obtain the temporal consistency confidence score. .

6. The robust planning system for autonomous driving based on dual-path perception according to claim 2, characterized in that, A consistency assessment is performed on the structured perception results from the camera and the structured perception results from the lidar to obtain the fusion confidence score. ,include: Extract the first position from the camera from the structured perception results. and the second position from the lidar ; Calculate the first position of the same object With the second position European distance ; Calculate the Euclidean distance between multiple objects average distance ; For the average distance Normalization is performed to obtain the fusion confidence score. The mathematical expression for the fusion confidence level is: In the formula, This is the maximum reference distance set.

7. The robust planning system for autonomous driving based on dual-path perception according to claim 1, characterized in that, Based on the aforementioned three-dimensional occupancy grid, path planning is performed to obtain a second planned path, including: Extract target voxels whose occupied probability is higher than a preset probability threshold from the three-dimensional occupied grid; Construct a cost function, wherein the cost function includes path smoothing cost, velocity cost, obstacle cost and orientation cost, and the obstacle cost is determined based on the nearest distance between the path node and the target voxel; Based on the cost function and the three-dimensional occupied grid, the RRT* fast random tree algorithm is executed to obtain the second planned path.

8. The robust planning system for autonomous driving based on dual-path perception according to claim 7, characterized in that, The mathematical expression for the cost function is: In the formula, , , and All are weighting coefficients. Represents the cost of smoothness. This indicates a typical value representing the degree of path curvature. Indicates the cost of speed. Indicates the cost of obstacles. Indicates the cost of direction. This represents the position vector of the path in space. For arc length parameters, For vehicle speed, To limit the speed, This represents the typical speed value. Representing path points The closest distance to the target voxel. Represents the path element. The path direction vector. The target direction vector.

9. A robust planning method for autonomous driving based on dual-path perception, characterized in that, Including the following steps: The system acquires image sequences captured by multiple cameras of the vehicle and point cloud data collected by LiDAR, wherein the image sequences include multiple frames of images captured by multiple cameras in the current time period, and the current time period includes images from multiple time points prior to the current time point; The image sequence and the point cloud data are input into a pre-built BEV Transformer model to obtain structured perception results, wherein the structured perception results include dynamic obstacle trajectories, static scene elements, drivable area boundaries, and semantic categories. The image sequence and the point cloud data are input into a pre-constructed occupancy grid generation model to obtain a three-dimensional occupancy grid, wherein the three-dimensional occupancy grid includes the occupancy probability of multiple voxels; The structured perception results are evaluated for confidence to obtain a confidence level. When the confidence level is greater than or equal to a preset confidence threshold, the structured perception results are output to the spatiotemporal joint planner to obtain a first planning path. When the confidence level is less than the preset confidence threshold, path planning is performed based on the three-dimensional occupancy grid to obtain a second planning path.

Images