A multi-element feature fusion automatic driving course reinforcement learning training method

Abstract

The application discloses a kind of multi-element feature fusion's automatic driving course reinforcement learning training method, including global path planning according to training task, traffic state information vector is obtained by using vehicle-mounted sensor to collect vehicle surrounding environment information, and local occupancy grid map of the current position of vehicle is obtained based on Flainer coordinate system;Design the action space and reward function of behavior decision reinforcement learning, build deep reinforcement learning backbone network model;In the training process of automatic driving, according to the difficulty of subtask, the extracted multi-element feature is input into the constructed deep reinforcement learning backbone network model, and the discrete optimal driving behavior is output;According to optimal driving behavior, the current optimal trajectory of unmanned vehicle is generated in real time by multi-resolution trajectory planning algorithm based on sampling.The application makes the automatic driving car spontaneously learn optimal driving strategy in a data-driven manner, and completes the navigation task in the urban structured road environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multi-feature fusion method for reinforcement learning training of autonomous driving courses. Background Technology

[0002] Autonomous vehicles play a vital role in ensuring traffic safety, reducing transportation costs, improving vehicle efficiency, and reducing air pollution, and have a wide range of applications. Behavioral decision-making, as the "brain" of an autonomous vehicle, connects the environmental perception system above and guides planning and control below, playing a crucial role in improving the autonomy, intelligence, and safety of autonomous vehicles. Therefore, research on behavioral decision-making methods for autonomous vehicles is of great significance to promoting their development. The behavioral decision-making layer is the strategic formulation layer for autonomous vehicle driving behavior. Its main function is to use intelligent decision-making algorithms to output intelligent, reasonable, and legal advanced driving behaviors based on local environmental information expressed by the perception module as input, including vehicle information and local traffic conditions.

[0003] The shortcomings of existing technologies lie in the fact that current conventional methods are rule-based behavioral decision-making approaches. These methods analyze the environmental conditions of different scenarios and road conditions, building a knowledge base of behavioral decision-making rules based on traffic laws and regulations and the prior driving knowledge of experienced drivers. While rule-based decision-making methods have a clear structure and are easy to implement, as reactive methods, they also suffer from poor flexibility, poor scalability, poor maintainability, the existence of suboptimal solutions, and low intelligence. On the one hand, to cover as many scenarios as possible, finite state machine models require an unlimited number of rules. However, expanding such decision graphs is a labor-intensive project. Therefore, rule-based behavioral decision-making models require a significant investment of time and effort to transform human prior experience into rules to cope with the most complex traffic and driving scenarios. On the other hand, as the rule base expands, the coupling of various rules becomes more complex, making rule maintenance and updates more difficult. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology. To achieve the above objective, a multi-feature fusion-based autonomous driving course reinforcement learning training method is adopted to solve the problems mentioned in the background technology.

[0005] A multi-feature fusion reinforcement learning training method for autonomous driving courses, comprising the following steps:

[0006] Step S1: Perform global path planning based on the training task, use on-board sensors to collect traffic state information vectors from the surrounding environment of the vehicle, and obtain the local occupancy grid map of the vehicle's current position based on the Flyner coordinate system.

[0007] Step S2: Design the action space and reward function for behavioral decision reinforcement learning, and build a deep reinforcement learning backbone network model;

[0008] Step S3: During the training process of autonomous driving, the sub-tasks are graded according to their difficulty. The extracted multi-dimensional features are input into the deep reinforcement learning backbone network model, and the discrete optimal driving behavior is output.

[0009] Step S4: Based on the obtained optimal driving behavior, generate the current optimal trajectory of the unmanned vehicle in real time using a sampling-based multi-resolution trajectory planning algorithm.

[0010] As a further technical solution of the present invention, the specific steps in step S1 include:

[0011] Define the road scenario and training task, and use the Astar-based global planning algorithm to generate a lane-level global path as the global reference path for training.

[0012] By acquiring multi-sensor data collected by vehicle-mounted equipment and processing the multi-sensor data, a traffic state information vector can be obtained. The traffic state information vector consists of a vehicle state vector and a traffic environment element vector.

[0013] The vehicle state vector is represented as [v, γ], where v is the normalized speed and γ is the collision risk in the current lane;

[0014] The formula for calculating the normalized velocity v is: v = v ego / v max ;

[0015] Among them, v ego v is the speed of the vehicle. max Set the current lane's maximum permissible speed or the preset maximum speed;

[0016] Then the safe distance d between the vehicle and the vehicle in front is obtained. s The calculation formula is:

[0017]

[0018] Where v1 is the speed of the vehicle in front, a ego a1 and a2 are the decelerations of the vehicle and the vehicle in front, respectively, and t represents the reaction time of the vehicle.

[0019] The formula for calculating the collision risk γ within the current lane is: γ = d / d s γ max ;

[0020] Where d is the actual distance between the vehicle and the vehicle in front, γ max This is the maximum ratio of the actual distance to the safe distance.

[0021] The vector of traffic environment elements can be represented as [e l ,e c ,e r ,l l ,l c ,l r [,r,g,b,δ1,δ2];

[0022] Among them, [e l ,e c ,e r [l] indicates whether the left, middle, and right lanes exist, respectively. l ,l c ,l r [r, g, b] represent whether lane changing is permitted under traffic regulations in the left, middle, and right lanes, respectively; [r, g, b] represent the status of the traffic lights; and [δ1, δ2] represent the normalized values ​​of the distance to the next intersection and the normalized values ​​of the distance to the global path endpoint, respectively.

[0023] A local occupancy raster is constructed based on the Flyner coordinate system, and any position in the local occupancy raster is regarded as a linear combination of the horizontal and vertical vectors.

[0024] As a further technical solution of the present invention, the specific steps in step S2 include:

[0025] Design action space: Design actions that can complete all target tasks, while compressing the dimensions of the solution space to obtain five-dimensional driving behavior actions A = {left lane change, right lane change, acceleration, deceleration, lane keeping}.

[0026] Design the reward function: The reward function mainly includes sparse rewards for the main line and dense rewards for trajectory planning feedback;

[0027] The mainline sparse reward R1(s) is the reward obtained by the autonomous vehicle under special conditions, which mainly includes the round termination state reward r1(s) and the navigation task progress reward r2(s), and their calculation formulas are as follows:

[0028] R1(s) = r1(s) + r2(s)

[0029]

[0030]

[0031] In the formula, k1 is a hyperparameter, and v is the current speed of the autonomous vehicle;

[0032] The trajectory planning feedback dense reward jointly optimizes trajectory planning and behavioral decision-making, using the expected value of the trajectory planning result cost within the behavioral decision-making cycle as the input to the CR mapping. Its calculation formula is as follows:

[0033]

[0034] R2 = -k2(Cb);

[0035] In the formula, N represents the number of trajectory planning operations completed within the current behavior decision-making cycle, and c n The cost of the nth trajectory planning result within the current behavior decision cycle, where k2 and b are positive hyperparameters;

[0036] A deep reinforcement learning backbone network model is built based on the Soft Actor-Critic algorithm, which includes one Actor network and four Critic networks.

[0037] As a further technical solution of the present invention, the specific steps in step S3 include:

[0038] Based on the principle of difficulty level classification of course tasks, the number of sub-tasks of different difficulty levels is classified. Sub-tasks include global route length, route curves, traffic light conditions, other traffic vehicles, and interaction with dynamic vehicles.

[0039] During the course reinforcement learning and training process, the decision to reschedule the course is based on the completion status of the course tasks in the most recent 20 rounds.

[0040] If the success rate of the course task reaches 90% or more, and the normalized root mean square NRMSD is less than the threshold of 0.1, then the current course learning task is completed and the training and learning of the next course task begins.

[0041] The formula for calculating the normalized root mean square NRMSD is as follows:

[0042]

[0043] In the formula, r' represents the expected return, r i This represents the reward for round i, where N is the total number of rounds.

[0044] As a further technical solution of the present invention, the specific steps in step S4 include:

[0045] Based on the current vehicle speed, a suitable preview point is selected in the global reference path, and a series of terminal states are generated by sampling to both sides along the lateral direction of the global reference path at the preview point position.

[0046] A family of curves with continuous curvature is generated from the vehicle's current position to the sampled terminal state position using third-order Bézier curves, serving as a candidate trajectory family for the autonomous vehicle's current position. The third-order Bézier curve is defined as follows:

[0047] C(t)=B0(t)H0+B1(t)H1+B2(t)H2+B3(t)H3;

[0048] Among them, H i B is the control point of the third-order Bézier curve. i The coefficients for the corresponding control points are i = 1, 2, 3, and the range of parameter t is (0, 1).

[0049] B i The expression for (t) is:

[0050] B0 = (1-t) 3

[0051] B1 = 3(1-t) 2 t

[0052] B2=3(1-t)t 2

[0053] B3 = t 3

[0054] Control point H i The calculation formula is:

[0055]

[0056] Among them, S x and S y ω represents the lateral and longitudinal offsets of the terminal state relative to the vehicle itself, ω is the relative angle between the terminal state direction and the X-axis, and l is the distance between control points H0 and H1, and also the distance between H2 and H3.

[0057] Based on the obtained trajectory clusters, a multi-attribute evaluation function is established to find the trajectory with the minimum cost, which is then taken as the current optimal trajectory. The formula for calculating the optimal trajectory is as follows:

[0058]

[0059] Where N represents the number of trajectories in the trajectory set, C j L C j S C j D and C j K λ represents trajectory length, safety assessment, lateral offset, and curvature smoothing cost, respectively. L , λ L , λ D and λ K These are the weights of the j-th candidate trajectory in terms of trajectory length, safety assessment, lateral offset, and curvature smoothing cost;

[0060] The calculation formulas for each evaluation indicator are as follows:

[0061]

[0062]

[0063]

[0064]

[0065] Among them, L j Let L be the length of the j-th candidate trajectory. max L min d represents the maximum and minimum lengths of the candidate trajectories in the trajectory set, respectively. max d min Let d represent the maximum and minimum distances between the trajectory and the obstacle in the trajectory set, respectively. j Let be the distance between the j-th candidate trajectory and the obstacle. Let m be the lateral offset between the i-th trajectory point in the j-th candidate trajectory and the global reference path, and let m be the number of trajectory points in the j-th candidate trajectory. j max Let be the maximum lateral offset between the j-th candidate trajectory and the global reference path. Let K be the curvature of the position of the i-th trajectory point in the j-th candidate trajectory. j max Let be the maximum curvature in the j-th candidate trajectory.

[0066] Compared with the prior art, the present invention has the following technical advantages:

[0067] The above technical solution collects sensor data through various vehicle-mounted devices and combines it with the urban structural environment to obtain a local occupancy grid map and traffic state information vectors as the state space observation input for a deep reinforcement learning network. Simultaneously, a five-dimensional discrete driving action representation action space is designed. From a decision-making and planning perspective, trajectory planning cost is used as an indicator to guide behavioral decisions, ultimately achieving the goal of joint optimization of behavioral decisions and trajectory planning. Furthermore, based on the difficulty level of driving tasks and task scheduling methods, the difficulty of training scenarios is gradually increased to achieve the training objective. This avoids the need for existing technologies to consume a large amount of time and effort to cope with complex traffic and driving scenarios. It also solves the problem of complex coupling of various rules as the rule base expands, making rule maintenance and iteration much simpler. Attached Figure Description

[0068] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0069] Figure 1This is a schematic diagram illustrating the steps of a training method according to some embodiments disclosed in this application;

[0070] Figure 2 This is a flowchart illustrating the training methods of some embodiments disclosed in this application;

[0071] Figure 3 This is a schematic diagram of a single-lane collision model of some embodiments disclosed in this application;

[0072] Figure 4 This is a schematic diagram of the Fleischer coordinate system transformation for some embodiments disclosed in this application;

[0073] Figure 5 Here are some neural network structure diagrams of embodiments disclosed in this application;

[0074] Figure 6 This is a schematic diagram illustrating the trajectory generation of a third-order Bézier curve according to some embodiments disclosed in this application;

[0075] Figure 7 This is a schematic diagram of a multi-resolution trajectory planning algorithm for some embodiments disclosed in this application. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] Please refer to Figure 1 and Figure 2 In this embodiment of the invention, a multi-feature fusion-based reinforcement learning training method for autonomous driving courses includes the following steps:

[0078] Step S1: Acquire multi-sensor data collected by the vehicle-mounted equipment, then obtain a local occupancy grid map based on the Flyner coordinate system, and extract traffic state information vectors around the vehicle based on the urban structured environment. Specific steps include:

[0079] In this embodiment, the data consists of a high-precision map, a reference trajectory, and information from various sensors of the unmanned vehicle.

[0080] like Figure 3 As shown, the diagram illustrates a collision model within a single lane.

[0081] Define the road scenario and training task, and use the Astar-based global planning algorithm to generate a lane-level global path as the global reference path for training.

[0082] By acquiring multi-sensor data collected by vehicle-mounted equipment and processing the multi-sensor data, a traffic state information vector can be obtained, which consists of a vehicle state vector and a traffic environment element vector.

[0083] The vehicle state vector is represented as [v, γ], where v is the normalized speed and γ is the collision risk in the current lane;

[0084] The formula for calculating the normalized velocity v is: v = v ego / v max ;

[0085] Among them, v ego v is the speed of the vehicle. max Set the current lane's maximum permissible speed or the preset maximum speed;

[0086] Then we obtain the formula for the safe distance d between the vehicle and the vehicle in front. s The calculation formula is:

[0087]

[0088] Where v1 is the speed of the vehicle in front, a ego a1 and a2 are the decelerations of the vehicle and the vehicle in front, respectively, and t represents the reaction time of the vehicle.

[0089] The formula for calculating the collision risk γ within the current lane is: γ = d / d s γ max ;

[0090] Where d is the actual distance between the vehicle and the vehicle in front, γ max This is the maximum ratio of the actual distance to the safe distance.

[0091] The resulting vector representation of traffic environment elements is [e l ,e c ,e r ,l l ,l c ,l r [,r,g,b,δ1,δ2];

[0092] Among them, [e l ,e c ,e r [l] indicates whether the left, middle, and right lanes exist, respectively. l ,l c ,l r [r, g, b] represent whether lane changing is permitted under traffic regulations in the left, middle, and right lanes, respectively; [r, g, b] represent the status of the traffic lights; and [δ1, δ2] represent the normalized values ​​of the distance to the next intersection and the normalized values ​​of the distance to the global path endpoint, respectively.

[0093] Specifically, the [r,g,b] vector is obtained through image processing from the autonomous vehicle's camera sensors. The remaining vector information is inferred from the global reference path, high-precision road network topology map, and the autonomous vehicle's own real-time positioning information.

[0094] like Figure 4 As shown, the diagram illustrates the transformation of the Fleiner coordinate system.

[0095] A local occupancy raster is constructed based on the Flyner coordinate system, and any position in the local occupancy raster is regarded as a linear combination of the horizontal and vertical vectors.

[0096] In a specific implementation, a 50*7 obstacle occupies a grid map representing a range of 72 meters longitudinally in front of the vehicle, 18 meters longitudinally behind it, and 12 meters laterally. The target's Freina coordinates are then determined. We can use the normal vector n r tangent vector t r The solution process involves projecting the target vehicle onto the reference trajectory of the self-vehicle, obtaining the offset distance d and the projection point on the S-axis. Solve the objective It can be expressed as follows:

[0097]

[0098] Step S2: Design the action space and reward function for behavioral decision-making reinforcement learning, and build a deep reinforcement learning backbone network model. Specific steps include:

[0099] like Figure 5 As shown, the diagram illustrates the structure of a neural network.

[0100] In this embodiment, based on the three principles of completeness, efficiency and legality, the action design is ensured to be capable of completing all target tasks, while compressing the dimension of the solution space, efficiently and legally completing the decision-making task, and designing five-dimensional driving behavior actions.

[0101] Design action space: Design actions that can complete all target tasks, while compressing the dimensions of the solution space to obtain five-dimensional driving behavior actions A = {left lane change, right lane change, acceleration, deceleration, lane keeping}.

[0102] Design the reward function: The reward function mainly includes sparse rewards for the main line and dense rewards for trajectory planning feedback;

[0103] The mainline sparse reward R1(s) is the reward obtained by the autonomous vehicle under special conditions, which mainly includes the round termination state reward r1(s) and the navigation task progress reward r2(s), and their calculation formulas are as follows:

[0104] R1(s) = r1(s) + r2(s)

[0105]

[0106]

[0107] In the formula, k1 is a hyperparameter, and v is the current speed of the autonomous vehicle;

[0108] The trajectory planning feedback dense reward R² jointly optimizes trajectory planning and behavioral decision-making. The expected value of the optimal trajectory cost for all time-limited trajectory planning steps within the behavioral decision-making cycle is used as the input to the CR mapping, and its calculation formula is as follows:

[0109]

[0110] R2 = -k2(Cb);

[0111] In the formula, N represents the number of trajectory planning operations completed within the current behavior decision-making cycle, and c n The cost of the nth trajectory planning result within the current behavior decision cycle, where k2 and b are positive hyperparameters;

[0112] A deep reinforcement learning backbone network model is built based on the Soft Actor-Critic algorithm, which includes one Actor network and four Critic networks.

[0113] In a specific implementation, the input features include a traffic state information vector and a local occupancy grid map. In the Actor network, the obstacle occupancy grid map first passes through three convolutional neural network layers and fully connected layers to form a one-dimensional 256 intermediate result. This result is then combined with the one-dimensional 256 intermediate result formed by the traffic information vector through three fully connected layers, and finally fitted through three fully connected layers to output the probabilities of five discrete actions. The Critic network has the same network structure as the Actor network, but the Critic network needs to fit the Q-value of the current state, i.e., the value of the action value function. Therefore, the last layer does not need to set an activation function.

[0114] Step S3: During the training process of autonomous driving, the sub-tasks are graded according to their difficulty. The extracted multi-dimensional features are input into the constructed deep reinforcement learning backbone network model, and the output is the optimal driving behavior with high-level discrete characteristics. The specific steps include:

[0115] In this embodiment, a method for classifying the difficulty of autonomous vehicle tasks and scheduling tasks is proposed based on factors such as global route length, intersection curves, static obstacles, and dynamic traffic participants.

[0116] Based on the principle of difficulty level classification of course tasks, the number of sub-tasks of different difficulty levels is classified. Sub-tasks include global route length, route curves, traffic light conditions, other traffic vehicles, and interaction with dynamic vehicles.

[0117] Specifically, interacting with dynamic vehicles is the most difficult sub-task, followed by traffic lights and road conditions around curves, while the overall route length is considered the easiest sub-task. Combinations of lower-difficulty sub-tasks in increasing numbers constitute higher-difficulty tasks.

[0118] During the course reinforcement learning and training process, the decision to reschedule the course is based on the completion status of the course tasks in the most recent 20 rounds.

[0119] If the success rate of the course task reaches 90% or more, and the normalized root mean square NRMSD is less than the threshold of 0.1, then the current course learning task is completed and the training and learning of the next course task begins.

[0120] The formula for calculating the normalized root mean square NRMSD is as follows:

[0121]

[0122] In the formula, r' represents the expected return, ri represents the return in the i-th round, and N is the total number of rounds.

[0123] Step S4: Based on the obtained optimal driving behavior, the current optimal trajectory of the unmanned vehicle is generated in real time using a multi-resolution trajectory planning algorithm based on sampling. Specific steps include:

[0124] Based on the current vehicle speed, a suitable preview point is selected in the global reference path, and a series of terminal states are generated by sampling to both sides along the lateral direction of the global reference path at the preview point position.

[0125] like Figure 6 As shown in the figure, the diagram illustrates the trajectory generation of a third-order Bézier curve.

[0126] A family of curves with continuous curvature is generated from the vehicle's current position to the sampled terminal state position using third-order Bézier curves, serving as a candidate trajectory family for the autonomous vehicle's current position. The third-order Bézier curve is defined as follows:

[0127] C(t)=B0(t)H0+B1(t)H1+R2(t)H2+B3(t)H3

[0128] Among them, H i B is the control point of the third-order Bézier curve. i The coefficients for the corresponding control points are i = 1, 2, 3, and the range of parameter t is (0, 1).

[0129] B iThe expression for (t) is:

[0130] B0 = (1-t) 3

[0131] B1 = 3(1-t) 2 t

[0132] B2=3(1-t)t 2

[0133] B3 = t 3

[0134] Control point H i The calculation formula is:

[0135]

[0136] Among them, S x and S y ω represents the lateral and longitudinal offsets of the terminal state relative to the vehicle itself, ω is the relative angle between the terminal state direction and the X-axis, and l is the distance between control points H0 and H1, and also the distance between H2 and H3.

[0137] like Figure 7 As shown in the figure, this is a schematic diagram of a multi-resolution trajectory planning algorithm;

[0138] Based on the obtained trajectory clusters, a multi-attribute evaluation function is established to find the trajectory with the minimum cost, which is then taken as the current optimal trajectory. The formula for calculating the optimal trajectory is as follows:

[0139]

[0140] Where N represents the number of trajectories in the trajectory set, C j L C j S C j D and C j K λ represents trajectory length, safety assessment, lateral offset, and curvature smoothing cost, respectively. L , λ L , λ D and λ K These are the weights of the j-th candidate trajectory in terms of trajectory length, safety assessment, lateral offset, and curvature smoothing cost;

[0141] The calculation formulas for each evaluation indicator are as follows:

[0142]

[0143]

[0144]

[0145]

[0146] Among them, L j Let L be the length of the j-th candidate trajectory. max L min d represents the maximum and minimum lengths of the candidate trajectories in the trajectory set, respectively. max d min Let d represent the maximum and minimum distances between the trajectory and the obstacle in the trajectory set, respectively. j Let be the distance between the j-th candidate trajectory and the obstacle. Let m be the lateral offset between the i-th trajectory point in the j-th candidate trajectory and the global reference path, and let m be the number of trajectory points in the j-th candidate trajectory. j max Let be the maximum lateral offset between the j-th candidate trajectory and the global reference path. Let K be the curvature of the position of the i-th trajectory point in the j-th candidate trajectory. j max Let be the maximum curvature in the j-th candidate trajectory.

[0147] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. A multi-feature fusion method for reinforcement learning training of autonomous driving courses, characterized in that, The specific steps include: Step S1: Perform global path planning based on the training task. Use onboard sensors to collect traffic state information vectors from the vehicle's surrounding environment and obtain a local occupancy grid map of the vehicle's current position based on the Flyner coordinate system. The specific steps include: Define the road scenario and training task, and use the Astar-based global planning algorithm to generate a lane-level global path as the global reference path for training. By acquiring multi-sensor data collected by vehicle-mounted equipment and processing the multi-sensor data, a traffic state information vector can be obtained. The traffic state information vector consists of a vehicle state vector and a traffic environment element vector. The vehicle state vector is represented as [ v, γ], where v Here, γ represents the normalized speed, and γ represents the collision risk within the current lane. The normalized speed v The calculation formula is: v=v ego / v max ; in, v ego For the speed of the vehicle, v max Set the current lane's maximum permissible speed or the preset maximum speed; This gives the safe distance between your vehicle and the vehicle in front. d s The calculation formula is: ； in, v 1 represents the speed of the vehicle in front. a ego and a 1 represents the deceleration of the vehicle and the vehicle in front, respectively. t Indicates the vehicle's reaction time; The formula for calculating the current lane collision risk γ is: γ =d / d s γ max ; in, d γ represents the actual distance between the vehicle and the vehicle in front. max This is the maximum ratio of the actual distance to the safe distance. The vector of traffic environment elements can be represented as [ e l ,e c ,e r ,l l ,l c ,l r ,r,g,b,δ 1 ,δ 2]; in,[ e l ,e c ,e r ] indicates whether the left, middle, and right lanes exist, respectively. l l ,l c ,l r The symbols ] indicate whether changing lanes is permitted under traffic regulations in the left, middle, and right lanes, respectively. r,g,b [ indicates the status of the traffic light] δ 1 ,δ [2] These represent the normalized values ​​of the distance to the next intersection and the normalized values ​​of the distance to the global path endpoint, respectively. A local occupancy raster is constructed based on the Flener coordinate system, and any position in the local occupancy raster is regarded as a linear combination of horizontal and vertical vectors; Step S2: Design the action space and reward function for behavioral decision-making reinforcement learning, and build a deep reinforcement learning backbone network model. The specific steps include: Design Action Space: Design actions that can complete all target tasks, while compressing the dimensions of the solution space to obtain five-dimensional driving behavior actions. A= {Left lane change, right lane change, acceleration, deceleration, lane keeping}; Design the reward function: The reward function mainly includes sparse rewards for the main line and dense rewards for trajectory planning feedback; The main line sparse returns R 1( s This refers to the rewards obtained by autonomous vehicles under special conditions, primarily including rewards for the end of a round. r 1( s ) and navigation task progress reports r 2( s The calculation formulas are as follows: In the formula, k 1 is a hyperparameter. v This represents the current speed of the driverless car. The trajectory planning feedback dense reward (CR) method jointly optimizes trajectory planning and behavioral decision-making. It uses the expected value of the trajectory planning result cost within the behavioral decision-making cycle as the input to the CR mapping, and its calculation formula is as follows: R 2= -k 2( Cb )； In the formula, N This represents the number of trajectory planning operations completed within the current behavioral decision-making cycle. c n For the first time in the current behavioral decision-making cycle n The cost of secondary trajectory planning results k 2. b It is a positive hyperparameter; A deep reinforcement learning backbone network model is built based on the Soft Actor-Critic algorithm, which includes one Actor network and four Critic networks. Step S3: During the training process of autonomous driving, the sub-tasks are graded according to their difficulty. The extracted multi-dimensional features are input into the constructed deep reinforcement learning backbone network model, which outputs discrete optimal driving behavior. The specific steps include: Based on the principle of difficulty level classification of course tasks, the number of sub-tasks of different difficulty levels is classified. Sub-tasks include global route length, route curves, traffic light conditions, other traffic vehicles, and interaction with dynamic vehicles. During the course reinforcement learning and training process, the decision to reschedule the course is based on the completion status of the course tasks in the most recent 20 rounds. If the success rate of the course task reaches 90% or more, and the normalized root mean square NRMSD is less than the threshold of 0.1, then the current course learning task is completed and the training and learning of the next course task begins. The normalized root mean square NRMSD The calculation formula is: ； In the formula, r’ Indicates expected return. r i Representing the i The reward of the round, N The total number of rounds; Step S4: Based on the obtained optimal driving behavior, generate the current optimal trajectory of the unmanned vehicle in real time using a sampling-based multi-resolution trajectory planning algorithm. The specific steps include: Based on the current vehicle speed, a suitable preview point is selected in the global reference path, and a series of terminal states are generated by sampling to both sides along the lateral direction of the global reference path at the preview point position. A family of curves with continuous curvature is generated from the vehicle's current position to the sampled terminal state position using third-order Bézier curves, serving as a candidate trajectory family for the autonomous vehicle's current position. The third-order Bézier curve is defined as follows: in, H i These are the control points of the third-order Bézier curve. B i The coefficients for the corresponding control points. i=1,2,3 ,parameter t The range of values ​​for is (0,1); B i ( t The expression for ) is: Control Points H i The calculation formula is: in, S x and S y This refers to the lateral and longitudinal offsets of the terminal state relative to the vehicle itself. ω The angle between the terminal state direction and the X-axis. l Control point H 0 and H The distance between 1 and 1 is also H 2 and H The distance between 3; Based on the obtained trajectory clusters, a multi-attribute evaluation function is established to find the trajectory with the minimum cost, which is then taken as the current optimal trajectory. The formula for calculating the optimal trajectory is as follows: ； in, N This indicates the number of trajectories in the trajectory set. C j L , C j S , C j D and C j K These represent trajectory length, safety assessment, lateral offset, and curvature smoothing cost, respectively. λ L , λ S , λ D and λ K The first j The weights of candidate trajectories in terms of trajectory length, safety assessment, lateral offset, and curvature smoothing cost; The calculation formulas for each evaluation indicator are as follows: in, L j For the first j The length of the candidate trajectories, L max , L min These represent the maximum and minimum lengths of the candidate trajectories in the trajectory set, respectively. d max , d min These represent the maximum and minimum distances between the trajectory and the obstacle in the trajectory set, respectively. d j For the first j The distance between the candidate trajectory and the obstacle. D i j For the first j The first candidate trajectory i The lateral offset between each trajectory point and the global reference path m For the first j The number of trajectory points in the candidate trajectories, D j max For the first j The maximum lateral offset between the candidate trajectory and the global reference path. K i j For the first j The first candidate trajectory i The curvature at the position of each trajectory point K j max For the first j The maximum curvature among the candidate trajectories.

Images