A commercial vehicle platoon path planning method, controller, and storage device combining deep reinforcement learning and RSS strategy

Abstract

The present invention discloses a commercial vehicle platoon path planning method, controller, and storage device that combine deep reinforcement learning and RSS strategies. The A3C framework is introduced, and a multi-threaded method is used to allow vehicles in the platoon to interact and learn with the environment in multiple threads at the same time. Each thread summarizes the learning results and stores them in Global_net. In addition, the learning results of different vehicles in the platoon are regularly retrieved from Global_net to guide future learning interactions between the platoon and the environment. At the same time, the Lattice algorithm is used to take the ST graph for speed planning, which can effectively improve the stability and comfort of the platoon's driving and ensure the smoothness of the commercial vehicle's driving trajectory. Finally, the present invention combines a safety constraint RSS strategy, and the autonomous driving car safety strategy based on mathematical formulas provides a framework for implicit rules, thereby achieving organic integration with other participants on the road, and can effectively solve the safety problem when the platoon encounters other vehicles merging.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving in artificial intelligence, and relates to a commercial vehicle platoon path planning method, controller, and storage device that combine deep reinforcement learning and an RSS (responsibility-sensitive safety) model. Background Art

[0002] Smart cars are high-tech products based on environmental perception, intelligent driving, wireless communications, and computer technologies. The transformation and upgrading of the automotive industry will be a process of gradually achieving intelligent vehicles. Intelligently operating vehicles will prioritize safety, environmental protection, energy conservation, and comfort, collaborating to build an efficient and orderly transportation network.

[0003] Currently, commercial convoys are widely used in engineering applications. Commercial vehicles are primarily categorized into five categories: buses, trucks, semi-trailer tractors, incomplete buses, and incomplete trucks. These vehicles are large, heavy, and have significant blind spots for the driver. Currently, commercial convoy path planning faces numerous challenges during training. First, the simultaneous training of multiple vehicles complicates training and can even hinder network convergence. Second, reward function design is challenging. Each convoy member has their own reward function, and the actions output by convoy members interfere with each other, leading to reward cancellation and difficulty in exploration during training. Finally, due to their large size and heavy payload, commercial vehicles are not always safe when unmanned. Therefore, finding a commercial vehicle convoy planning method that balances safety and efficiency has become a crucial topic. Summary of the Invention

[0004] To address the aforementioned commercial vehicle platooning issues, the present invention introduces the A3C framework. Using a multi-threaded approach, A3C allows vehicles in the platoon to interact and learn with the environment simultaneously in multiple threads. Each thread aggregates the learning outcomes and stores them in Global_net. Furthermore, the learning outcomes of different vehicles in the platoon are regularly retrieved from Global_net to guide future learning interactions with the environment. Simultaneously, the Lattice algorithm is used to perform speed planning using the ST graph, effectively improving the stability and comfort of the platoon's driving and ensuring the smoothness of the commercial vehicle's driving trajectory. Finally, the present invention incorporates a safety constraint strategy, the RSS (Responsibility-Sensitive Safety) strategy. This mathematically-based autonomous vehicle safety strategy provides a framework for implicit rules, enabling organic integration with other participants on the road and effectively resolving safety issues encountered when a platoon encounters other vehicles merging into it.

[0005] The present invention provides a commercial vehicle platoon planning method that combines deep reinforcement learning and RSS (Responsibility Sensitive Safety) strategy. It uses the A3C framework to improve the learning efficiency of the platoon, while improving the safety and stability of the platoon during driving through the constraints of the Lattice algorithm and RSS strategy.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The commercial vehicle platoon planning method that combines deep reinforcement learning and RSS (Responsibility Sensitive Safety) strategy includes the following steps:

[0008] Step 1: In order to better obtain information about the surrounding traffic environment, the present invention designs an effective and concise time-series bird's-eye view as the state quantity of the policy network, which greatly improves the learning of the policy network and ensures the safety of the output trajectory. The generation of the time-series bird's-eye view includes the following two steps: (1) Obtaining the surrounding environment information, including dynamic and static obstacles, and lane lines. Using the prediction module (both LSTM and GCN networks) to obtain the dynamic obstacles in the future 0~t end (2) The information obtained by the perception module and prediction is used to generate a bird's-eye view of features in three dimensions: horizontal, vertical, and time.

[0009] Step 2: Perform Frenet coordinate transformation to obtain the current state of the intelligent body from the feature bird's-eye view. In the Cartesian coordinate system, the vehicle's coordinate information (X, θ X ,k X ,v X ,a X ) can be transformed into Among them, X is the coordinate of the vehicle in the Cartesian coordinate system, which is a vector, θ X is the orientation of the vehicle in the Cartesian coordinate system, k X is the curvature, v X is the linear velocity of the vehicle in the Cartesian coordinate system, a X is the acceleration of the vehicle in the Cartesian coordinate system. s is the longitudinal displacement in the Frenet coordinate system, is the first-order derivative of the longitudinal displacement s with respect to time in the Frenet coordinate system, is the second-order derivative of the longitudinal displacement s with respect to time in the Frenet coordinate system. Thus, the state quantity is obtained: The action space is designed as the longitudinal end state of the trajectory:

[0010] Step 3: This invention uses the A3C algorithm framework and, during the exploration process, uses the fleet-shared network training method to generate training samples to populate the experience pool. All agents share the policy network and participate in network training together to avoid network non-convergence issues.

[0011] The obtained state quantity and action space As input, the policy gradient algorithm is used to improve the Lattice planning algorithm, and the reward function is designed in combination with the RSS (responsibility-sensitive safety) strategy to train the final state sampling points of the agent.

[0012] Policy Network π θ The optimization goal of (z,a) is to maximize the expected return of the output planning trajectory:

[0013]

[0014] Where z is the state characteristic of the surrounding traffic environment, a is the network output action (i.e., the longitudinal end state of the trajectory), θ is the network parameter, p(τ,θ) is the probability of executing action a and outputting trajectory τ under parameters θ and state z, r(τ) is the reward function of trajectory τ, θ represents the network parameter, and π represents the policy network.

[0015] The above policy network π θ The optimization method for (z,a) is the gradient ascent method: α represents the learning rate of the gradient descent.

[0016] Calculate the derivative of the optimization objective with respect to the network parameters θ:

[0017]

[0018] In the actual sampling process, the agent continuously obtains trajectories and rewards from traffic scenes, and then adjusts the strategy according to the rewards, and uses the experience data in real time.<z,a,τ,r> Store in the experience pool (Memory). During training, use the Monte Carlo method to randomly sample n pieces of experience data from the experience pool.<z,a,τ,r> , according to the law of large numbers, the gradient of the objective function Make a simplified approximation:

[0019]

[0020] The update direction of the final strategy parameter θ is:

[0021]

[0022] To reduce the variance, we add a baseline b to the reward r(τ) to reduce the variance:

[0023] J(π)=∑ τ p(τ,θ)·[r(τ)-b]

[0024] Next, the baseline part of the objective function J(π) is split into:

[0025] Derivative of the network parameter θ for the part BL containing the baseline:

[0026]

[0027] Obviously, according to the derivative of BL with respect to the network parameter θ, adding the baseline b to the objective function J(π) will not affect the gradient of the final optimization target J(π) Adding a baseline b that is unrelated to action a does not affect the gradient of the final policy.

[0028] The variance is calculated according to the formula:

[0029]

[0030] Obviously, The smaller the value, the smaller the variance. Design the function of this part about b

[0031] Then find the derivative of f(b) with respect to b:

[0032]

[0033] The derivative of f(b) with respect to b, f'(b), is at b = ∑ τ r(τ) is 0, that is, when b=∑ τ When r(τ) is small, f(b) is the smallest and the variance is the smallest. Obviously, ∑ τ r(τ) is the implicit state value V(z), so the state value V(z) can be used as the baseline b to reduce the variance and improve the convergence speed and effect of the policy network.

[0034] During the training process, the intelligent agent of reinforcement learning continuously improves its own capabilities. In this process, the intelligent agent needs to constantly try and error in an unfamiliar state space. Under unfamiliar state characteristics, new behaviors may enable the intelligent agent to obtain higher rewards, but it may also make the behavior worse. "Exploration behavior" is to try some new actions. "Exploitation behavior" is to take actions that are known to obtain the maximum reward, and only strategic actions need to be clearly executed. Too much "exploration behavior" (less "exploitation behavior") will make it more difficult for the intelligent agent to converge, and too little "exploration behavior" (more "exploitation behavior") is very likely to make the intelligent agent converge in a local optimal space. Therefore, a trade-off needs to be achieved between "exploration behavior" and "exploitation behavior". In order to strengthen the intelligent agent's exploration ability in an unfamiliar state space during training and prevent the intelligent agent from falling into a local optimal space during training, the strategy network π designed by the present invention is used. θ The output of (z,a) will conform to the normal distribution. Specifically, it includes two parts: mean μ(z,θ) and variance σ(z,θ):

[0035]

[0036] In theory, the policy network π θ (z,a)In the learning process, the output mean μ(z,θ) will continue to approach the optimal strategy arg max Q(z,a), Q(z,a) represents the action value function, arg max The goal is to find the variable points z and a that make Q reach its maximum value. The output variance σ(z,θ) will continue to approach 0, and the randomness of the strategy will decrease. When executing the strategy, actions are sampled from the normal distribution. Output and execute.

[0037] Step 4: Longitudinal trajectory polynomial fitting. Using the current longitudinal state of the vehicle and the optimal longitudinal final state output by reinforcement learning As a boundary condition, there is a fifth-degree polynomial in s with respect to time t:

[0038]

[0039] The boundary conditions are:

[0040]

[0041] According to the fifth-order polynomial of the longitudinal trajectory and the boundary conditions:

[0042]

[0043] According to the obtained a1a2a3a4a5, the longitudinal trajectory fifth-order polynomial s can be obtained trajectory .

[0044] The optimal trajectory is obtained and input to the control module.

[0045] The present invention provides an intelligent automobile controller, wherein the controller has a built-in execution program of the method.

[0046] The present invention also provides a storage device which has the program code of the method built in.

[0047] Beneficial effects of the present invention:

[0048] (1) This paper addresses the problem of commercial vehicle platooning by combining the Lattice algorithm and deep reinforcement learning under the RSS (Responsibility Sensitive Safety) strategy for autonomous driving tasks. Using the A3C framework, training efficiency is greatly improved, promoting network convergence. Furthermore, under the RSS (Responsibility Sensitive Safety) framework, the safety of reinforcement learning-based path planning is significantly enhanced.

[0049] (2) Compared with the Lattice algorithm, this invention abandons the time-consuming sampling and evaluation process of each alternative trajectory cost function, greatly improving the timeliness of the algorithm. At the same time, the reinforcement learning training process is more universal, and the design of the reward function based on the final control effect will make it more adaptable to complex traffic scenarios and complex vehicle dynamics. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flow chart of the method of the present invention;

[0051] Figure 2 The Policy Gradient network neural network structure used in the present invention;

[0052] Figure 3 ST graph trajectory point sampling. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings.

[0054] The present invention provides a commercial vehicle platoon path planning method that combines deep reinforcement learning (DRL) and Lattice algorithm under the RSS (responsibility-sensitive safety) strategy, which can improve the safety and stability of large commercial vehicles when traveling in a platoon. Figure 1 As shown, the specific steps include:

[0055] like Figure 1 As shown, using the A3C framework, during the exploration process, a shared network training method for the fleet is employed to generate individual gradients to populate Global_net. Compared to taking the states of all fleet members as input and outputting a trajectory of the number of members, this method only takes the state of each agent as input and outputs its own trajectory, promoting network convergence and preventing interference between member actions and reward cancellation. Simultaneously, all intelligent connected vehicles share a decision-making network and participate in network training. The following describes the individual training process for each agent.

[0056] The agent trained using deep reinforcement learning outputs the longitudinal final state sampling points:

[0057] (1) Feature Bird's-Eye View Design. This paper designs an effective and concise time-series bird's-eye view as the state quantity of the policy network, which greatly improves the learning of the policy network and ensures the security of the output trajectory.

[0058] The generation of the time series bird's-eye view includes the following two steps: (1) According to the perception module of the autonomous vehicle, the surrounding environment information is obtained, including dynamic and static obstacles and lane lines. The prediction module is used to obtain the dynamic obstacles in the future 0~t end(2) The information obtained by the perception module and prediction is used to generate a bird's-eye view of features in three dimensions: horizontal, vertical, and time.

[0059] The dimensions of the three-dimensional time-series bird's-eye view matrix are (40, 400, 80). The first dimension 40 represents the lateral range of 10 meters to the left and right of the reference line, with a lateral displacement interval of 0.5 meters; the second dimension 400 represents the longitudinal range of 200 meters forward from the vehicle as the origin, with a longitudinal displacement interval of 0.5 meters; and the third dimension 80 represents the time range within the next 8 seconds, with a time interval of 1 second. Specifically, when a point [α, β, γ] in the time-series bird's-eye view matrix is ​​-1, it indicates that there is an obstacle or a non-drivable area at that point in the time-space; when a point [α, β, γ] in the time-series bird's-eye view matrix is ​​0, it indicates that the point is a drivable area in the time-space; when a point [α, β, γ] in the time-series bird's-eye view matrix is ​​1, it indicates that the point is a point on the reference line.

[0060] (2) State quantity design. In order to obtain the state quantity of the intelligent body at the current moment from the characteristic bird's-eye view, the Frenet coordinate transformation is performed. In the Cartesian coordinate system, the coordinate information of the vehicle (X, θ X ,k X ,v X ,a X ) can be transformed into Among them, X is the coordinate of the vehicle in the Cartesian coordinate system, which is a vector, θ X is the orientation of the vehicle in the Cartesian coordinate system, k X is the curvature, v X is the linear velocity of the vehicle in the Cartesian coordinate system, a X is the acceleration of the vehicle in the Cartesian coordinate system. s is the longitudinal displacement in the Frenet coordinate system, is the first-order derivative of the longitudinal displacement s with respect to time in the Frenet coordinate system, is the second-order derivative of the longitudinal displacement s with respect to time in the Frenet coordinate system. Thus, the state quantity is obtained: The action space is designed as the longitudinal end state of the trajectory:

[0061] (3) Design of strategy network. and action space As input, the policy gradient algorithm is used to improve the Lattice planning algorithm, and the reward function is designed in combination with the RSS (responsibility-sensitive safety) strategy to train the final state sampling point of the intelligent agent. θ The optimization goal of (z,a) is to maximize the expected return of the output planning trajectory:

[0062]

[0063] Where z is the state characteristic of the surrounding traffic environment, a is the network output action (i.e., the longitudinal end state of the trajectory), θ is the network parameter, p(τ,θ) is the probability of executing action a and outputting trajectory τ under parameters θ and state z, and R(τ) is the reward function of trajectory τ.

[0064] Policy Network π θ The optimization method for (z,a) is the gradient ascent method:

[0065] Derivative the expected return J(θ) of the parameter θ to find the optimal θ. At this time, the policy network π is optimal, the trajectory is optimal, and the derivative of the optimization objective with respect to the network parameter θ is calculated:

[0066]

[0067] In the actual sampling process, the agent continuously obtains trajectories and rewards from traffic scenes, and then adjusts the strategy according to the rewards, and uses the experience data in real time.<z,a,τ,r> Store in the experience pool (Memory). During training, use the Monte Carlo method to randomly sample n pieces of experience data from the experience pool.<z,a,τ,r> , according to the law of large numbers, the gradient of the objective function Make a simplified approximation:

[0068]

[0069] To reduce the variance, we add a baseline b to the reward r(τ) to reduce the variance:

[0070] J(π)=∑ τ p(τ,θ)·[r(τ)-b]

[0071] Next, the baseline part of the objective function J(π) is split into:

[0072] Derivative of the network parameter θ for the part BL containing the baseline:

[0073]

[0074] Obviously, according to the derivative of BL with respect to the network parameter θ, adding the baseline b to the objective function J(π) will not affect the gradient of the final optimization target J(π) Adding a baseline b that is unrelated to action a does not affect the gradient of the final policy.

[0075] The variance is calculated according to the formula:

[0076]

[0077] Obviously, The smaller the value, the smaller the variance. Design the function of this part about b

[0078] Then find the derivative of f(b) with respect to b:

[0079]

[0080] The derivative of f(b) with respect to b, f'(b), is at b = ∑ τ r(τ) is 0, that is, when b=∑ τ When r(τ) is small, f(b) is the smallest and the variance is the smallest. Obviously, ∑ τ r(τ) is the implicit state value V(z), so the state value V(z) can be used as the baseline b to reduce the variance and improve the convergence speed and effect of the policy network.

[0081] In order to enhance the agent's ability to explore in unfamiliar state spaces during training and prevent the agent from falling into local optimal spaces during training, the strategy network π designed in this invention is θ The output of (z,a) will conform to the normal distribution. Specifically, it includes two parts: mean μ(z,θ) and variance σ(z,θ):

[0082]

[0083] In theory, the policy network π θ (z,a)In the learning process, the output mean μ(z,θ) will continue to approach the optimal strategy arg max Q(z,a), the output variance σ(z,θ) will continue to approach 0, and the randomness of the strategy will decrease. When executing the strategy, actions are sampled from the normal distribution Output and execute.

[0084] (4) Design the reward function. The reward function reward consists of the following parts, where k1 to k3 are the proportional coefficients corresponding to each part of the reward:

[0085] reward = k1·r speed +k2·r acc +k3·r safe

[0086] where r speed is the speed reward, the goal is to keep the vehicle speed at the target speed, v target is the desired target speed, t total is the number of trajectory points corresponding to the trajectory in time units, v t The speed of the planned trajectory at time t is:

[0087]

[0088] where r acc For longitudinal comfort bonus, the goal is to keep the longitudinal acceleration small. is the longitudinal acceleration of the planned trajectory at time t:

[0089]

[0090] where r safe For safety rewards, the goal is to ensure that the generated trajectory meets the safety standards. Under the RSS (Responsibility Sensitive Safety) strategy, the reward function is further designed.

[0091] Longitudinal safety distance:

[0092]

[0093] v f is the speed of the preceding vehicle, v r is the speed of the following vehicle, ρ is the driver's reaction time, a min,brake is the minimum braking acceleration, a max,brake is the maximum braking acceleration, a max,accel is the maximum acceleration.

[0094] Horizontal safety distance:

[0095]

[0096] Among them, v1 is the speed of the vehicle itself, v2 is the lateral speed of the vehicle trying to cut in, μ is the minimum lateral distance when the lateral speed of the two vehicles is 0. is the maximum lateral acceleration, is the minimum lateral braking acceleration, and ρ is the driver's reaction time.

[0097] When driving according to the trajectory generated by the policy network, if the horizontal and vertical distances to the vehicles in front and behind or other vehicles that squeeze into the platoon are less than the minimum safe distance, the reward is -100, otherwise it is 0:

[0098]

[0099] Where d is the distance to other vehicles.

[0100] (5) Longitudinal trajectory polynomial fitting. Using the current longitudinal state of the vehicle and the optimal longitudinal final state output by reinforcement learning As a boundary condition, there is a fifth-degree polynomial in s with respect to time t:

[0101]

[0102] The boundary conditions are:

[0103]

[0104] According to the fifth-order polynomial of the longitudinal trajectory and the boundary conditions:

[0105]

[0106] According to the obtained a0a1a2a3a4a5, the longitudinal trajectory fifth-order polynomial s can be obtained trajectory .

[0107] Finally, the obtained trajectory is input into the control module for trajectory tracking control.

[0108] like Figure 2 As shown, the policy network π θ (z,a) specifically includes two parts: the convolutional (CNN) feature extraction network and the fully connected network (FCN). Among them, z is the input state of the policy network, including the time-series bird's-eye view matrix and the historical trajectory of the vehicle; a is the output of the policy network, that is, the final state of the planned trajectory. θ is the weight and bias parameter of the network. The input of the convolutional (CNN) feature extraction network is the above-mentioned spatiotemporal bird's-eye view matrix, and the output is the final extracted environmental feature information. The input of the fully connected network (FCN) is the environmental feature information output by the convolutional (CNN) feature extraction network and the historical trajectory information of the autonomous vehicle, and the output is the final state of the trajectory.

[0109] The convolutional neural network of the policy network includes three convolutional layers, two pooling layers and three fully connected layers. The input layer merges three 256*256*3 matrices into a 256*256*9 matrix; the convolution layer Conv1 is composed of a convolution kernel of (3*3*9)*32 and a stride of 2, and its input is the output of the input layer, which is a 256*256*9 matrix, and its output is a 128*128*32 feature; the pooling layer Pool1 is composed of a pooling kernel of (2*2) and a stride of 2, and its input is the output of the convolution layer Conv1, which is a 128*128*32 feature, and its output is a 64*64*32 feature; the convolution layer Conv2 is composed of a convolution kernel of (3*3*32)*64 and a stride of 2, and its input is the output of the pooling layer Pool1, which is a 64*6 4*32 features, and its output is 32*32*128 features; the pooling layer Pool2 is composed of a pooling kernel of (2*2) and a step size of stride=2, and its input is the output of the convolution layer Conv2, which is a 32*32*128 feature, and its output is a 16*16*128 feature; the convolution layer Conv3 is composed of a convolution kernel of (3*3*128)*128 and a step size of stride=2, and its input is the output of the pooling layer Pool2, which is a 16*16*128 feature, and its output is a 8*8*128 feature; the size of the fully connected layer FC is (8*8*128)*512, and its input is the output of the convolution layer Conv3, which is a 8*8*128 feature, and its output is a 1*1*512 feature. The fully connected layers FC-μ and FC-σ are connected in parallel. Their inputs are features extracted by the convolutional neural network (CNN), which are 1x1x512 features. The outputs of the fully connected layers FC-μ and FC-σ are 1x1x512 features, and the outputs of the fully connected layers FC-σ are 1x1x512 features. The features extracted by the fully connected layers FC-μ and FC-σ together constitute the state feature z.

[0110] like Figure 3 As shown in the ST diagram, traffic scenes are mainly divided into two main situations:

[0111] (1) There is no obstacle in front of the vehicle. The trajectory is fitted based on the initial state and the final state trajectory points trained by deep reinforcement learning, and longitudinal velocity planning is performed.

[0112] (2) There is an obstacle in front of the vehicle. The present invention depicts the obstacle as a parallelogram that blocks part of the road during a specific time period. For example, in the animation below, the prediction module predicts that the vehicle will enter the lane between t0 and t1 and will occupy positions s0 to s1 during this period. Therefore, a rectangle is drawn on the ST diagram that will block positions s0 to s1 during the time period t0 to t1. To avoid collision, the speed curve must not intersect this rectangle.

[0113] When following a vehicle, the speed curve is below the following vehicle sampling lower bound. When overtaking a vehicle, the speed curve is above the overtaking vehicle sampling lower bound.

[0114] In summary, the present invention aims at automatic driving tasks and, under the RSS (responsibility-sensitive safety) strategy, adopts a method combining the Lattice algorithm and deep reinforcement learning to solve the problem of commercial vehicle platoon driving. The use of the A3C framework greatly improves the efficiency of training and promotes network convergence. At the same time, under the RSS (responsibility-sensitive safety) framework, the security of the reinforcement learning planning path is greatly improved. Compared with the Lattice algorithm, the present invention abandons the sampling and evaluation process of each alternative trajectory cost function with higher time complexity, which greatly improves the timeliness of the algorithm. At the same time, the training process of reinforcement learning is more universal, and the design of the reward function based on the final control effect will make it more adaptable to complex traffic scenarios and complex vehicle dynamics.

[0115] In addition, an embodiment of the present invention further provides an intelligent automobile controller, wherein the controller has a built-in execution program of the above method.

[0116] An embodiment of the present invention further provides a storage device which has the program code of the above method built in.

[0117] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent methods or changes that do not deviate from the technology of the present invention should be included in the scope of protection of the present invention.

Claims

1. A commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy, characterized in that: The steps include: S1. Design a time sequence bird’s-eye view as the state quantity of the strategy network; S2. Perform Frenet coordinate transformation to obtain the state of the agent at the current moment from the feature bird's-eye view And design the action space as the longitudinal end state of the trajectory: Where s is the longitudinal displacement in the Frenet coordinate system, is the first-order derivative of the longitudinal displacement s with respect to time t in the Frenet coordinate system, is the second-order derivative of the longitudinal displacement s with respect to time t in the Frenet coordinate system, S3, the obtained state quantity and action space As the input of the policy network, the policy gradient algorithm is used to improve the Lattice planning algorithm. At the same time, the reward function is designed in combination with the RSS strategy to train the final longitudinal state of the agent; The policy network π θ (z,a) includes the convolutional feature extraction network CNN and the fully connected network FCN; z is the input state of the policy network, including the time series bird's-eye view matrix and the historical trajectory of the vehicle; a is the output of the policy network, that is, the final state of the planned trajectory θ is the weight and bias parameter of the network. The input of the convolutional feature extraction network CNN is the above-mentioned spatiotemporal bird's-eye view matrix, and the output is the final extracted environmental feature information. The input of the fully connected network FCN is the environmental feature information output by the convolutional feature extraction network CNN and the historical trajectory information of the autonomous vehicle, and the output is the final state of the trajectory. The convolutional neural network of the policy network includes three convolutional layers, two pooling layers and three fully connected layers. The input layer merges three 256*256*3 matrices into a 256*256*9 matrix; the convolution layer Conv1 is composed of a convolution kernel of (3*3*9)*32 and a stride of 2. Its input is the output of the input layer, which is a 256*256*9 matrix, and its output is a 128*128*32 feature; the pooling layer Pool1 is composed of (2*2) and a stride of The convolution layer Conv2 is composed of a convolution kernel of (3*3*32)*64 and stride=2, whose input is the output of the pooling layer Pool1, which is a feature of 64*64*32, and whose output is a feature of 32*32*128; the pooling layer Pool2 is composed of a convolution kernel of (2*2)*64 and stride=2, whose input is the output of the pooling layer Pool1, which is a feature of 64*64*32, and whose output is a feature of 32*32*128; the pooling layer Pool2 is composed of a convolution kernel of (2*2)*64 and stride= The convolution layer Conv3 is composed of a convolution kernel of (3*3*128)*128, with a stride of 2, and its input is the output of the pooling layer Pool2, which is a feature of 16*16*128, and its output is a feature of 8*8*128; the size of the fully connected layer FC is (8*8*128)*512, and its input is a feature of 16*16*128, and its output is a feature of 8*8*128. The input is the output of the convolutional layer Conv3, which is a feature of 8*8*128, and its output is a feature of 1*1*512. The fully connected layer FC-μ and the fully connected layer FC-σ are parallel structures. The input is the feature extracted by the convolutional neural network, which is a feature of 1*1*512. The output of the fully connected layer FC-μ is a feature of 1*1*512, and the output of the fully connected layer FC-σ is a feature of 1*1*512. The features extracted by the fully connected layer FC-μ and the fully connected layer FC-σ together constitute the state feature z; The reward function of the policy network is designed as follows: reward=k1·r speed +k2·r acc +k3·r safe Among them, k1~k3 are the proportional coefficients corresponding to each part of the reward, r speed is the speed reward, the goal is to keep the vehicle speed at the target speed, v target is the desired target speed, t total is the number of trajectory points corresponding to the trajectory in time units, v t The speed of the planned trajectory at time t is: where r acc For longitudinal comfort bonus, the goal is to keep the longitudinal acceleration small. is the longitudinal acceleration of the planned trajectory at time t: where r safe For safety rewards, the goal is to ensure that the generated trajectory meets safety standards; Longitudinal safety distance: v f is the speed of the preceding vehicle, v r is the speed of the following vehicle, ρ is the driver's reaction time, a min,brake is the minimum braking acceleration, a max,brake is the maximum braking acceleration, a max,accel is the maximum acceleration; Horizontal safety distance: v1 is the vehicle's speed, v2 is the lateral speed of the other vehicle, and μ is the minimum lateral distance when the lateral speeds of the two vehicles are 0. is the maximum lateral acceleration, is the minimum lateral braking acceleration, ρ is the driver’s reaction time; When driving according to the trajectory generated by the policy network, if the horizontal and vertical distances to the vehicles in front and behind or other vehicles that squeeze into the platoon are less than the minimum safe distance, the reward is -100, otherwise it is 0: d is the distance to other vehicles; S4. Using the current longitudinal state of the vehicle and the final longitudinal state As the boundary condition, the longitudinal trajectory polynomial fitting is performed to obtain the optimal trajectory.

2. The commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy according to claim 1 is characterized in that: In S1, the generation of the time series bird's-eye view includes the following two steps: (1) obtaining the surrounding environment information, including dynamic and static obstacles and lane lines, and predicting the dynamic obstacles in the future 0~t end (2) Generate a bird's-eye view of features in three dimensions: horizontal, vertical, and time, using the obtained environmental information and predicted information.

3. The commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy according to claim 1 is characterized in that: The size of the time-series bird's-eye view matrix is ​​(40, 400, 80), where the first dimension 40 represents the lateral range of 10 meters to the left and right of the reference line, with a lateral displacement interval of 0.5 meters; the second dimension 400 represents the longitudinal range of 200 meters forward from the vehicle as the origin, with a longitudinal displacement interval of 0.5 meters; the third dimension 80 represents the time range within the next 8 seconds, with a time interval of 1 second; when the point [α, β, γ] in the time-series bird's-eye view matrix is ​​-1, it means that there is an obstacle or a non-drivable area at this point in the time space; when the point [α, β, γ] in the time-series bird's-eye view matrix is ​​0, it means that this point is a drivable area in the time space; when the point [α, β, γ] in the time-series bird's-eye view matrix is ​​1, it means that this point is a point on the reference line.

4. The commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy according to claim 1 is characterized in that: In S2, the Frenet coordinate transformation is as follows: In the Cartesian coordinate system, the vehicle's coordinate information (X, θ X ,k X ,v X ,a X ) is transformed into Among them, X is the coordinate of the vehicle in the Cartesian coordinate system, which is a vector, θ X is the orientation of the vehicle in the Cartesian coordinate system, k X is the curvature, v X is the linear velocity of the vehicle in the Cartesian coordinate system, a X is the acceleration of the vehicle in the Cartesian coordinate system.

5. The commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy according to claim 1 is characterized in that: In S3, the policy network π θ The optimization goal of (z,a) is to maximize the expected return of the output planning trajectory: Where z is the state characteristic of the surrounding traffic environment, a is the network output action, that is, the longitudinal final state of the trajectory, θ is the network parameter, p(τ,θ) is the probability of executing action a and outputting trajectory τ under parameters θ and state z, and r(τ) is the reward function of trajectory τ; The policy network π θ The optimization method for (z,a) is the gradient ascent method: Derivative J(θ) is taken to calculate the derivative of the optimization objective with respect to the network parameter θ: In the actual sampling process, the agent continuously obtains trajectories and rewards from the traffic scene, and then adjusts the strategy according to the rewards, and stores the experience data <z,a,τ,r> into the experience pool in real time. During training, the Monte Carlo method is used to randomly sample n pieces of experience data <z,a,τ,r> from the experience pool, and the gradient of the objective function is calculated according to the law of large numbers. Make a simplified approximation: To reduce the variance, we add a baseline b to the reward r(τ) to reduce the variance: J(π)=∑ τ p(τ,θ)·[r(τ)-b] Next, the baseline part of the objective function J(π) is split into: Derivative of the network parameter θ for the part BL containing the baseline: Calculate the variance: The smaller it is, the smaller the variance is. Design this part as a function of b Then find the derivative of f(b) with respect to b: The derivative of f(b) with respect to b, f'(b), is at b = ∑ τ r(τ) is 0, that is, when b=∑ τ When r(τ) is the smallest, f(b) is the smallest, the variance is the smallest, ∑ τ r(τ) is the implicit state value V(z), so the state value V(z) can be used as the baseline b to reduce the variance and improve the convergence speed and effect of the policy network; The policy network π θ The output of (z, a) conforms to the normal distribution, specifically including the mean μ(z, θ) and the variance σ(z, θ): Policy Network π θ (z,a)In the learning process, the output mean μ(z,θ) will continue to approach the optimal strategy arg max Q(z,a), the output variance σ(z,θ) will continue to approach 0, the randomness of the strategy will decrease, and when executing the strategy, actions are sampled from the normal distribution. Output and execute.

6. The commercial vehicle platoon path planning method combining deep reinforcement learning and RSS strategy according to claim 1 is characterized in that: In S4, the longitudinal trajectory polynomial fitting is as follows: The boundary conditions are: According to the fifth-order polynomial of the longitudinal trajectory and the boundary conditions: According to the obtained a0a1a2a3a4a5, the longitudinal trajectory fifth-order polynomial s can be obtained trajectory .

7. An intelligent automobile controller, characterized in that: The controller has a built-in execution program for the method according to any one of claims 1 to 6.

8. A storage device, characterized in that: It has built-in program code of the method described in any one of claims 1 to 6.

Images