An autonomous driving commercial vehicle learning type emergency collision avoidance control method, system and storage medium

Abstract

The application discloses a kind of automatic driving commercial vehicle learning type emergency collision avoidance control method, system and storage medium.The method comprises the following steps: based on soft actor critic architecture, reinforcement learning network model is constructed, corresponding bayesian policy network is constructed for commercial vehicle driving task and control action, and replay buffer of decay expert experience is extended, so as to produce joint control strategy suitable for emergency collision avoidance working condition;Vehicle applies the continuous control action generated by bayesian policy network to driving decision task, and under the guidance of reward function, collision avoidance task without collision under high speed is taken as target, and automatic driving is carried out.The method of the application utilizes bayesian policy network and decay expert experience guidance technology, realizes the learning efficiency of automatic driving commercial vehicle emergency collision avoidance control and driving strategy improvement, is conducive to solving the long tail problem of safety of automatic driving commercial vehicle, and helps automatic driving commercial vehicle application promotion.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a learning-based emergency collision avoidance control method, system, and storage medium for autonomous commercial vehicles. Background Technology

[0002] With the rapid development of artificial intelligence and automotive technology, autonomous vehicles (intelligent vehicles) are changing the traditional way of controlling vehicles, freeing drivers from the closed-loop control of "vehicle-road-person", thereby improving the efficiency and safety of the transportation system.

[0003] Currently, autonomous driving technology for commercial vehicles is the most feasible for practical deployment due to its economic cost and the unique nature of its application scenarios. However, due to the large size, weight, and blind spots of commercial vehicles, their requirements for emergency collision avoidance are far higher than those for other vehicles. Furthermore, because controlling commercial vehicles is more difficult than controlling ordinary vehicles, emergency collision avoidance demands higher levels of scene perception and motion control.

[0004] Existing technologies for solving the upper-level action decision-making problem of autonomous commercial vehicles are mainly divided into modular autonomous driving technology and learning-based autonomous driving technology. Modular autonomous driving technology divides the autonomous driving problem into modules such as perception, decision-making, planning, and control. Commonly used planning and control algorithms, such as dynamic A* and MPC, have high computational complexity, resulting in long decision-making and control times and placing a significant burden on the computing power of the onboard domain controller. Furthermore, they require sophisticated vehicle modeling and have limited applicability. Learning-based autonomous driving technology, on the other hand, directly maps the input information from onboard sensors to the output of vehicle control actions through a neural network model. Algorithms such as imitation learning and deep reinforcement learning are used. These methods achieve self-learning based on data or exploration, without the need for manually defined rules.

[0005] The invention disclosed in CN119150652A presents a method for planning safe driving behavior inspired by defensive driving experience, including S1: initializing the traffic environment and setting the vehicle state based on the Gym library; S2: designing the state space; S3: designing the action space; S4: designing the reward function; and S5: training the algorithm until convergence. This invention applies soft safety rules such as "defensive driving" and the "road sense" and "driving intuition" of human drivers to current autonomous driving systems, making autonomous driving functions more complete, increasing the ability to plan the motion of intelligent vehicles, and enabling autonomous driving systems to respond more intelligently to different emergency road conditions. The invention disclosed in CN116176572A presents a vehicle emergency collision avoidance control method based on DQN deep reinforcement learning, which can solve the problem that current AEB systems cannot avoid collisions in sudden situations where the distance to an obstacle is less than the total longitudinal braking distance, and that relying solely on sensor information, they can only perform the same braking action when facing different obstacles. The method, system, medium, and device for T-shaped emergency collision avoidance control of autonomous vehicles, disclosed in CN113460090B, can maximize the collision avoidance potential of autonomous vehicles and improve their performance in high-speed emergency avoidance and extreme driving conditions. However, in the aforementioned deep reinforcement learning-based methods, when faced with complex control tasks, the logical relationship of control action outputs is rarely considered, leading to problems such as slow training efficiency and poor policy performance. Summary of the Invention

[0006] The purpose of this invention is to provide a learning-based emergency collision avoidance control method, system, and storage medium for autonomous commercial vehicles, enabling autonomous commercial vehicles to learn emergency collision avoidance control strategies through training and efficiently complete emergency collision avoidance tasks in application scenarios.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention discloses a learning-based emergency collision avoidance control method for autonomous commercial vehicles, the method comprising the following steps:

[0009] Step 1: Collect driving data from human drivers, construct an expert experience dataset, and add the expert experience dataset to the replay buffer in the form of a decay ratio;

[0010] Step 2: Construct a reinforcement learning network model based on the soft actor critic architecture. The reinforcement learning network model includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, and the value network and Q-value network evaluate the driving actions generated by the policy network. The policy network, value network, and Q-value network are trained using observation-action pairs in the replay buffer to update the corresponding network parameters.

[0011] Step 3: Based on the reinforcement learning network model built in Step 2, construct a corresponding Bayesian policy network for the commercial vehicle driving task and control actions. Extend the policy network structure and update process from Step 2 to generate a joint control strategy adapted to emergency collision avoidance conditions. The node set of the Bayesian policy network is the steering δ... k throttle a k and braking k A strategy involving three actions;

[0012] Step 4: Apply the driving actions generated by the Bayesian policy network to the vehicle's driving task, and train the model to generate the optimal policy by interacting with the driving environment and aiming at collision avoidance without collision at high speeds, under the guidance of the reward function.

[0013] Furthermore, in step 1, the collected expert experience dataset is arranged in the replay buffer with a priority of ε, which helps the autonomous commercial vehicle to map the state to actions based on the demonstrations provided by experts for initial exploration; where ε is set as a gradually decaying proportional coefficient that gradually decays to zero as training progresses.

[0014] Further, in step 2, the reinforcement learning network model includes a policy network, a value network, and two Q-value networks. The policy network, designed according to the dimension of state information, is used to generate continuous driving actions based on the current sensor input observations. The state space of the reinforcement learning network model is s = [θ]. k ,d k ,d axis ,v k ,w k ,n k ,f c ,op k In the formula, θ k d is the angle between the vehicle's direction and the center of the road. k d represents the distance between vehicles and the road edge within a 200-meter range returned by the sensor. axis The normalized distance between the vehicle and the road centerline; v k It includes three speed scalars: the vehicle speed along the direction of travel, the vehicle speed laterally along the axis of travel, and the vehicle speed along the Z-axis; w k It contains n scalars, representing the rotational speeds of the vehicle's n wheels, n k f is the engine's speed per minute. c It is a binary flag indicating a collision between a vehicle and other road users, op k Store information about the nearest vehicle to the current vehicle, including relative distance and angle; the action space is a = [δ]. k ,ak ,br k In the formula, δ k For steering actions, the range is [-1, +1], a k To accelerate the action, the range is [0,1], br k This is a braking action, with a range of [0,1].

[0015] Furthermore, in step 2, the policy network, value network, and Q-value network are all constructed from a 5-layer neural network, namely a state input layer, two hidden layers containing 256 neurons each, a normalization layer, and a driving action output layer.

[0016] The observation information acquired by the sensor is normalized and sequentially input into the state input layer. After feature extraction and processing through two hidden layers, it is activated by the ReLU function to obtain the tensor corresponding to the observation information. In the policy network, the tensor is activated by the linear function of the driving action output layer to obtain the final action space. In the value network and Q-value network, the tensor is activated by the linear function of the driving action output layer to obtain the final action evaluation.

[0017] Furthermore, in step 2, the learning rate of the policy network is 0.0003, the learning rate of the Q-value network is 0.003, the influence weight of future action reward is 0.96, the soft update parameter of the target network is 0.005, the replay buffer size is 1,000,000, and the sampling batch size is 256.

[0018] Furthermore, in step 3, the Bayesian policy network outputs the driving action policy T = (t i ) i∈I This can be described as being driven by an acceleration control strategy t a Steering control strategy t δ and braking control strategy t br The probability π of the three sub-policy actions T The joint probability function is composed of the following relationships:

[0019] π T (t δ ,t a ,t br )=π1(t a )π2(t δ |t a )π3(t br |t a ).

[0020] Furthermore, in step 2, the joint control strategy adopted by the Bayesian policy network is as follows:

[0021]

[0022] In the formula, A is the joint action distribution of policy π. It is each sub-strategy π i Action space, θ i These are the corresponding parameters;

[0023] The value network uses the value function V(A) in the following formula. t ,s) Evaluate the performance of the joint control strategy:

[0024]

[0025] Wherein, π(A) t |s;θ) represents the action A taken in state s. t The probability, q(s,A) t ;ω) represents the action A taken in state s. t The expected return, where θ and ω are the corresponding network parameters.

[0026] The gradient descent of the corresponding parameters is calculated using the following formula:

[0027]

[0028] The joint control strategy uses the following objective function J: v (π) is calculated to maximize the sub-strategy π. i The reward r(s) t A t ) and entropy sum:

[0029]

[0030] Here, α is a hyperparameter used to control the weight of the entropy regularization term.

[0031] In the evaluation process of soft policy iteration, the soft Q value is calculated iteratively starting from an arbitrary function Q. And repeatedly apply the modified Bellman backup operator. The soft value function of the updated sub-policy, the Bellman backup operator, is given by the following equation:

[0032]

[0033] In the improvement step of each sub-policy, the corresponding policy is updated for the state at each time step according to the following formula:

[0034]

[0035] in, It is the old strategy action value function. It is the normalization factor.

[0036] In each sub-policy iteration, soft policy evaluation and soft policy improvement are performed alternately to ensure optimal maximum entropy convergence among sub-policy combinations.

[0037] Further, in step 4, the reward function is:

[0038]

[0039] in, For driving efficiency indicators, For driving safety indicators, For driving comfort indicators, As an indicator of the completion rate of emergency collision avoidance missions;

[0040]

[0041]

[0042] In the formula, v k It is the vehicle's speed, v limit With v max These represent the maximum speed limit of the road and the maximum speed the vehicle can reach, respectively; ttc is the expected time of collision at the current speed, and d... lat f is the lateral distance between vehicles. c v is a binary indicator of whether a vehicle collision has occurred. x It is the vehicle's current velocity along the axis of travel. ρ ttc ρ lat k c All are hyperparameters; ρ thr With ρ ste These are the hyperparameters for the longitudinal acceleration term and the lateral acceleration term, respectively; Let c be the Euclidean distance between the autonomous commercial vehicle and the background vehicle. r This is a hyperparameter.

[0043] The second invention discloses a learning-based emergency collision avoidance control system for autonomous commercial vehicles as described above, characterized in that the system comprises:

[0044] The dataset building module is used to collect driving data from human drivers, build an expert experience dataset, and add the expert experience dataset to the replay buffer in the form of a decay ratio.

[0045] The reinforcement learning network model construction module is used to construct a reinforcement learning network model based on the soft actor critic architecture. The reinforcement learning network model includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, and the value network and Q-value network evaluate the driving actions generated by the policy network. The policy network, value network, and Q-value network are trained using observation-action pairs in the replay buffer to update the corresponding network parameters.

[0046] The joint control strategy module is used to construct corresponding Bayesian policy networks based on the built reinforcement learning network model, targeting the driving tasks and control actions of commercial vehicles. It extends the policy network structure and update process to generate joint control strategies adapted to emergency collision avoidance conditions. The node set of the Bayesian policy network is the steering δ... k throttle a k and braking k A strategy involving three actions;

[0047] The optimal policy output module is used to apply the driving actions generated by the Bayesian policy network to the vehicle's driving tasks. By interacting with the driving environment, it aims to achieve collision avoidance without collisions at high speeds and trains the model under the guidance of the reward function to generate the optimal policy.

[0048] Thirdly, the present invention discloses a storage medium storing a computer program, which, when executed in a computer, causes the computer to perform the emergency collision avoidance control method as described above.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] First, the autonomous driving commercial vehicle learning-based emergency collision avoidance control method, system, and storage medium of the present invention, by constructing a Bayesian policy network to model the interaction relationship of vehicle control commands and designing a decaying expert experience return buffer to guide the learning of the intelligent agent (autonomous commercial vehicle), solves the problem that the control action output logic relationship of existing learning-based autonomous driving methods is difficult to learn and the reward is sparse in complex scenarios, improves the learning efficiency of reinforcement learning and ensures the optimality and robustness of the final driving strategy.

[0051] Secondly, the learning-based emergency collision avoidance control method, system, and storage medium for autonomous commercial vehicles of the present invention utilize Bayesian policy networks and attenuation expert experience guidance technology to improve the learning efficiency and driving strategy of emergency collision avoidance control for autonomous commercial vehicles. This helps to solve the long-tail problem of safety in autonomous commercial vehicles, promotes the application of autonomous commercial vehicles, and eliminates the need for complex system modeling, resulting in a simplified algorithm architecture. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the deep reinforcement learning network architecture of the present invention;

[0053] Figure 2 This is a schematic diagram of the driving strategy decomposition method and the probability distribution of the joint strategy of the present invention;

[0054] Figure 3 This is the Bayesian policy network structure diagram of the soft actor critic algorithm of this invention. Detailed Implementation

[0055] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0056] This invention discloses a learning-based emergency collision avoidance control method for autonomous commercial vehicles, the method comprising the following steps:

[0057] Step 1: Collect driving data from human drivers, construct an expert experience dataset, and add the expert experience dataset to the replay buffer in the form of a decay ratio;

[0058] Step 2: Construct a reinforcement learning network model based on the soft actor critic architecture. The reinforcement learning network model includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, and the value network and Q-value network evaluate the driving actions generated by the policy network. The policy network, value network, and Q-value network are trained using observation-action pairs in the replay buffer to update the corresponding network parameters.

[0059] Step 3: Based on the reinforcement learning network model built in Step 2, construct a corresponding Bayesian policy network for the commercial vehicle driving task and control actions. Extend the policy network structure and update process from Step 2 to generate a joint control strategy adapted to emergency collision avoidance conditions. The node set of the Bayesian policy network is the steering δ... k throttle a k and braking k A strategy involving three actions;

[0060] Step 4: Apply the driving actions generated by the Bayesian policy network to the vehicle's driving task, and train the model to generate the optimal policy by interacting with the driving environment and aiming at collision avoidance without collision at high speeds, under the guidance of the reward function.

[0061] The emergency collision avoidance control method of the present invention specifically includes the following steps:

[0062] Step 1: Collect driving data from human drivers and build an expert experience dataset.

[0063] Specifically, using a driving simulator and simulation platform, the sensor perception status information of the autonomous vehicle at each moment during a high-speed emergency collision avoidance situation, along with the corresponding human driver control actions (including steering wheel, accelerator, and brake pedal openings), are collected to form a human expert demonstration dataset. This dataset is stored in a matrix format to facilitate the use of human expert driving experience in subsequent steps.

[0064] Step 2: Construct a reinforcement learning network model based on the soft actor critic architecture. This reinforcement learning network model specifically includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, while the value network and Q-value network evaluate the actions. The policy network, value network, and Q-value network are trained using observation-action pairs from the replay buffer to update the corresponding network parameters.

[0065] Specifically, the state space is s = [θ k ,d k ,d axis ,v k ,w k ,n k ,f c ,op k ], where θ k d is the angle between the vehicle's direction and the center of the road. k d represents the distance between vehicles and the road edge within a 200-meter range returned by the sensor. axis v is the normalized distance between the vehicle and the road centerline. k It includes three speed scalars: the vehicle speed along the direction of travel, the vehicle speed laterally along the axis of travel, and the vehicle speed along the Z-axis. k It contains n scalars, representing the rotational speeds of the n wheels of the vehicle. The specific values ​​are determined by the actual number of wheels used in the commercial vehicle. k The engine's speed per minute; the action space is a = [δ] k ,a k ,br k ], δ k For steering actions, the range is [-1, +1], a k To accelerate the action, the range is [0,1], br k This is a braking action, with a range of [0,1]. f c This is a binary flag indicating a collision between the vehicle and other road users. k Stores information about the nearest vehicle to your vehicle, including the relative distance and angle.

[0066] The reinforcement learning network model of this invention includes a policy network, a value network, and two Q-value networks. The policy network, designed according to the dimension of state information, generates continuous driving actions based on the current sensor input observations. A value function is used to estimate the value function of the current driving policy; this invention includes two estimation methods: the value network and the Q-value network. The policy network updates its parameters based on the value function. The autonomous commercial vehicle (i.e., the agent) applies continuous driving actions to enter the next state, receives reward feedback, and stores the state-action pairs between the obtained observations and the current action in a replay buffer. The replay buffer is sampled, and the sampled information is used to update the policy network, value network, and Q-value networks.

[0067] In this embodiment, each neural network comprises five layers: a state input layer, two hidden layers containing 256 neurons each, a normalization layer, and a driving action output layer. First, the observation information acquired by the sensors is normalized and sequentially input into the state input layer. Features are extracted and processed through the two hidden layers and activated using the ReLU function. In the policy network, the tensor is activated by a linear function in the output layer to obtain the final action space. In the value network, the tensor is activated by a linear function in the output layer to obtain the final action evaluation.

[0068] The hyperparameters of the reinforcement learning network model are set to the hyperparameters of the soft actor critic algorithm, where the learning rate of the policy network is 0.0003, the learning rate of the Q network is 0.003, the influence of future action reward (discount factor) is 0.96, the soft update parameter of the target network is 0.005, the replay buffer size is 1,000,000, and the sampling batch size is 256.

[0069] Step 3: Based on the soft actor critic reinforcement learning architecture built in Step 2, construct a corresponding Bayesian policy network for commercial vehicle driving tasks and control actions. Extend the policy network structure and network update process of the soft actor critic reinforcement learning architecture to generate a joint control strategy adapted to emergency collision avoidance conditions.

[0070] Specifically, the three actions a driver can take while driving a vehicle—steering, accelerator, and braking—are independent random variables. To model the process of a skilled driver selecting joint driving actions during emergency collision avoidance and to improve the learning efficiency of the agent, this invention decomposes complex policy combinations into three simple sub-policies based on driving actions. Each of these three policy networks learns its respective action output, and the three sub-policies are constructed into a joint policy using a Bayesian network structure.

[0071] In this invention, the node set of the Bayesian network is respectively the turning δ k throttle a k and brakingk The strategies for the three actions, and the specific relationships between them, such as Figure 2 As shown, the final output driving action strategy T = (t i ) i∈I This can be described as being driven by the acceleration control strategy t a Steering control strategy t δ and braking control strategy t br The probability π of the three sub-policy actions T The joint probability function formed according to this relationship:

[0072] π T (t δ ,t a ,t br )=π1(t a )π2(t δ |t a )π3(t br |t a ).

[0073] This invention uses a corresponding deep convolutional neural network to approximate the joint policy so as to learn the optimal policy in the DRL algorithm:

[0074]

[0075] In the formula, A is the joint action distribution of policy π. It is each sub-strategy π i Action space, θ i These are the corresponding parameters.

[0076] Value networks use the value function V(A) in the following formula. t ,s) Evaluate the performance of a specific joint strategy A.

[0077]

[0078] Wherein, π(A) t |s;θ) represents the action A taken in state s. t The probability, q(s,A) t ;ω) represents the action A taken in state s. t The expected return, where θ and ω are the corresponding network parameters.

[0079] The gradient descent of the corresponding parameters can be calculated using the following formula.

[0080]

[0081] Regarding the concept of maximum entropy for soft-actor critics, due to the additivity of entropy, this invention represents the entropy of a joint policy system as the sum of the entropies of its independent sub-policies. In this invention, the joint policy J... V(π) will be calculated using the objective function of the following formula to maximize the sub-policy π. i The reward r(s) t A t ) and entropy sum.

[0082]

[0083] Here, α is a hyperparameter used to control the weight of the entropy regularization term.

[0084] In the evaluation process of soft policy iteration, the soft Q value can be calculated iteratively starting from any function Q, i.e. And repeatedly apply the modified Bellman backup operator. The soft value function of the updated sub-policy, the Bellman backup operator, is given by the following equation:

[0085]

[0086] In the improvement step of each sub-policy, the corresponding policy is updated for the state at each time step according to the following formula:

[0087]

[0088] in, It is the old strategy action value function. It is the normalization factor.

[0089] In each sub-policy iteration, soft policy evaluation and soft policy improvement are performed alternately to ensure optimal maximum entropy convergence among sub-policy combinations. This invention utilizes two Q-networks to mitigate positive bias in the policy improvement step and alternately uses experience gathered from the environment by the current policy and stochastic gradients from mini-batch samples from the replay buffer to update the function approximator.

[0090] Step 4: The vehicle applies the driving actions generated by the Bayesian policy network to the driving task and interacts with the driving environment to achieve emergency collision avoidance at high speeds in dangerous situations. Model training is performed under the guidance of the reward function. The expert dataset from Step 1 is added to the replay buffer with a decay ratio to optimize the model training effect.

[0091] Specifically, in using reinforcement learning algorithms to guide autonomous commercial vehicles (i.e., intelligent agents) to learn high-speed driving emergency collision avoidance tasks, the design of the reward function is crucial. In this invention, a total reward function is set to guide the training and learning of autonomous commercial vehicles (i.e., intelligent agents) by considering multiple indicators such as driving efficiency, driving safety, driving comfort, and emergency collision avoidance task completion.

[0092] Driving efficiency is set as the basis of the reward function:

[0093]

[0094] Among them, v k It is the vehicle's speed, v limit With v max These are the maximum speed limit for the road and the maximum speed a vehicle can reach.

[0095] Driving safety is configured as a constrained reward function as follows:

[0096]

[0097] Where ttc is the expected time of collision at the current vehicle speed, and d lat f is the lateral distance between vehicles. c v is a binary indicator of whether a vehicle collision has occurred. x It is the vehicle's current velocity along the axis of travel. ρ ttc ρ lat k c All of these are hyperparameters.

[0098] Driving comfort is configured as a constrained reward function as follows:

[0099]

[0100] Where, ρ thr With ρ ste These are the hyperparameters for the longitudinal acceleration term and the lateral acceleration term, respectively.

[0101] The incentive reward function for emergency collision avoidance mission completion is set as follows:

[0102]

[0103] in, Let c be the Euclidean distance between the autonomous commercial vehicle and the background vehicle. r This is a hyperparameter.

[0104] Based on this, the total reward function of the present invention is set as follows:

[0105]

[0106] During model training guided by the established reward function, to address the reward sparsity problem in the application scenario of this invention, the expert experience data collected in step 1 is arranged in a replay buffer with a priority of ε. This helps the autonomous commercial vehicle (i.e., the intelligent agent) to map its state to actions based on the demonstrations provided by the experts for initial exploration. ε is set as a gradually decaying proportional coefficient, gradually decreasing to zero as training progresses.

[0107] By using the reinforcement learning algorithm architecture provided in the above steps, a driving strategy for autonomous commercial vehicles that can achieve high-speed emergency collision avoidance can be obtained by training the model and selecting the optimal strategy through a computer.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, causing a series of operational steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that run on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0113] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A learning-based emergency collision avoidance control method for autonomous commercial vehicles, characterized in that, The method includes the following steps: Step 1: Collect driving data from human drivers, construct an expert experience dataset, and add the expert experience dataset to the replay buffer in the form of a decay ratio; Step 2: Construct a reinforcement learning network model based on the soft actor critic architecture. The reinforcement learning network model includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, and the value network and Q-value network evaluate the driving actions generated by the policy network. The policy network, value network, and Q-value network are trained using observation-action pairs in the replay buffer to update the corresponding network parameters. Step 3: Based on the reinforcement learning network model built in Step 2, construct a corresponding Bayesian policy network for the commercial vehicle driving task and control actions. Extend the policy network structure and update process from Step 2 to generate a joint control strategy adapted to emergency collision avoidance conditions. The node set of the Bayesian policy network is the steering δ... k throttle a k and braking k A strategy involving three actions; Step 4: Apply the driving actions generated by the Bayesian policy network to the vehicle's driving task, and train the model to generate the optimal policy by interacting with the driving environment and aiming at collision avoidance without collision at high speeds, under the guidance of the reward function. The Bayesian policy network outputs the driving action policy T = (t i ) i∈I Described as being controlled by acceleration policy t a Steering control strategy t δ and braking control strategy t br The probability π of the three sub-policy actions T The joint probability function is composed of the following relationships: π T (t δ ,t a ,t br )=π1(t a )π2(t δ |t a )π3(t br |t a ); The joint control strategy adopted by the Bayesian policy network is as follows: In the formula, A is the joint action distribution of policy π. It is each sub-strategy π i Action space, θ i These are the corresponding parameters; The value network uses the value function V(A) in the following formula. t ,s) Evaluate the performance of the joint control strategy: Wherein, π(A) t |s;θ) represents the action A taken in state s. t The probability, q(s,A) t ;ω) represents the action A taken in state s. t The expected return, where θ and ω are the corresponding network parameters; The gradient descent of the corresponding parameters is calculated using the following formula: The joint control strategy uses the following objective function J: V (π) is calculated to maximize the sub-strategy π. i The reward r(s) t A t ) and entropy sum: Where α is a hyperparameter used to control the weight of the entropy regularization term; In the evaluation process of soft policy iteration, the soft Q value is calculated iteratively starting from an arbitrary function Q. And repeatedly apply the modified Bellman backup operator. The soft value function of the updated sub-policy, the Bellman backup operator, is given by the following equation: In the improvement step of each sub-policy, the corresponding policy is updated for the state at each time step according to the following formula: in, It is the old strategy action value function. It is a normalization factor; In each sub-policy iteration, soft policy evaluation and soft policy improvement are performed alternately to ensure optimal maximum entropy convergence among sub-policy combinations.

2. The learning-based emergency collision avoidance control method for autonomous commercial vehicles according to claim 1, characterized in that, In step 1, the collected expert experience dataset is arranged into the replay buffer with a priority of ε, which helps autonomous commercial vehicles to map states to actions for initial exploration based on demonstrations provided by experts; where ε is set as a gradually decaying proportional coefficient that gradually decays to zero as training progresses.

3. The learning-based emergency collision avoidance control method for autonomous commercial vehicles according to claim 1, characterized in that, In step 2, the reinforcement learning network model includes a policy network, a value network, and two Q-value networks. The policy network, designed according to the dimension of state information, is used to generate continuous driving actions based on the current sensor input observations. The state space of the reinforcement learning network model is s = [θ]. k ,d k ,d a is ,v k ,w k ,n k ,f c ,op k In the formula, θ k d is the angle between the vehicle's direction and the center of the road. k d represents the distance between vehicles and the road edge within a 200-meter range returned by the sensor. axis The normalized distance between the vehicle and the road centerline; v k It includes three speed scalars: the vehicle speed along the direction of travel, the vehicle speed laterally along the axis of travel, and the vehicle speed along the Z-axis; w k It contains n scalars, representing the rotational speeds of the vehicle's n wheels, n k f is the engine's speed per minute. c It is a binary flag indicating a collision between a vehicle and other road users, op k Store information about the nearest vehicle to the current vehicle, including relative distance and angle; the action space is a = [δ]. k ,a k ,br k In the formula, δ k For steering actions, the range is [-1, +1], a k To accelerate the action, the range is [0,1], br k This is a braking action, with a range of [0,1].

4. The learning-based emergency collision avoidance control method for autonomous commercial vehicles according to claim 3, characterized in that, In step 2, the policy network, value network, and Q-value network are all constructed from a 5-layer neural network, namely a state input layer, two hidden layers containing 256 neurons each, a normalization layer, and a driving action output layer. The observation information acquired by the sensor is normalized and sequentially input into the state input layer. After feature extraction and processing through two hidden layers, it is activated by the ReLU function to obtain the tensor corresponding to the observation information. In the policy network, the tensor is activated by the linear function of the driving action output layer to obtain the final action space. In the value network and Q-value network, the tensor is activated by the linear function of the driving action output layer to obtain the final action evaluation.

5. The learning-based emergency collision avoidance control method for autonomous commercial vehicles according to claim 1, characterized in that, In step 2, the learning rate of the policy network is 0.0003, the learning rate of the Q-value network is 0.003, the influence weight of future action reward is 0.96, the soft update parameter of the target network is 0.005, the replay buffer size is 1,000,000, and the sampling batch size is 256.

6. The learning-based emergency collision avoidance control method for autonomous commercial vehicles according to claim 1, characterized in that, In step 4, the reward function is: in, For driving efficiency indicators, For driving safety indicators, For driving comfort indicators, As an indicator of the completion rate of emergency collision avoidance missions; In the formula, v k It is the vehicle's speed, v limit With v max These represent the maximum speed limit of the road and the maximum speed the vehicle can reach, respectively; ttc is the expected time of collision at the current speed, and d... lat f is the lateral distance between vehicles. c v is a binary indicator of whether a vehicle collision has occurred. x It is the current speed of the vehicle along the axis of travel; ρ ttc ρ lat k c All are hyperparameters; ρ thr With ρ ste These are the hyperparameters for the longitudinal acceleration term and the lateral acceleration term, respectively; Let c be the Euclidean distance between the autonomous commercial vehicle and the background vehicle. r This is a hyperparameter.

7. A learning-based emergency collision avoidance control system for autonomous commercial vehicles based on the method described in any one of claims 1-6, characterized in that, The system includes: The dataset building module is used to collect driving data from human drivers, build an expert experience dataset, and add the expert experience dataset to the replay buffer in the form of a decay ratio. The reinforcement learning network model construction module is used to construct a reinforcement learning network model based on the soft actor critic architecture. The reinforcement learning network model includes a policy network, a value network, and a Q-value network. The policy network generates continuous driving actions based on sensor observation data, and the value network and Q-value network evaluate the driving actions generated by the policy network. The policy network, value network, and Q-value network are trained using observation-action pairs in the replay buffer to update the corresponding network parameters. The joint control strategy module is used to construct corresponding Bayesian policy networks based on the built reinforcement learning network model, targeting the driving tasks and control actions of commercial vehicles. It extends the policy network structure and update process to generate joint control strategies adapted to emergency collision avoidance conditions. The node set of the Bayesian policy network is the steering δ... k throttle a k and braking k A strategy involving three actions; The optimal policy output module is used to apply the driving actions generated by the Bayesian policy network to the vehicle's driving tasks. By interacting with the driving environment, it aims to achieve collision avoidance without collisions at high speeds and trains the model under the guidance of the reward function to generate the optimal policy.

8. A storage medium, characterized in that, The storage medium stores a computer program that, when executed in a computer, causes the computer to perform the emergency collision avoidance control method as described in any one of claims 1-6.

Images