A Safety-Enhanced Deep Inverse Reinforcement Learning-Based Decision-Making and Planning Method for Autonomous Driving
By introducing deep neural networks and inverse reinforcement learning methods with positive and negative demonstrations into autonomous driving, the joint reward function is directly fitted, solving the problem of insufficient safety in traditional deep inverse reinforcement learning. This achieves a balance between high efficiency and human-likeness, and improves the decision robustness and safety of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2026-02-28
- Publication Date
- 2026-06-02
AI Technical Summary
Traditional deep inverse reinforcement learning algorithms suffer from insufficient safety, low learning efficiency, and lack of ability to model dangerous behaviors in autonomous driving. Furthermore, existing safety constraint mechanisms can only avoid collisions at the outcome level and fail to improve the safety and robustness of the strategy from the learning mechanism itself.
By employing a joint reward function based on deep neural networks and combining positive and negative demonstrations with an inverse reinforcement learning method, the joint reward network is trained to directly fit complex nonlinear driving behavior patterns by maximizing the likelihood probability of positive demonstrations and minimizing the probability of negative demonstrations, and safety constraints are embedded in the learning process.
It significantly improves the model's generalization ability and the modeling accuracy of multimodal driving behavior, achieving an organic unity of human-likeness and safety, and enhancing the robustness and safety of autonomous driving decisions.
Smart Images
Figure CN122126309A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, specifically to an autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning. Background Technology
[0002] With the rapid development of autonomous driving technology, intelligent decision-making algorithms such as reinforcement learning and inverse reinforcement learning are widely used in the behavioral decision-making and trajectory planning of autonomous vehicles. Inverse reinforcement learning can automatically learn implicit reward functions from expert demonstration data, thereby mimicking human driving behavior and achieving human-like driving decisions. However, traditional inverse reinforcement learning often uses linearly weighted reward functions, relies on manually defined feature terms, and struggles to accurately depict the driver's true decision-making intent in complex traffic scenarios, resulting in limited generalization ability. Deep inverse reinforcement learning, by introducing deep neural networks to fit nonlinear reward functions, improves the model's expressive power and human-likeness, but it still suffers from insufficient safety, low learning efficiency, and a lack of ability to model dangerous behaviors in actual autonomous driving decision-making.
[0003] Existing deep inverse reinforcement learning algorithms typically rely solely on positive demonstration data from experts for training, aiming to maximize the likelihood probability of positive samples and thus statistically mimic the behavioral distribution of human drivers. However, this one-way learning mechanism ignores the constraint information inherent in negative demonstrations, making the model unable to identify behaviors that should not occur. In scenarios with strong interaction or safety boundaries, it is prone to outputting decisions with high collision risk. Furthermore, the optimization process of deep inverse reinforcement learning requires iterative evaluation of the reward function in a high-dimensional continuous space, resulting in high computational complexity. Since the reward function is non-convex, it may contain multiple local extrema, making gradient-based optimization methods prone to getting trapped in local optima, leading to unstable learning results. To enhance the safety of decisions, some studies have attempted to add safety constraints or rule modules outside the learning model to filter output actions. However, such post-safety mechanisms only prevent collisions at the outcome level and fail to improve the safety and robustness of the policy from the learning mechanism itself. Summary of the Invention
[0004] The purpose of this invention is to provide an autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning. This method directly fits the joint reward function in autonomous driving decision-making through a deep neural network, overcoming the limitation of the expressive power of traditional linear reward functions. Furthermore, it proposes a deep inverse reinforcement learning method based on positive and negative demonstrations, enabling the learned reward function to possess both human-like driving characteristics and inherent safety.
[0005] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: an autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning, applied to a dual-vehicle interaction scenario, comprising the following steps: Receive the state parameters of the dual-vehicle interaction scenario, including the longitudinal position, speed and acceleration parameters of the vehicle and the interacting vehicle; Based on the scene state parameters, candidate joint trajectories are constructed, which include candidate trajectories of the autonomous vehicle and candidate trajectories of the interactive vehicle. The candidate trajectories of the autonomous vehicle and the interactive vehicle are determined by the acceleration sequences of the autonomous vehicle and the interactive vehicle, respectively. A joint reward network is constructed based on a deep neural network. The joint reward network takes scene state parameters and candidate joint trajectories as input and outputs the joint reward for the corresponding candidate joint trajectory. The joint reward network is trained using an inverse reinforcement learning method based on positive and negative demonstrations. Positive demonstrations are expert safe demonstration trajectories, while negative demonstrations are unsafe trajectories selected from candidate joint trajectories. The network parameters are iteratively updated by maximizing the likelihood of positive demonstrations and minimizing the probability of negative demonstrations until the joint reward network parameters converge. Based on the current state of the autonomous vehicle and the interactive vehicle, a set of candidate joint trajectories is generated by sampling. Each candidate joint trajectory in the set of candidate joint trajectories is input into the converged joint reward network to calculate the reward value of each candidate joint trajectory. The candidate joint trajectory with the highest reward value is selected as the planning result output.
[0006] Furthermore, the input to the joint reward network includes three parts: the scene state, the candidate trajectory of the autonomous vehicle, and the candidate trajectory of the interacting vehicle. In a dual-vehicle interaction scenario, the scene state includes the longitudinal position of the autonomous vehicle. ,speed and acceleration and the longitudinal position of the interactive vehicle ,speed and acceleration ; Given an initial state, the trajectories of the autonomous vehicle and the interactive vehicle are determined by the actions of the two vehicles, specifically the acceleration of the autonomous vehicle. and the acceleration of interactive vehicles Therefore, the input to the joint reward network is the scene state. , , , , , ] and the evaluated candidate joint action[ , ].
[0007] Furthermore, a joint reward network will be constructed, the details of which are as follows: Input layer: 8 neurons in total. Its input consists of a concatenation of the scene state vector and the joint action vector. Scene state vector: includes the initial longitudinal position, velocity, and acceleration of the main vehicle and the target vehicle; Candidate joint motion vector: includes the planned accelerations of the evaluated master vehicle and the target vehicle; Hidden layers: two layers, each with 30 neurons, using the tanh activation function to enhance the model's non-linear fitting ability; Output layer: 1 neuron, using the Sigmoid activation function to restrict the output value to the range (0,1), representing the normalized joint reward evaluation of the candidate joint action in a given scene state.
[0008] Furthermore, the joint reward network is trained using an inverse reinforcement learning method based on positive and negative demonstrations, as follows: (41) Initialize the parameters of the joint reward network; (42) Read the pre-collected expert demonstration dataset, which contains expert trajectories as positive demonstrations and collision trajectories selected from candidate joint trajectories as negative demonstrations. (43) Calculate the rewards for positive and negative demonstration trajectories using the current joint reward network; (44) Calculate the gradient of the updated parameters and update the parameters of the joint reward network using the gradient ascent algorithm; (45) Repeat steps (43) to (44) until the joint reward network parameters converge.
[0009] Furthermore, the candidate joint trajectory set is composed of pairwise pairings of candidate trajectories from the autonomous vehicle and candidate trajectories from the interacting vehicle. The corresponding candidate joint trajectory set is sampled as follows: The trajectories of the autonomous vehicle and the interacting vehicle are represented as a time series of vehicle states. Vehicle status Including the longitudinal position of the vehicle ,speed and acceleration The longitudinal dynamics model of the vehicle is then expressed as: Once the initial state of the vehicle is determined, the trajectory of the vehicle can be determined by the acceleration sequence within the planning field of view. Therefore, a candidate trajectory set is constructed by sampling candidate acceleration sequences. In addition, the negative demonstration trajectory is selected from the candidate joint trajectory set, and the trajectory where the minimum distance between the autonomous vehicle and the interactive vehicle is less than the safety threshold is selected.
[0010] Furthermore, the reward value of the candidate joint trajectory is calculated, including calculating the rewards for the positive and negative demonstration trajectories using the current joint reward network, as follows: According to the maximum entropy inverse reinforcement learning algorithm, the probability of a trajectory being selected is proportional to the natural exponent of its reward: In the formula, Represents the joint trajectory of the autonomous vehicle and the interactive vehicle. The probability of being selected; Represents the joint trajectory Joint rewards; The parameters represent the joint reward function; It is called the normalization function or partition function; Represents the set of all joint trajectories; In sampling-based inverse reinforcement learning methods, the partition function Approximated as: in, It is a joint trajectory The corresponding set of candidate trajectories; Therefore, in sampling-based inverse reinforcement learning, joint trajectory The probability of being selected is: In the formula, Represents the joint trajectory Joint rewards; This represents the parameters of the joint reward function.
[0011] Furthermore, an objective function is designed using the front and back demonstration trajectories for training the joint reward network, as follows: (71) The objective function used to train the joint neural network is designed as follows: in, This represents the human-like terms in the objective function; This represents the safety term in the objective function; Represents the weighting coefficient of the safety objective item; (72) For the human-like items, the parameters of the joint reward function need to be adjusted to maximize the likelihood of positive representation: in This represents a positive demonstration dataset; It is a set of joint trajectories in the positive demonstration dataset; Represents the joint trajectory The probability of being selected; Therefore, the human-like terms in the objective function are represented as: (73) In the safety section, the parameters of the reward function need to be adjusted to maximize the likelihood of not producing negative demonstrations: Therefore, the safety term in the objective function is expressed as: In the formula, This indicates the number of scenes in the training data; Indicates the first A dataset of all negative demonstrations in each scenario; Represents a negative demonstration dataset A set of joint trajectories; Indicates in the scene The sum of probabilities of all joint trajectories that will collide, i.e., the probability of a collision occurring; Representing a scene The probability that these negative demonstrations will not occur, i.e. the probability of no collision.
[0012] According to a second aspect of the present invention, the present invention provides an autonomous driving decision-making and planning system based on forward and reverse demonstration deep inverse reinforcement learning, for implementing the autonomous driving decision-making and planning method based on safety enhancement deep inverse reinforcement learning described in the first aspect, comprising: The receiving module is used to receive the state parameters of the dual-vehicle interaction scenario, including the longitudinal position, speed and acceleration parameters of the vehicle and the interacting vehicle. The candidate joint trajectory construction module is used to construct candidate joint trajectories based on scene state parameters. The candidate joint trajectory includes the candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle. The candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle are determined by the acceleration sequence of the autonomous vehicle and the acceleration sequence of the interactive vehicle, respectively. A joint reward network construction module is used to construct a joint reward network based on a deep neural network. The joint reward network takes scene state parameters and candidate joint trajectories as input and outputs the joint reward of the corresponding candidate joint trajectory. The model training module is used to train the joint reward network using an inverse reinforcement learning method based on positive and negative demonstrations. The positive demonstration is an expert safe demonstration trajectory, and the negative demonstration is an unsafe trajectory selected from the candidate joint trajectories. The network parameters are iteratively updated by maximizing the likelihood of the positive demonstration and minimizing the probability of the negative demonstration until the joint reward network parameters converge. The calculation module is used to sample and generate a set of candidate joint trajectories based on the current state of the autonomous vehicle and the interactive vehicle, input each candidate joint trajectory in the set of candidate joint trajectories into the converged joint reward network, and calculate the reward value of each candidate joint trajectory. The output module is used to select the candidate joint trajectory with the highest reward value as the planning result output.
[0013] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor, and when the processor loads and executes the computer program, it employs the inverse reinforcement learning autonomous driving decision-making and planning method based on positive and negative demonstrations described in the first aspect.
[0014] According to a fourth aspect of the present invention, the present invention provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to execute the inverse reinforcement learning decision-making and planning method for autonomous driving based on positive and negative demonstrations as described in the first aspect.
[0015] The beneficial effects of this invention are: 1. This invention constructs a deep neural network as a joint reward function, taking the scene state and candidate joint trajectories as input, and directly outputs the joint reward value of the trajectory. This deep network structure can automatically learn complex nonlinear driving behavior patterns from demonstration data, breaking through the inherent limitations of traditional inverse reinforcement learning that relies on manually designed features and linear weighted combinations. It can effectively characterize the driver's true decision-making intention in a highly interactive scenario in a high-dimensional continuous space without the need for predefined expert knowledge. It overcomes the limitations of traditional methods that rely on manually designed reward functions, and significantly enhances the model's generalization ability and modeling accuracy for multimodal driving behavior.
[0016] 2. This invention innovatively introduces negative demonstration data and designs the training objective function as a weighted combination of human-like and safety terms. The human-like terms ensure that the learned strategy remains highly human-like by maximizing the likelihood probability of positive expert demonstration trajectories. The safety terms enable the model to proactively identify and avoid potential dangerous behaviors during the learning phase by maximizing the probability of avoiding negative demonstrations (collision trajectories). This positive-negative demonstration collaborative learning mechanism embeds safety constraints into the learning process of the reward function, rather than externally correcting them afterward. This fundamentally improves the robustness and safety of autonomous driving decisions and achieves an organic unity of human-likeness and safety. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the method described in this invention; Figure 2 This is a flowchart illustrating the training process of the joint reward network in this invention. Detailed Implementation
[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0021] Example 1: Explanation of related terms: Frontal demonstration trajectory: refers to safe driving behavior data.
[0022] Negative demonstration trajectory: refers to dangerous driving behavior data, i.e., collision trajectory.
[0023] Expert demonstration dataset: Includes expert trajectories as positive demonstrations and collision trajectories as negative demonstrations.
[0024] Autonomous vehicle: Defines an autonomous vehicle as an autonomous vehicle.
[0025] Interactive vehicle: Defined as the target vehicle that interacts and passes with the vehicle.
[0026] An autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning consists of three stages: joint trajectory generation, joint reward evaluation, and optimal trajectory selection. Specifically, firstly, based on the current position, speed, acceleration, and environmental perception information of the autonomous vehicle, multiple sets of candidate joint trajectories for the autonomous vehicle and the interacting vehicle are generated; then, a trained and converged joint reward neural network evaluates each candidate joint trajectory and calculates the corresponding joint reward value, which can comprehensively characterize the human-like driving characteristics and interaction safety of the candidate joint trajectory; finally, the trajectory with the largest joint reward value is selected from the candidate joint trajectories as the prediction and planning output of the autonomous vehicle, which is used to guide the vehicle's decision-making and control execution.
[0027] First, the parameters such as weights and biases used to fit the joint reward network are initialized to ensure that the network has basic nonlinear fitting capabilities to characterize complex driving interaction behaviors.
[0028] Reference Figures 1-2This paper presents a decision-making and planning method for autonomous driving based on safety-enhancing deep inverse reinforcement learning, including the following steps: S1. Construct a joint reward network, taking the scene state and the candidate joint trajectory being evaluated as input, and outputting the joint reward for the corresponding candidate joint trajectory; Construct a joint reward network as follows: Input layer: 8 neurons in total, its input consists of two parts concatenated together: Scene state vector (6-dimensional): includes the initial longitudinal position, velocity, and acceleration of the main vehicle and the target vehicle; Candidate joint action vector (2D): includes the planned accelerations of the evaluated master vehicle and the target vehicle; Hidden layers: two layers, each with 30 neurons, using the tanh activation function to enhance the model's non-linear fitting ability; Output layer: 1 neuron, using the Sigmoid activation function to restrict the output value to the range (0,1), representing the normalized joint reward evaluation of the candidate joint action in a given scene state; The input to the joint reward network consists of three parts: the scene state, the candidate trajectory of the autonomous vehicle, and the candidate trajectory of the interacting vehicle. In a two-vehicle interaction scenario, the scene state includes the longitudinal position of the autonomous vehicle. ,speed and acceleration and the longitudinal position of the interactive vehicle ,speed and acceleration ; Given an initial state, the trajectories of the autonomous vehicle and the interactive vehicle are determined by the actions of the two vehicles, specifically the acceleration of the autonomous vehicle. and the acceleration of interactive vehicles Therefore, the input to the joint reward network is the scene state. , , , , , ] and the evaluated candidate joint action[ , ]; S2. Train the network using an inverse reinforcement learning method based on positive and negative demonstrations until the joint reward network parameters converge. (S21) Initialize the parameters of the joint reward network; (S22) Read the pre-collected expert demonstration dataset, which contains expert trajectories as positive demonstrations and collision trajectories selected from candidate joint trajectories as negative demonstrations. (S23) Calculate the rewards for the positive and negative demonstration trajectories using the current joint reward network; (S24) Calculate the gradient of the updated parameters and update the parameters of the joint reward network using the gradient ascent algorithm; (S25) Repeat steps (S23) to (S24) until the joint reward network parameters converge; Furthermore, an objective function is designed using the front and back demonstration trajectories for training the joint reward network, as follows: The objective function used to train the joint neural network is designed as follows: in, This represents the human-like terms in the objective function; This represents the safety term in the objective function; Represents the weighting coefficient of the safety objective item; The objective function further includes human-like terms and safety terms, as detailed below: In the human-like category, the parameters of the joint reward function need to be adjusted to maximize the likelihood of positive representation: in This represents a positive demonstration dataset; It is a set of joint trajectories in the positive demonstration dataset; Represents the joint trajectory The probability of being selected; Therefore, the human-like terms in the objective function can be represented as: In the safety section, the parameters of the reward function need to be adjusted to maximize the likelihood of not producing negative demonstrations: in, This indicates the number of scenes in the training data; Indicates the first A dataset of all negative demonstrations in each scenario; Represents a negative demonstration dataset A set of joint trajectories; Indicates in the scene The sum of probabilities of all joint trajectories that will collide is the probability of a collision. Representing a scene The probability that these negative demonstrations will not occur, i.e. the probability of no collision.
[0029] Therefore, the security term in the objective function can be expressed as: Furthermore, the parameters of the joint reward network are optimized using a mini-batch gradient ascent method. Specifically, within each training epoch, the expert positive demonstration dataset is first randomly divided into multiple batches of equal size; then, the objective function with respect to the network parameters is calculated based on the current batch data. The gradient is calculated; finally, the network parameters are updated using the gradient ascent algorithm. S3. Based on the states of the autonomous vehicle and the interactive vehicle, sample candidate joint trajectories, and calculate the reward for the candidate joint trajectories of the autonomous vehicle and the interactive vehicle using the converged joint reward function; The candidate joint trajectory set consists of pairwise pairings of candidate trajectories from the autonomous vehicle and candidate trajectories from the interactive vehicle. The corresponding candidate joint trajectory sets are sampled, and negative demonstration trajectories are selected from them, as detailed below: Based on the candidate joint trajectory sampled from the expert demonstration trajectory of each pair of human drivers, each pair of candidate joint trajectories and the corresponding expert demonstration trajectory have the same initial state; The trajectories of the autonomous vehicle and the interacting vehicle are represented as a time series of vehicle states. Vehicle status Including the longitudinal position of the vehicle ,speed and acceleration The longitudinal dynamics model of the vehicle is then expressed as: Once the initial state of the vehicle is determined, the trajectory of the vehicle can be determined by the acceleration sequence within the planning field of view. Therefore, a candidate trajectory set can be constructed by sampling candidate acceleration sequences.
[0030] In addition, negative demonstration trajectories are selected from this set. The core criterion for selection is safety. By calculating the minimum distance between the autonomous vehicle and the interactive vehicle in each joint trajectory, trajectories with a distance less than a preset safety threshold are identified as negative samples with collision risk and included in the negative demonstration dataset.
[0031] Furthermore, to quantify the probability of different trajectories being selected, this implementation uses a maximum entropy inverse reinforcement learning framework for modeling. To handle the vast trajectory space in practical computation, this embodiment employs a sampling-based approximation method, as follows: According to the maximum entropy inverse reinforcement learning algorithm, the probability of a trajectory being selected is proportional to the natural exponent of its reward: in, Represents the joint trajectory of the autonomous vehicle and the interactive vehicle. The probability of being selected; Represents the joint trajectory Joint rewards; The parameters represent the joint reward function; It is called the normalization function or partition function; Represents the set of all joint trajectories; In sampling-based inverse reinforcement learning methods, the partition function It can be approximated as: in, It is a joint trajectory The corresponding set of candidate trajectories; Therefore, in sampling-based inverse reinforcement learning, joint trajectory The probability of being selected is: In the formula, Represents the joint trajectory Joint rewards; The parameters represent the joint reward function; S4. Select the candidate joint trajectory with the largest reward value as the planning result output.
[0032] In summary, this invention proposes a deep inverse reinforcement learning ensemble decision-making and planning framework based on positive and negative demonstrations. By combining a joint reward network with a positive and negative demonstration learning mechanism, it overcomes the limitations of traditional methods in reward function design and security assurance. Furthermore, by introducing a deep neural network to construct the joint reward function, this invention can automatically learn complex driving behavior patterns from demonstration data, avoiding the insufficient expressive power caused by relying on manually designed features and linear combinations in traditional methods. This significantly improves the accuracy of the reward function and the scene generalization ability. At the same time, this invention can accurately learn multimodal reward functions in complex interactive scenarios, significantly improving trajectory prediction accuracy and planning safety, while maintaining human-like characteristics of driving behavior. This provides core technical support for the safe implementation of autonomous driving in highly interactive scenarios.
[0033] Example 2: This invention provides a deep inverse reinforcement learning-based autonomous driving decision-making and planning system based on positive and negative demonstrations, used to implement the safety-enhancing deep inverse reinforcement learning-based autonomous driving decision-making and planning method described in the first aspect, comprising: The receiving module is used to receive the state parameters of the dual-vehicle interaction scenario, including the longitudinal position, speed and acceleration parameters of the vehicle and the interacting vehicle. The candidate joint trajectory construction module is used to construct candidate joint trajectories based on scene state parameters. The candidate joint trajectory includes the candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle. The candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle are determined by the acceleration sequence of the autonomous vehicle and the acceleration sequence of the interactive vehicle, respectively. A joint reward network construction module is used to construct a joint reward network based on a deep neural network. The joint reward network takes scene state parameters and candidate joint trajectories as input and outputs the joint reward of the corresponding candidate joint trajectory. The model training module is used to train the joint reward network using an inverse reinforcement learning method based on positive and negative demonstrations. The positive demonstration is an expert safe demonstration trajectory, and the negative demonstration is an unsafe trajectory selected from the candidate joint trajectories. The network parameters are iteratively updated by maximizing the likelihood of the positive demonstration and minimizing the probability of the negative demonstration until the joint reward network parameters converge. The calculation module is used to sample and generate a set of candidate joint trajectories based on the current state of the autonomous vehicle and the interactive vehicle, input each candidate joint trajectory in the set of candidate joint trajectories into the converged joint reward network, and calculate the reward value of each candidate joint trajectory. The output module is used to select the candidate joint trajectory with the highest reward value as the planning result output.
[0034] Example 3: The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it adopts the autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning described in Embodiment 1.
[0035] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.
[0036] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.
[0037] Example 4: The present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning described in Embodiment 1.
[0038] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.
[0039] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. An autonomous driving decision-making and planning method based on safety-enhanced deep inverse reinforcement learning, applied to dual-vehicle interaction scenarios, characterized in that... Specifically, the following steps are included: Receive the state parameters of the dual-vehicle interaction scenario, including the longitudinal position, speed and acceleration parameters of the vehicle and the interacting vehicle; Based on the scene state parameters, candidate joint trajectories are constructed, which include candidate trajectories of the autonomous vehicle and candidate trajectories of the interactive vehicle. The candidate trajectories of the autonomous vehicle and the interactive vehicle are determined by the acceleration sequences of the autonomous vehicle and the interactive vehicle, respectively. A joint reward network is constructed based on a deep neural network. The joint reward network takes scene state parameters and candidate joint trajectories as input and outputs the joint reward for the corresponding candidate joint trajectory. The joint reward network is trained using an inverse reinforcement learning method based on positive and negative demonstrations. Positive demonstrations are expert safe demonstration trajectories, while negative demonstrations are unsafe trajectories selected from candidate joint trajectories. The network parameters are iteratively updated by maximizing the likelihood of positive demonstrations and minimizing the probability of negative demonstrations until the joint reward network parameters converge. Based on the current state of the autonomous vehicle and the interactive vehicle, a set of candidate joint trajectories is generated by sampling. Each candidate joint trajectory in the set of candidate joint trajectories is input into the converged joint reward network to calculate the reward value of each candidate joint trajectory. The candidate joint trajectory with the highest reward value is selected as the planning result output.
2. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 1, characterized in that: The input to the joint reward network consists of three parts: the scene state, the candidate trajectory of the autonomous vehicle, and the candidate trajectory of the interacting vehicle. In a two-vehicle interaction scenario, the scene state includes the longitudinal position of the autonomous vehicle. ,speed and acceleration and the longitudinal position of the interactive vehicle ,speed and acceleration ; Given an initial state, the trajectories of the autonomous vehicle and the interactive vehicle are determined by the actions of the two vehicles, specifically the acceleration of the autonomous vehicle. and the acceleration of interactive vehicles Therefore, the input to the joint reward network is the scene state. , , , , , ] and the evaluated candidate joint action[ , ].
3. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 2, characterized in that: The joint reward network is constructed as follows: Input layer: 8 neurons in total. Its input consists of a concatenation of the scene state vector and the joint action vector. Scene state vector: includes the initial longitudinal position, velocity, and acceleration of the main vehicle and the target vehicle; Candidate joint motion vector: includes the planned accelerations of the evaluated master vehicle and the target vehicle; Hidden layers: two layers, each with 30 neurons, using the tanh activation function to enhance the model's non-linear fitting ability; Output layer: 1 neuron, using the Sigmoid activation function to restrict the output value to the range (0,1), representing the normalized joint reward evaluation of the candidate joint action in a given scene state.
4. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 3, characterized in that: The joint reward network is trained using an inverse reinforcement learning method based on positive and negative demonstrations, as follows: (41) Initialize the parameters of the joint reward network; (42) Read the pre-collected expert demonstration dataset, which contains expert trajectories as positive demonstrations and collision trajectories selected from candidate joint trajectories as negative demonstrations. (43) Calculate the rewards for positive and negative demonstration trajectories using the current joint reward network; (44) Calculate the gradient of the updated parameters and update the parameters of the joint reward network using the gradient ascent algorithm; (45) Repeat steps (43) to (44) until the joint reward network parameters converge.
5. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 4, characterized in that: The candidate joint trajectory set consists of pairwise pairings of candidate trajectories from the autonomous vehicle and candidate trajectories from the interacting vehicle. The corresponding candidate joint trajectory set is sampled as follows: The trajectories of the autonomous vehicle and the interacting vehicle are represented as a time series of vehicle states. Vehicle status Including the longitudinal position of the vehicle ,speed and acceleration The longitudinal dynamics model of the vehicle is then expressed as: Once the initial state of the vehicle is determined, the trajectory of the vehicle can be determined by the acceleration sequence within the planning field of view. Therefore, a candidate trajectory set is constructed by sampling candidate acceleration sequences. In addition, the negative demonstration trajectory is selected from the candidate joint trajectory set, and the trajectory where the minimum distance between the autonomous vehicle and the interactive vehicle is less than the safety threshold is selected.
6. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 5, characterized in that: The reward value for the candidate joint trajectory is calculated, including calculating the rewards for the positive and negative demonstration trajectories using the current joint reward network, as follows: According to the maximum entropy inverse reinforcement learning algorithm, the probability of a trajectory being selected is proportional to the natural exponent of its reward: In the formula, Represents the joint trajectory of the autonomous vehicle and the interactive vehicle. The probability of being selected; Represents the joint trajectory Joint rewards; The parameters represent the joint reward function; It is called the normalization function or partition function; Represents the set of all joint trajectories; In sampling-based inverse reinforcement learning methods, the partition function Approximated as: in, It is a joint trajectory The corresponding set of candidate trajectories; Therefore, in sampling-based inverse reinforcement learning, joint trajectory The probability of being selected is: In the formula, Represents the joint trajectory Joint rewards; This represents the parameters of the joint reward function.
7. The autonomous driving decision-making and planning method based on safety-enhancing deep inverse reinforcement learning according to claim 6, characterized in that: An objective function is designed using the front and back demonstration trajectories for training the joint reward network, as follows: (71) The objective function used to train the joint neural network is designed as follows: in, This represents the human-like terms in the objective function; This represents the safety term in the objective function; Represents the weighting coefficient of the safety objective item; (72) For the human-like items, the parameters of the joint reward function need to be adjusted to maximize the likelihood of positive representation: in This represents a positive demonstration dataset; It is a set of joint trajectories in the positive demonstration dataset; Represents the joint trajectory The probability of being selected; Therefore, the human-like terms in the objective function are represented as: (73) In the safety section, the parameters of the reward function need to be adjusted to maximize the likelihood of not producing negative demonstrations: Therefore, the safety term in the objective function is expressed as: In the formula, This indicates the number of scenes in the training data; Indicates the first A dataset of all negative demonstrations in each scenario; Represents a negative demonstration dataset A set of joint trajectories; Indicates in the scene The sum of probabilities of all joint trajectories that will collide, i.e., the probability of a collision occurring; Representing a scene The probability that these negative demonstrations will not occur, i.e. the probability of no collision.
8. A deep inverse reinforcement learning-based autonomous driving decision-making and planning system based on positive and negative demonstrations, used to implement any one of the autonomous driving decision-making and planning methods based on safety-enhancing deep inverse reinforcement learning according to claims 1 to 7, characterized in that, include: The receiving module is used to receive the state parameters of the dual-vehicle interaction scenario, including the longitudinal position, speed and acceleration parameters of the vehicle and the interacting vehicle. The candidate joint trajectory construction module is used to construct candidate joint trajectories based on scene state parameters. The candidate joint trajectory includes the candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle. The candidate trajectory of the autonomous vehicle and the candidate trajectory of the interactive vehicle are determined by the acceleration sequence of the autonomous vehicle and the acceleration sequence of the interactive vehicle, respectively. A joint reward network construction module is used to construct a joint reward network based on a deep neural network. The joint reward network takes scene state parameters and candidate joint trajectories as input and outputs the joint reward of the corresponding candidate joint trajectory. The model training module is used to train the joint reward network using an inverse reinforcement learning method based on positive and negative demonstrations. The positive demonstration is an expert safe demonstration trajectory, and the negative demonstration is an unsafe trajectory selected from the candidate joint trajectories. The network parameters are iteratively updated by maximizing the likelihood of the positive demonstration and minimizing the probability of the negative demonstration until the joint reward network parameters converge. The calculation module is used to sample and generate a set of candidate joint trajectories based on the current state of the autonomous vehicle and the interactive vehicle, input each candidate joint trajectory in the set of candidate joint trajectories into the converged joint reward network, and calculate the reward value of each candidate joint trajectory. The output module is used to select the candidate joint trajectory with the highest reward value as the planning result output.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the inverse reinforcement learning autonomous driving decision-making and planning method based on positive and negative demonstrations as described in any one of claims 1 to 7.
10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the inverse reinforcement learning autonomous driving decision-making and planning method based on positive and negative demonstrations as described in any one of claims 1 to 7.