Multi-agent driving learning method based on teacher-student normal form combined training mechanism

Abstract

The invention discloses a multi-agent driving learning method based on a teacher-student normal form joint training mechanism. The multi-agent driving learning method comprises the following steps: step 1, constructing a human expert driving data set D in an observation-action pair form; 2, modeling human driving behaviors into a graph structure Gt, extracting inter-vehicle interaction characteristics from the graph structure Gt by a GAT network, fusing the inter-vehicle interaction characteristics with a characteristic matrix of a vehicle to obtain H't, processing the H't by Graphformer to output normal distribution of expected actions, and updating a network parameter theta epsilon of a teacher agent through a minimum loss function L (theta epsilon); and step 3, embedding the teacher intelligent agent established in the step 2 into a multi-student intelligent agent reinforcement learning system, and adopting a two-stage training mode of a teacher guiding learning stage and a student intelligent agent free exploration stage, according to the invention, a teacher-student normal form joint training mechanism is introduced, and the advantages of human driving expert knowledge and neural network strategy learning are effectively fused.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous driving technology, and in particular to a multi-agent driving learning method based on a joint training mechanism of a teacher-student paradigm. Background Technology

[0002] In multi-agent autonomous driving simulation training, background vehicle behavior modeling is one of the core technologies for achieving high-fidelity traffic interaction. Current mainstream methods are mostly based on multi-agent reinforcement learning (MARL), which learns optimal behavioral policies through interactions between various agents to simulate real-world traffic systems where cooperation and competition coexist.

[0003] Despite the initial progress made by MARL in simulating multi-vehicle interactions, it still faces the following core technical bottlenecks: 1) Low training efficiency and difficulty in cold start. MARL relies on trial-and-error policy learning. In the early stages of training, student agents lack effective experience, often generating a large number of invalid or even dangerous samples (such as collisions, lane departures, etc.), resulting in low sample efficiency and slow training convergence; 2) Policies are prone to getting trapped in local optima, making it difficult to guarantee safety. To avoid extreme dangerous behaviors, traditional methods often set strong safety constraints (such as immediate termination) in the training process. While this improves safety, it also severely limits the agent's exploration ability, making it impossible to learn recovery strategies to cope with dangers, ultimately affecting the generalization performance of the policy; 3) Limited utilization of expert experience and inability to effectively integrate human knowledge. Although existing work has attempted to introduce behavioral cloning or expert demonstrations to improve training effects, most of them ignore the "interactivity" and "temporal nature" of the traffic environment, and cannot simulate the dynamic decision-making patterns of human drivers in multi-vehicle scenarios.

[0004] This invention addresses the problems of low training efficiency, difficulty in ensuring policy safety, and limited utilization of expert knowledge in existing multi-agent reinforcement learning in autonomous driving scenarios. It proposes a multi-agent driving learning method based on a joint training mechanism of a teacher-student paradigm that integrates human driving expert knowledge and has policy guidance and safety protection functions. Summary of the Invention

[0005] The purpose of this invention is to address the technical deficiencies in the existing technology by providing a multi-agent driving learning method based on a joint training mechanism of a teacher-student paradigm.

[0006] The technical solution adopted to achieve the purpose of this invention is:

[0007] A multi-agent driving learning method based on a teacher-student paradigm joint training mechanism includes the following steps:

[0008] Step 1, construct a human expert driving dataset D in the form of observation-action pairs;

[0009] Step 2: Construct a teacher agent using a graph attention fusion time-series Graphformer, which is obtained by combining a GAT network with a Transformer: modeling human driving behavior as a graph structure G. t Using GAT networks from the graph structure G t Extracting inter-vehicle interaction features Inter-vehicle interaction features and the feature matrix of its own vehicle The fusion yields H′ t Graphformer processes H′ t The output follows a normal distribution of the expected action, by minimizing the loss function L(θ). ε Update the network parameters θ of the teacher agent. ε ;

[0010] Step 3: Design of the "Teacher-Student Paradigm" Joint Training Mechanism: The teacher agent established in Step 2 is embedded into a multi-student agent reinforcement learning system. The training method of the multi-student agent reinforcement learning system includes two stages: a teacher-guided learning stage and a student agent free exploration stage. In different stages, the student agents execute different strategies. Obtain the optimal network parameters for the student agent.

[0011] During the teacher-guided learning phase, teacher strategies ε(·∣o) developed by human driving experts are utilized. i,t Interact with the environment to generate a demonstration trajectory τ in each state. i Utilizing each τ i Network parameters θ for training student agents i ;

[0012] During the independent exploration phase, a safety protection cost function C is introduced. t The optimal network parameters for the student agent and the Lagrange multiplier λ Apply constraints and use the gradient of the objective function. Update the optimal network parameters of the student agent.

[0013] In the above technical solution, the dataset D in the form of observation-action pairs in step 1 is represented as:

[0014]

[0015] In the formula, o ego,t and a ego,tLet N represent the observations and actions of the human driver at time t, and N be the total number of observations and actions collected.

[0016] In the above technical solution, the action a of the human driver at time t is... ego,t From steering action and vertical movement composition;

[0017] The human driver's observation ego,t Including the status of the vehicle itself Information on other vehicles Represented as:

[0018]

[0019] In the formula, d l ,d r p represents the distance from the vehicle to the left and right road boundaries. x ,p y v represents the lateral and longitudinal deviations between the vehicle and the target point. ego For the speed of one's own vehicle, For the deviation between the vehicle's own heading and the reference heading, The rate of change of the vehicle's heading. These represent the relative lateral and longitudinal distances between the nth vehicle and its own vehicle, respectively. These are the relative lateral and longitudinal velocities of the nth vehicle and its own vehicle, respectively. Let k be the heading of the nth vehicle. t -1 represents the number of vehicles in the surrounding area.

[0020] In the above technical solution, the loss function L(θ) of the teacher agent in step 2 ε The formula for calculating ) is:

[0021]

[0022] In the formula, θ ε Here are the network parameters for the teacher agent's strategy, where N is the total number of time intervals, and a... t Let σ(θ) be the action of the teacher agent at time t. ε (o ego,t )) 2 The variance of the action distribution output by the teacher agent, μ(θ) ε (o ego,t ) represents the mean of the action distribution output by the teacher agent.

[0023] In the above technical solution, the graph structure G mentioned in step 2 t Represented as:

[0024] Gt =(N t E t )

[0025] In the formula, N t The node feature matrix, E t It is an adjacency matrix. in,

[0026] The inter-vehicle interaction features The extraction process is as follows:

[0027]

[0028] In the formula, G t For graph structures, GATConv uses GAT networks to extract data from the graph structure G. t Feature extraction operations.

[0029] In the above technical solution, the student agent in step 3 executes strategies at different stages. Represented as:

[0030]

[0031] In the formula, I E This indicates the teacher-guided learning stage; ε(·∣o i,t ) indicates teacher strategies, o i,t Let be the observation of the i-th student agent at time t; Let be the student agent strategy formed by the i-th student agent during the autonomous exploration phase.

[0032] In the above technical solution, the optimal network parameters of the student agent in step 3 are... Represented as:

[0033]

[0034] In the formula, r i,t =R i (s i,t ,a i,t R represents the reward for each student agent at time t. i (·) is the reward function for the i-th student agent, s i,t and a i,t Let be the state and action of the i-th student agent at time t, respectively, and γ be the discount factor. Let E represent the strategies of student agents at different stages, and E be the expected value. and These are the corresponding times when the i-th student agent enters the environment and when the environment terminates, respectively.

[0035] In the above technical solution, the optimal network parameters for the student agent during the free exploration phase in step 3 are... Represented as:

[0036]

[0037] In the formula, r i,t =R i (s i,t ,a i,t R represents the reward for each student agent at time t. i (·) is the reward function for the i-th student agent, s i,t and a i,t Let be the state and action of the i-th student agent at time t, respectively, and γ be the discount factor. Let E be the policy of the i-th student agent, and E be the expected value. and C represents the corresponding times when the i-th student agent enters and exits the environment. t For the safety protection cost function, C thd The protection threshold is set by humans, and λ is the Lagrange multiplier.

[0038] In the above technical solution, the aforementioned is the security protection cost function C. t The calculation formula is:

[0039]

[0040] In the formula, a i,t For the action of the i-th student agent, o i,t For the observation of the i-th student agent, A ξ A set of safe actions for teachers, ξ t Let ε be the time-varying confidence boundary, ε be the teacher agent's policy, and a be the policy of the teacher agent. i,t -μ indicates the required level of intervention.

[0041] In the above technical solution, the objective function J of the student agent in step 3 during the free exploration phase... i (θ i ) is represented as:

[0042]

[0043] In the formula, The goal of strategy optimization for student agents is... Minimize the target for security protection;

[0044] The goal of minimizing security protection Defined as:

[0045]

[0046] In the formula, a i,t For the action of the i-th student agent, o i,t Let B be the observation of the i-th student agent, and let B be the replay buffer containing the experience of the teacher agent. Strategies for student intelligent agents C is the penalty value. thd Protection thresholds set manually;

[0047] The update rule for the Lagrange multiplier λ is defined as follows:

[0048]

[0049] In the formula, When δ≤0, the safety constraints are satisfied, and the Lagrange multiplier λ is zero; when δ>0, the safety constraints are violated, and a proportional-integral-derivative control mechanism is used to update λ, and the term K... p δ provides an immediate proportional response to a current constraint violation, item Accumulate past violations for correction, item The rate of change of δ is considered to adapt the model to potential future changes.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. This invention constructs a teacher agent, Graphformer, which integrates a graph attention network (GAT) and a Transformer structure. It is the first to realize graph-temporal joint modeling of multi-vehicle interaction behavior, which improves the model's ability to express human driving experience and generalize. It can effectively capture the temporal features and driving intentions in complex traffic interactions, and provide more human-like policy guidance for multi-agent systems.

[0052] 2. Improve learning efficiency and reduce training risks: This invention introduces the demonstration behavior of a teacher agent, guiding the student agent to imitate and learn during the guided learning phase. By imitating expert behavior, it replaces the traditional trial-and-error exploration, significantly increasing the proportion of positive samples, reducing sample waste in the cold start phase, and accelerating policy convergence. This effectively shortens policy convergence time and avoids the high risk and low efficiency problems caused by extensive trial and error in traditional reinforcement learning.

[0053] 3. Achieving a Balance Between Dynamic Safety Protection and Strategic Autonomy: This invention designs a "strategy guidance and safety protection" mechanism based on a teacher-agent agent. It proposes a safety protection mechanism under soft constraints, combining teacher confidence boundary judgment with security protection cost assessment. This allows the student agent to approach boundaries without overstepping them during autonomous exploration, gradually shifting from teacher dependence to autonomous decision-making while ensuring driving safety. Simultaneously, it proactively intervenes and provides corrective actions before detecting potentially dangerous behaviors, effectively avoiding the generation of extreme samples and ensuring the safety of the training process. Furthermore, it prevents the student agent from becoming dependent on the teacher through punitive constraints, thus balancing safety and autonomous exploration capabilities.

[0054] 4. This invention, while maintaining the optimization capabilities of reinforcement learning strategies, introduces a joint training mechanism based on a teacher-student paradigm, effectively integrating the advantages of human driving expert knowledge and neural network strategy learning. On the one hand, through the behavioral demonstration and imitation learning of the teacher agent, the student agent can quickly acquire feasible strategies in the early stages of training, significantly reducing training instability and safety risks, and improving strategy convergence speed. On the other hand, after entering the autonomous exploration stage, the system achieves real-time monitoring and intervention of the agent's exploration behavior through a soft-constraint safety protection mechanism. By setting confidence boundaries and safety protection cost functions, the system guides the student agent to avoid high-risk actions while maintaining its exploration capabilities, thus balancing the initiative of strategy learning with the controllability of behavior. Attached Figure Description

[0055] Figure 1 The diagram shows the multi-agent reinforcement learning training framework based on the teacher agent Graphformer of this invention.

[0056] Figure 2 The image shows the teacher-guided learning framework of this invention.

[0057] Figure 3 The diagram shown is a schematic of the security protection mechanism of this invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0059] Reference Figure 1 A multi-agent driving learning method based on a teacher-student paradigm joint training mechanism includes the following steps:

[0060] Step 1, constructing a human expert driving dataset: using the HIL simulation system to utilize the real operation of humans in the virtual driving environment, the human driver is introduced into the virtual simulation environment to operate. The human driver controls the vehicle operation in the simulation environment through real control devices (such as steering wheel, accelerator, brake pedal, etc.). The following contents are collected in real time through the human-computer interaction simulation platform (such as MetaDrive): (1) Driving operation data: including lateral control (steering wheel angle) and longitudinal control (accelerator / brake); (2) Environmental state perception data: including the self-vehicle state, the relative state of surrounding vehicles and road geometry information; (3) Time series information: fully record the state, action and time of each step to support time series modeling.

[0061] To improve the stability and generalization ability of the model training, the original dataset was cleaned and standardized, including: removing missing and outlier values; normalizing continuous variables (such as speed, relative distance, etc.) to a uniform range based on their maximum and minimum values; truncating or zero-padding the lists of surrounding vehicles of different lengths; and serializing and storing the dataset D in the form of observation-action pairs.

[0062]

[0063] In the formula, o ego,t For the human driver's observation at time t, a ego,t Let D represent the actions of a human driver at time t, and N be the total number of observations and actions collected. The dataset D is divided into three subsets: training set (70%), validation set (15%), and test set (15%).

[0064] The actions of the human driver at time t, a ego,t From steering action and vertical movement composition.

[0065] The human driver's observation at time t ego,t Represented by target-level perception data, including observations of the vehicle itself. Observation of other vehicles Represented as:

[0066]

[0067] In the formula, d l ,d r p represents the distance from the vehicle to the left and right road boundaries. x ,p y v represents the lateral and longitudinal deviations between the vehicle and the target point. ego Let Δ be the speed of the vehicle and Δ be the heading of the vehicle. For reference heading deviation, The rate of change of heading, These represent the relative lateral and longitudinal distances between the nth vehicle and its own vehicle, respectively. These are the relative lateral and longitudinal velocities of the nth vehicle and its own vehicle, respectively. Let k be the heading of the nth vehicle. t -1 represents the number of vehicles in the surrounding area.

[0068] Step 2, Construction of the Teacher Agent with Graph Attention Fusion Time Series: Modeling human driving behavior as a graph structure G t Using GAT networks to extract inter-vehicle interaction features Inter-vehicle interaction features and the feature matrix of its own vehicle The fusion yields H′ t By combining the normal distribution of the expected actions output by the Transformer, a teacher agent with human-style decision-making ability based on Graphformer (a combination of GAT and Transformer) is constructed. This is achieved by minimizing the loss function L(θ). ε Update the network parameters θ of the teacher agent. ε And use the training and test sets from step 1 to test the network parameters θ of the teacher agent. ε Training and testing are performed, using a validation set to evaluate the network parameters θ of the teacher agent. ε Verification is required;

[0069] To construct a teacher agent with graph attention fusion time series, it is first necessary to clarify the graph structure. In order to model the interaction behavior in a multi-vehicle scenario, the scene at time t is abstracted into a graph structure G. t =(N t E t ), where N t The node feature matrix, The node's attributes include relative vehicle speed and relative position, and are normalized. t It is an adjacency matrix. It reflects the mutual influence between vehicles and contains a series of weighted edges, where From node To the node The edge is only It exists at times, among which the larger ones The value indicates a strong influence between vehicles.

[0070] Distance, as an intuitive and direct physical metric, is used to define the weights of edges, effectively capturing and representing the interactions between vehicles in traffic scenarios. elements The calculation formula is:

[0071]

[0072] By will be from the node To the node Normalize the distance by dividing the edge by the maximum distance in the graph, ensuring

[0073] Using GAT networks from graph structure G t Extract hidden features The process is as follows:

[0074]

[0075] In the formula, To obtain the original graph structure G at time t t Extracted hidden features. Used for graph structure G. t The GATConv operation is represented as:

[0076]

[0077] In the formula, H l Denotes the node feature matrix of the l-th layer; σ is the non-linear activation function; A ij The values ​​from the adjacency matrix indicate the importance or weight of the connection between nodes i and j; set W represents the neighbors of node i. l H is the weight matrix of the l-th layer, used to transform the node feature matrix H from the (l-1)-th layer. l-1 ;α ij The attention coefficient is calculated using the following formula:

[0078]

[0079] In the formula, a l h is a learnable weight vector; LeakyReLU is the activation function used in the attention mechanism; i h j and h k These are the feature vectors of nodes i, j, and k, respectively; α ij The normalized attention coefficients indicate the importance of node j's features to node i; A ik The value is from the adjacency matrix, representing the connection weight between nodes i and k.

[0080] The vehicle's own state vector obtained in step 1 is processed through a series of linear layers. Processing to generate feature matrix feature matrix Features based on GAT Cascaded to form a unified fusion feature H′ t .

[0081] Graphformer processes the fused feature H′ t By adding positional encoding to the input, temporal dependencies are further captured, ensuring that the order of events is preserved for correct sequence modeling. Graphformer computes attention weights Attention(Q,K,V) based on the query (Q), key (K), and value (V) matrix. The attention weights Attention(Q,K,V) are calculated as follows:

[0082]

[0083] In the formula, d k denoted as the dimension of the key matrix.

[0084] The output of the attention mechanism is then passed through a multi-layer encoder that refines the feature representation using a feedforward network and normalization techniques. The final output of the Graphformer is the predicted action distribution, denoted by the mean μ. t (a) and variance σ t (a) 2 This means that they define the normal distribution:

[0085] N(μ t (a),σ t (a) 2 )

[0086] The above components together constitute the teacher agent, and the parameter θ of the teacher agent... ε The training updates are used to minimize the loss function:

[0087]

[0088] Using the training and test sets from step 1, the network parameters θ of the teacher agent are evaluated. ε Training and testing are performed, using a validation set to evaluate the network parameters θ of the teacher agent. ε Verification is required.

[0089] Step 3: Design of the "Teacher-Student Paradigm" Joint Training Mechanism: Embed the teacher agent established in Step 2 into the multi-student agent reinforcement learning system, and adopt a two-stage training approach: a teacher-guided learning stage and a student agent free exploration stage, implementing different strategies at different stages. Obtain the optimal network parameters for the student agent.

[0090] Reference Figure 2In the teacher-guided learning phase, a Centralized Training with Decentralized Execution (CTDE) paradigm is adopted, introducing an imitation learning mechanism based on the teacher agent's demonstrated behavior. This mechanism utilizes teacher strategies ε(·∣o) constructed by human driving experts. i,t Interact with the environment to generate a demonstration trajectory τ in each state. i The demonstration trajectory τ in each state i Network parameters θ for training student agents i This approach enables student agents to learn by observing behaviors demonstrated by the teacher. During training, student agents share additional information to address non-stationarity and partial observability, thereby achieving effective coordination. After training, each student agent receives a copy of the Actor, allowing for distributed execution. At this stage, shared information is discarded, and each agent operates independently while still benefiting from centralized training.

[0091] As students' intelligence agents develop, they enter the stage of autonomous exploration and ultimately form student strategies. As the final decision-making strategy, a security protection cost function C is introduced. t The optimal network parameters for the student agent and the Lagrange multiplier λ Apply constraints and use the gradient of the objective function. Update the optimal network parameters of the student agent. Specifically, in the simulated driving environment, each vehicle is a student agent i, which undergoes reinforcement learning training using a teacher-student paradigm joint training mechanism, forming a multi-agent learning system. Each student agent i can essentially be modeled as an optimal network parameter. The reinforcement learning problem, whose learning objective is to maximize the long-term cumulative reward in a traffic environment, can be defined as:

[0092]

[0093] In the formula, r i,t =R i (s i,t ,a i,t R represents the reward for each student agent i at time t. i (·) is the reward function for the i-th student agent, s i,t and a i,t Let represent the state and action of the i-th student agent at time t; γ is the discount factor. The policy of the i-th student agent is given by the parameters θ of the i-th student agent. i Decision; E is the expected value; and These are the corresponding times when the i-th student agent enters the environment and when the environment terminates, respectively; The optimal parameters for the student agent network are those that maximize the expected return.

[0094] After introducing the "teacher-student paradigm" joint training mechanism, the optimal network parameters for each student agent i are... The learning problem was redefined as a phased optimization problem:

[0095]

[0096] In the formula, Let represent the different strategies executed by the i-th student agent in the two training phases (teacher-guided learning phase and self-directed exploration phase). The strategies followed in each phase are as follows:

[0097]

[0098] In the formula, I E This indicates the teacher-guided learning stage, expressed as a percentage of the total training steps set manually; ε(·∣o i,t ) indicates teacher strategies, Strategies for student intelligent agents.

[0099] The self-training method for student agents follows an Actor-Critic architecture. In this architecture, a centralized Critic evaluates the Actor's performance during training. In the centralized Critic setting, the value function for each student agent i is expressed as... The corresponding advantage function is represented as A i,t =A i (o t ,a i,t ,a -,i,t ) = r i,t +γV i (o i,t+1 )-V i (o i,t ), where a -,i,t The behavior of other student agents is represented. The objective function J of the i-th student agent is calculated using the policy gradient method. i (θ i The gradient of ) is:

[0100]

[0101] As a standard practice in the Independent Nearest Neighbor Policy Optimization (IPPO) algorithm, the importance sampling factor ρ is pruned to mitigate the distribution shift that occurs after several training epochs of policy updates.

[0102]

[0103] In the formula, This represents the behavioral strategy used to generate samples. This is the latest policy. To ensure stable policy updates and prevent excessive deviation from the behavioral policy, the IPPO algorithm introduces a pruning objective function:

[0104]

[0105] Among them, A i ∈ is the dominance function, and ∈ controls the range of clipping.

[0106] Reference Figure 3 When the student agent reaches the exploratory learning stage, a "safe action set" A is defined to generate the behavior of the teacher agent. ξ And introduce the "security protection cost function" C. t This is used to evaluate whether the student agent's current action has exceeded the bounds and to apply soft constraints. It utilizes the Lagrange multiplier λ and the penalty function. A constrained optimization problem is constructed to ensure that students' exploratory behavior can reach the boundary without causing serious risks. Furthermore, the intensity adjustment coefficients (such as confidence boundaries and thresholds) of teacher safety intervention strategies are updated to achieve dynamic adjustment and improved robustness. Specifically:

[0107] In the free exploration phase, student agents operate the vehicle independently, while the teacher agent acts as a guardian. During this phase, student agents are free to explore various actions and scenarios, and the teacher agent intervenes only in potentially dangerous situations. This protection comes at a cost to prevent student agents from becoming overly reliant on the protection mechanism and getting trapped in local optima. Therefore, the optimal network parameters for each student agent i are determined during this phase. The optimization problem is redefined as a constrained optimization problem, which involves finding the optimal network parameters for each student agent i. The calculation formula is:

[0108]

[0109] In the formula, Let C be the objective function for optimizing the policy of the i-th student agent. t and C thd These are the security protection cost function and the manually set protection threshold, respectively. The security protection cost function C... t As shown below:

[0110]

[0111] The above security protection cost function Ct The meaning is that the action a of the i-th student agent is... i,t Safety Actions for Teachers (Set A) ξ Compare the safety action set A. ξ Teacher strategy ε(·∣o i,t ) Decision; ξ t It is a time-varying confidence boundary. If a i,t ∈A ξ If the action is deemed safe and can be executed without modification, then the security protection cost C is [not specified]. t It is (0,0), where the first value indicates no intervention and the second value reflects a zero deviation from the mean μ of the teacher's strategy; if This action is considered unsafe, and a safety protection cost is incurred. This safety protection cost is quantified as (1, ||a||). i,t -μ||), where the first value indicates that intervention has occurred, a i,t -μ represents the required level of intervention. The protective mechanism then uses this level of intervention to... i,t Adjusted to a safer action Ensure that student agents operate within safe limits.

[0112] To solve the gradient of the objective function of the i-th student agent In constrained optimization problems in multi-agent reinforcement learning environments, the Lagrange method is employed. By introducing the Lagrange multiplier λ, the constrained optimization problem is transformed into an unconstrained optimization problem, as shown below:

[0113]

[0114] The first item indicates the time range. The expected cumulative return within the timeframe ensures that the strategy seeks to maximize long-term returns. The second term integrates the security protection cost constraint into the optimization objective using Lagrange multipliers λ, where λ ≥ 0.

[0115] Set penalty value To quantify the cumulative protection cost ( The penalty value The calculation is performed within a centralized Critic network and parameterized by a neural network, trained to minimize the error between the predicted protection cost and the actual protection cost derived from the sampled trajectory. The objective function is to minimize the protection cost. Defined as:

[0116]

[0117] In the formula, B is a replay buffer containing the agent's experience. The penalty value... The training process follows the Bellman equation, where the iterative update law for the penalty value and the target value is:

[0118]

[0119] In the formula, C t Let be the instantaneous protection cost at time t, and γ be the discount factor. This formula ensures... The model accurately reflects the long-term costs of security protection, driving Actor networks to optimize their actions under the constraints of security intervention.

[0120] Based on the optimal network parameters of the i-th student agent gradient of the objective function Policy optimization for student agents can optimize the policy objectives. Minimize security protection objectives Merge into the final target and update:

[0121]

[0122] The update rule for the Lagrange multiplier λ is defined as follows:

[0123]

[0124] In the formula, When δ ≤ 0, the safety constraints are satisfied, the Lagrange multiplier λ is zero, and the original optimization problem is guaranteed not to be affected by the constraints. However, when δ > 0, the safety constraints are violated, and a proportional-integral-derivative (PID) control mechanism is used to update λ. (Term K) p δ provides an immediate proportional response to the current constraint violation, item Past violations have been accumulated for correction. The rate of change of δ is taken into account, enabling the model to adapt to potential future changes.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-agent driving learning method based on a joint training mechanism of a teacher-student paradigm, characterized in that, comprising the steps of: Step 1, constructing a human expert driving dataset D in the form of observation-action pairs; Step 2: Construct a teacher agent using a graph attention fusion time-series Graphformer, which is obtained by combining a GAT network with a Transformer: modeling human driving behavior as a graph structure G. t Using GAT networks from the graph structure G t Extracting inter-vehicle interaction features Inter-vehicle interaction features and the feature matrix of its own vehicle The fusion yields H′ t Graphformer processes H′ t The output follows a normal distribution of the expected action, by minimizing the loss function L(θ). ε Update the network parameters θ of the teacher agent. ε ; Step 3, "teacher-student paradigm" joint training mechanism design: embed the teacher agent established in step 2 into a multi-student agent reinforcement learning system, and the training mode of the multi-student agent reinforcement learning system includes a teacher-guided learning stage and a student agent free exploration stage, and the student agent executes different strategies in different stages Obtain the optimal network parameters of the student agent In the teacher-guided learning stage, the teacher strategy ε(·|o i,t ) constructed by human driving experts is used to interact with the environment to generate demonstration trajectories τ i in each state, and the network parameters θ i of the student agent are trained using each τ i ; In the autonomous exploration phase, a safety protection cost function C is introduced t And the Lagrange multiplier λ of the optimal network parameters of the student agent Constraints are made and the gradient of the objective function is passed Update the optimal network parameters of the student agent 2. The multi-agent driving learning method based on the teacher-student paradigm-based joint training mechanism of claim 1, wherein, The dataset D in the form of observation-action pairs in step 1 is represented as: In the formula, o ego,t and a ego,t respectively the observation and action of the human driver at time t, and N is the total number of observations and actions collected.

3. The multi-agent driving learning method based on the teacher-student paradigm-based joint training mechanism of claim 2, wherein, the human driver's action a at time t ego,t consisting of a steering action and a longitudinal action consisting; The human driver observes o ego,t including a state of the ego vehicle and information of other vehicles is represented as: In the formula, d l ,d r p represents the distance from the vehicle to the left and right road boundaries. x ,p y v represents the lateral and longitudinal deviations between the vehicle and the target point. ego For the speed of one's own vehicle, For the deviation between the vehicle's own heading and the reference heading, The rate of change of the vehicle's heading. These represent the relative lateral and longitudinal distances between the nth vehicle and its own vehicle, respectively. These are the relative lateral and longitudinal velocities of the nth vehicle and its own vehicle, respectively. Let k be the heading of the nth vehicle. t -1 represents the number of vehicles in the surrounding area.

4. The multi-agent driving learning method based on the teacher-student paradigm-based joint training mechanism of claim 1, wherein, The loss function L(θ ε ) of the teacher agent in step 2 is calculated according to the following formula: In the formula, θ ε is the network parameter of the teacher agent policy, N is the total number of times, a t is the action of the teacher agent at time t, σ(θ ε (o ego,t ) 2 is the variance of the action distribution output by the teacher agent, μ(θ ε (o ego,t ) is the mean of the action distribution output by the teacher agent.

5. The multi-agent driving learning method based on the teacher-student paradigm-based joint training mechanism of claim 1, wherein, The graph structure G described in step 2 t is represented as: G t = (N t , E t ) In the formula, N t is a node feature matrix, E t is an adjacency matrix, wherein, The inter-vehicle interaction feature The extraction process is: In the formula, G t is a graph structure, GATConv is a GAT network used to extract features from the graph structure G t operation of extracting features.

6. The multi-agent driving learning method based on the teacher-student paradigm based joint training mechanism of claim 1, wherein, The strategy executed by the student agent described in step 3 at different stages is represented as: In the formula, I E represents the teacher guidance learning stage; ε(·|o i,t ) represents the teacher strategy, o i,t is the observation of the ith student agent at time t; is the student agent strategy formed by the ith student agent in the autonomous exploration stage.

7. The multi-agent driving learning method based on the teacher-student paradigm based joint training mechanism of claim 1, wherein, the optimal network parameters of the student agent described in step 3 is represented as: where r i,t = R i (s i,t , a i,t ) is the reward of each student agent at time t, R i (·) is the reward function of the i-th student agent, s i,t and a i,t are the state and action of the i-th student agent at time t, respectively, and γ is the discount factor, is the policy of the student agent at different stages, and E is the expectation calculation, and are the corresponding time when the i-th student agent enters the environment and the environment terminates, respectively.

8. The multi-agent driving learning method based on the teacher-student paradigm based joint training mechanism of claim 1, wherein, Optimal network parameters of the student agent in the free exploration phase of step 3 is represented as: where r i,t = R i (s i,t , a i,t ) is the reward of each student agent at time t, R i (·) is the reward function of the i-th student agent, s i,t and a i,t are the state and action of the i-th student agent at time t, respectively, and γ is the discount factor, is the policy of the i-th student agent, and E is the expectation calculation, and are the corresponding time when the i-th student agent enters and terminates the environment, respectively, C t is the safety protection cost function, C thd is the protection threshold set by humans, and λ is the Lagrange multiplier.

9. The multi-agent driving learning method based on the teacher-student paradigm-based joint training mechanism of claim 8, wherein, The safety protection cost function C t The calculation formula is: where a i,t is the action of the ith student agent, o i,t is the observation of the ith student agent, A ξ is the set of safe actions for the teacher, ξ t is the time-varying confidence bound, ε is the policy of the teacher agent, a i,t - μ denotes the required level of intervention.

10. The multi-agent driving learning method based on the teacher-student paradigm based joint training mechanism of claim 8, wherein, J of the objective function of the student agent in the free exploration phase in Step 3 i (θ i ) is expressed as: In the formula, The goal of the strategy optimization of the student agent is to Minimizing the goal of security protection; The security protection minimization objective is defined as: where a i,t is the action of the ith student agent, o i,t is the observation of the ith student agent, B is a replay buffer containing experiences of the teacher agent, is the policy of the student agent, is the penalty value, C thd is a human-set protection threshold; The update rule of the Lagrange multiplier λ is defined as: where, When δ < 0, it means that the safety constraint is satisfied, and the Lagrange multiplier λ is zero; when δ > 0, the safety constraint is violated, and a proportional-integral-derivative control mechanism is used to update λ, the term K p δ provides an immediate proportional response to the current constraint violation, the term δ accumulates past violations to correct, the term Considering the rate of change of δ makes the model adaptive to potential changes in the future.

Images