Vehicle cooperative adaptive cruise control method based on potential driving motor

Abstract

The invention relates to the technical field of intelligent traffic system and automatic driving vehicle control, and discloses a vehicle cooperative adaptive cruise control method based on a potential driving mover, and the method comprises the steps: configuring an independent actuator network, an evaluator network and a potential driving mover model based on variational inference for each networked automatic driving vehicle; then, a potential driving motivation is extracted and quantified from vehicle local observation through the model, then unique influence of the motivation on a decision is converted into a potential reward by adopting Monte Carlo sampling and KL divergence calculation, and the potential reward is fused with an environment reward to form a mixed reward signal, so that intelligent agent strategy optimization is accurately guided, and the cooperation efficiency is improved; according to the algorithm, an online priority experience playback mechanism is adopted for training, the delay and single-point fault risk of centralized communication are effectively avoided, the robustness and expandability of the system are enhanced, training is conducted through online priority experience playback, the delay and single-point fault of centralized communication are avoided, and the robustness of the system is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation systems and autonomous vehicle control technology, specifically to a vehicle cooperative adaptive cruise control method based on potential driving motives. Background Technology

[0002] With the rapid development of intelligent transportation and autonomous driving technologies, Cooperative Adaptive Cruise Control (CACC) has received widespread attention as a core technology to improve the safety and efficiency of fleet driving. CACC technology realizes information interaction between connected autonomous vehicles (CAVs) through vehicle-to-vehicle (V2V) communication, and coordinates fleet speed and spacing. Its core objective is to solve the problem of fleet series stability. Currently, most existing CACC systems adopt traditional control strategies or basic MARL algorithms, but they have significant shortcomings.

[0003] Traditional CACC control methods mostly achieve control by establishing an optimization model and adding series stability constraints. Although they can ensure the basic stability of the vehicle fleet under ideal conditions, they lack adaptability to complex dynamic traffic environments. When faced with disturbances such as sudden deceleration of the vehicle in front or random fluctuations in driving behavior, the control response is lagging and cannot flexibly adjust decisions.

[0004] MARL-based methods, with their advantages in sequential decision-making problems, have become an important research direction in CACC technology. However, existing basic MARL algorithms face key technical bottlenecks in multi-vehicle cooperative scenarios: First, information homogenization and overgeneralization are prominent issues. Most communication-based MARL algorithms (such as CommNet and ConseNet) struggle to distinguish the state differences between different vehicles during the learning process, failing to accurately capture potential effective information in CACC scenarios. This results in a lack of targeted cooperation between vehicles and delayed fleet response. Second, collision risk and convergence efficiency are poor. In typical CACC scenarios such as chasing and deceleration, existing algorithms (such as FPrint) have a collision rate as high as 45%, and convergence is also slow. First, the time required is too long, making it impossible to quickly achieve fleet stability. Second, there is insufficient robustness; existing research has poor adaptability to real-world driving scenarios, making it difficult to cope with sudden disturbances such as rapid vehicle deceleration, and also unable to effectively adapt to the uncertainty of human driving behavior. Third, there is a lack of energy consumption optimization integration; most MARL methods under decentralized frameworks only focus on stability and safety, neglecting the importance of energy consumption optimization, which does not meet the needs of energy-saving and environmentally friendly transportation development. Fourth, system reliability needs to be improved; some centralized MARL methods that rely on global data collection are susceptible to factors such as communication delays and equipment failures, leading to an increased system failure rate, while related research in decentralized frameworks has not yet fully solved the problem of collaborative efficiency under local information interaction. Summary of the Invention

[0005] The purpose of this invention is to provide a vehicle cooperative adaptive cruise control method based on potential driving motives, so as to solve the problems mentioned in the background art.

[0006] In this application, MAACPM is defined as an abbreviation for Multi-Agent Actor-Evaluator, CAV for Connected Autonomous Vehicle, PM for Latent Drive Motivation, LSTM for Long Short-Term Memory, V2V for Vehicle-to-Vehicle, CACC for Cooperative Adaptive Cruise Control, OVM (Optimal Velocity Model) for Optimal Speed ​​Model, Dec-POMDP for Observable Markov Decision Process, and MARL for Multi-Agent Reinforcement Learning.

[0007] To achieve the above objectives, a vehicle cooperative adaptive cruise control method based on potential driving motives is provided, comprising the following steps: S1. First, vehicle dynamics modeling is performed. Then, the MAACPM algorithm under the decentralized framework is constructed based on the MARL actuator-evaluator framework. Each CAV is configured with an independent actuator network, evaluator network, and PM model. Both the actuator network and the evaluator network adopt a three-layer architecture of fully connected layer + LSTM layer + fully connected layer. The PM model is used to infer the potential effective information or potential motivation in the vehicle CACC scenario. S2. Then, each CAV obtains the state information of the vehicles in front and behind through V2V communication and uses it as its own observation space. Then, it inputs the self-observation space into the PM model to infer the potential driving motivation of each CAV, thereby generating the input of the actuator network and realizing the longitudinal control decision of the vehicle. S3. Finally, based on the MAACPM algorithm in S1, the speed and spacing between each CAV in the convoy are coordinated.

[0008] Preferably, in step S1, the vehicle dynamics modeling is as follows: First, longitudinal dynamics modeling is performed, including continuous and discrete models of vehicle longitudinal dynamics, to describe the dynamic relationship between vehicle spacing, speed, and acceleration; then, a vehicle behavior model is constructed based on OVM, which accurately reflects the dynamic adjustment law of vehicles under real traffic conditions by quantitatively analyzing driving behavior; finally, multi-agent reinforcement learning normalization is performed, that is, the vehicle cooperative adaptive cruise control problem is formalized into a decentralized partially observable Markov decision process. The equation expression for the vehicle behavior model built based on OVM is as follows: ; in, and These are the vehicle spacing gain and relative speed gain, respectively, reflecting the vehicle's sensitivity to the state of the vehicle in front; To stop the car spacing, This is the full-speed vehicle spacing; The speed strategy function for vehicle spacing is as follows:

[0009] in, The maximum speed of the vehicle is given. This speed policy function is continuously differentiable, ensuring fast convergence in a multi-agent reinforcement learning framework. The CACC control problem is formalized as a decentralized partially observable Markov decision process (Dec-POMDP), whose tuples are defined as follows: The elements are defined as follows: Intelligent agent set I: consists of N CAVs, each CAV participating in collaborative decision-making as an independent intelligent agent; Global state space S: encompasses the number of all CAVs and the distance between them. ,speed acceleration And collision status (a collision is defined as a distance of less than 1m between vehicles); Observation space O: The observation space of agent i It includes the status information of itself and the vehicles in front and behind it. ,in To normalize the speed, The speed difference with the vehicle in front, In response to the target velocity, To normalize vehicle spacing, Normalized acceleration; Action space A: Actions of agent i Based on the OVM model, by The decision is made, and the set of actions is... ; State transition distribution P: Constrained by the vehicle longitudinal dynamics model, ensuring that state transitions conform to physical laws; Discount factor Used to balance immediate rewards and future rewards, with a value of 0.99.

[0010] Preferably, the V2V communication used in step S2 to obtain the state information of the vehicles in front and behind needs to be preprocessed. Specifically, a dynamic constraint threshold is used to determine abnormal data. When the distance between vehicles is less than 1m, the speed exceeds 0-30m / s, or the acceleration deviates from -2.5-2.5m / s², it is determined to be abnormal. Invalid distance data with a safe distance of less than 15m is marked by a minimum function and included in the penalty mechanism. During correction, abnormal data is supplemented by smooth interpolation of the observation values ​​at the previous and next times. Multi-source state information is integrated by combining sensor fusion technology, and the consistency of data time sequence is ensured by timestamp alignment. The preprocessed observation values ​​replace the global state input to the PM model.

[0011] Preferably, the first two layers of the evaluator network use the ReLU activation function to enhance nonlinear expressive power, and the last layer uses the linear activation function to output the value function; the last layer of the actuator network uses the softmax activation function to adapt to the action space.

[0012] Preferably, the PM model described in step S1 is constructed based on the variational inference method, and the specific steps are as follows: First, define the prior distribution. and posterior distribution Both are modeled as multivariate diagonal Gaussian distributions; then, a generative network approximates the prior distribution, and an inference network approximates the posterior distribution; both the generative and inference networks are three-layer neural network architectures; finally, a data generator is defined. Its input is the latent motivation of all CAVs. The output is the global state. The distribution; finally, a function is constructed based on the lower bound of the evidence. The PM model parameters are optimized by minimizing this loss function. In actual training, observations obtained using V2V communication are used. Replace global state To enhance real-time performance.

[0013] Preferably, the three-layer neural network architecture of the generator network and the inference network is as follows: the first layer is a fully connected layer, the second layer is an LSTM layer used to capture the temporal dependence of the sequence data, and the third layer consists of two fully connected structures that output multivariate means respectively. and variance .

[0014] The preferred design steps for potential reward signals are as follows: First, define the actual strategy. Counterfactual strategy In this process, the actual policy includes latent motives, while the counterfactual policy does not. Then, the Monte Carlo method is used to approximate the counterfactual policy. Next, the KL divergence is used to calculate the difference between the actual and counterfactual policies, which is defined as the latent reward. The results are then subjected to minimum-maximum normalization, i.e., the minimum-maximum normalized extreme value of the potential reward. This is determined based on the statistical results of the potential reward throughout the training process: the minimum potential reward of all samples during training is used as the lower limit of normalization, and the maximum potential reward is used as the upper limit, forming a statistical interval suitable for cooperative adaptive cruise control scenarios. Finally, a hybrid reward signal is constructed. And use it to update the parameters of the evaluator network and the actuator network; Among them, environmental awards The calculation formula is: ; Weighting coefficient , , , Adjust according to the scenario. Let be the distance between the i-th CAV and the vehicle in front at time t. For the target vehicle spacing, Let be the velocity of the i-th CAV at time t. For the target speed, Let be the acceleration of the i-th CAV at time t. For a safe distance, This is a reward for the fleet when the vehicle spacing meets the standard.

[0015] Preferably, the Monte Carlo method's approximate counterfactual strategy involves drawing at least two independent and identically distributed samples from the posterior distribution, and then taking their average as an approximation. Preferably, 1000 independent and identically distributed samples are drawn from the posterior distribution. ,calculate As an approximation.

[0016] Preferably, the MAACPM algorithm employs an online training method, specifically: First, an online playback buffer D is established to store the transformed data. Then, when the amount of data stored in buffer D reaches a preset threshold, a certain number of small batches of samples are sampled from it. Preferably, the sampling batch size is... The parameters are updated using a small batch of 60 samples. Finally, the parameters of the evaluator network, the actuator network, and the potential driving motivation model (PM model) are updated using these small batches of samples, achieving online training of the algorithm. The evaluator network is trained by minimizing the loss function. The actuator network updates parameters via gradients. Update parameters The dominant function Hyperparameter settings include total number of steps The discount factor γ = 0.99, and the learning rates of the actuator network and the evaluator network are... The learning rate of the PM model is The number of hidden layer units in the actuator network and the evaluator network are 64 and 128, respectively.

[0017] Preferably, the parameter update priority rules for the evaluator network, actuator network, and latent driving motivation model are as follows: First, update the evaluator network parameters. Based on the mixed reward signal and advantage function, optimize the value function output accuracy by minimizing the loss function, providing an accurate value assessment basis for the actuator network's policy adjustment. Second, update the actuator network parameters. Based on the advantage function output by the evaluator network, adjust the action generation strategy through gradient ascent to ensure that the strategy optimization aligns with the current value assessment, achieving the collaborative control objective. Finally, update the PM model parameters, which, as the core of latent motivation inference, are optimized by minimizing the lower bound of the evidence loss function using small-batch samples. To ensure the accuracy of potential motivation inference and the motivation-oriented approach of the reverse optimization strategy, all three rely on small-batch sample iterative updates in a synchronized manner through an online replay buffer to guarantee the consistency of parameter optimization and the stability of algorithm convergence. The gradient pruning threshold for parameter updates of the actuator network and PM model is set to 1.0 to meet the training stability requirements: this threshold can effectively suppress training oscillations caused by gradient explosion during parameter updates, avoiding large fluctuations in control indicators such as vehicle distance and speed; at the same time, it is compatible with the learning rate configuration of the actuator network and PM model, ensuring that the gradient propagates within a reasonable range, and together with small-batch sample iterative updates, ensuring the stability and convergence efficiency of model parameter optimization.

[0018] The present invention has the following advantages over the prior art: 1. This invention introduces a potential driving motivation model based on variational inference, which can infer differentiated and potentially effective information affecting the decision-making of each vehicle from high-dimensional, homogeneous local observation information. It also innovatively designs a mechanism to quantify potential motivation into reward signals and integrates these potential rewards into the training process of the agent. This guides the agent not only to focus on basic environmental rewards such as vehicle spacing and speed tracking, but also to actively explore and utilize deep information that has a unique impact on its decision-making, thereby improving the discriminativeness, adaptability and overall collaborative efficiency of the strategy. 2. The decentralized architecture adopted enables each vehicle to make independent decisions, avoiding overall failure caused by the failure of the central node. It also achieves local coordination through V2V communication, which improves the system's fault tolerance. At the same time, the algorithm has been verified by real autonomous driving datasets (such as real vehicle data from the European Commission's JRC) and human driving behavior data, showing that it can quickly respond to real disturbances such as sudden deceleration of the vehicle in front and effectively smooth out random fluctuations in human driving, demonstrating strong adaptability and robustness to complex and dynamic real traffic environments. 3. By incorporating an environmental reward function with an acceleration penalty term and optimizing it during training, the vehicle is indirectly guided to form a smooth acceleration curve, which helps reduce energy consumption and fills the gaps in existing research on energy consumption optimization integration. Attached Figure Description

[0019] Figure 1 This is the overall architecture of the CAV formation system in this invention; Figure 2 This outlines the structure of the PM model and the construction of its corresponding loss function in this invention. Figure 3 This is an overview of the MAACPM method in this invention; Figure 4 The learning curves of the MAACPM algorithm in this invention compared to other baseline algorithms; Figure 5 This invention provides vehicle driving data from the MAACPM and IA2C algorithms in a chase scenario, including curves showing the changes in vehicle distance and speed over time. Figure 6 This invention provides vehicle driving data from the MAACPM and IA2C algorithms in a deceleration scenario, including curves showing the changes in vehicle distance and speed over time. Figure 7 The data includes real and simulated driving data of the 5-vehicle convoy in this invention over 60 seconds, including the curves of vehicle distance and speed change over time. Figure 8 The data includes real and simulated driving data of a 7-vehicle convoy within 55 seconds in this invention, including the curves of vehicle distance and speed change over time. Figure 9 The driving data of the convoy in this invention, in which the lead vehicle is a human driver, includes the vehicle distance and speed curves over time. Figure 10 This invention addresses the hyperparameters in pursuit and deceleration scenarios. Perform ablation experiments; Figure 11 For the purpose of this invention, the size of the convoy is determined in chase and deceleration scenarios. An ablation experiment was conducted. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] In one implementation, such as Figures 1-11 The following is a detailed description of a vehicle cooperative adaptive cruise control method based on potential driving motives.

[0022] 1. System Overall Architecture Setup The overall system framework is as follows Figure 1 As shown, the CACC system constructed in this invention comprises N networked autonomous driving vehicles traveling along a straight road. The lead vehicle is the navigator CAV, followed by N-1 other CAVs. All CAVs possess uniform dynamic characteristics, such as the same maximum acceleration, maximum speed, friction coefficient, and vehicle weight, ensuring the simplicity and efficiency of the control algorithm. Each CAV acquires its own kinematic parameters, such as speed, acceleration, and distance between vehicles, through its installed sensors and achieves real-time information exchange via a V2V communication network, ensuring efficient sharing of local state information and supporting decentralized decision-making. The navigator CAV is responsible for determining the target distance between vehicles and performing longitudinal control. and target driving speed The system guides the entire fleet to move forward in a coordinated manner. The actuator outputs of each connected autonomous vehicle achieve localized synchronization through V2V communication: the actuator outputs of motion parameters such as vehicle spacing gain and relative speed gain, along with the vehicle's current speed, distance, and acceleration, are encapsulated and broadcast to adjacent vehicles via V2V communication. Receiving vehicles substitute the motion parameters of neighboring vehicles into their longitudinal dynamics models, verify the feasibility of coordination based on safety constraints, and adjust their own parameters if conflicts exist. The synchronization period is consistent with the sampling time to ensure coordinated updates of actions within the same time step. Large-scale fleets employ a hierarchical "lead vehicle-follower vehicle" mechanism. The lead vehicle broadcasts target reference information, and follower vehicles sequentially synchronize information from adjacent vehicles, ensuring serial stability and communication efficiency.

[0023] During system operation, to ensure driving safety and comfort, each CAV must maintain a specified target speed v and target vehicle distance h, and strictly adhere to the following kinematic constraints: Vehicle spacing constraints: , Let be the distance between the i-th CAV and the vehicle in front at time t, to ensure that no collision occurs between the vehicles; Speed ​​constraints: , Let be the speed of the i-th CAV at time t, which complies with the road speed regulations; Acceleration constraints: , Let be the acceleration of the i-th CAV at time t, to ensure driving comfort and safety.

[0024] 2. Vehicle Modeling 2.1 Longitudinal Dynamics Modeling

[0025] To accurately describe the dynamic behavior of the vehicle during driving, continuous and discrete longitudinal kinematic models were constructed respectively: 2.1.1 Continuous Model: To accurately describe the dynamic behavior of the vehicle during driving, the longitudinal kinematics of the vehicle are modeled as follows: Vehicle spacing relationship: This reflects the impact of the speed difference between the front and rear vehicles on the distance between vehicles; in Let be the distance between vehicle i and the vehicle in front of it i-1 at time t. Let be the speed of vehicle i at time t; Relationship of velocity changes: This indicates that the vehicle's acceleration directly determines the rate of change of velocity; in Let be the acceleration of vehicle i at time t; 2.1.2 Discretization Model: Considering sampling time The discrete longitudinal kinematic equations of vehicle i are: ; ; To meet the discretization requirements of online algorithm training; 2.2 Vehicle Behavior Modeling A vehicle behavior model is constructed based on the Optimal Velocity Model (OVM). By quantitatively analyzing driving behavior, it accurately reflects the dynamic adjustment patterns of vehicles in real traffic scenarios. Its equation is expressed as: ; in, and These are the vehicle spacing gain and relative speed gain, respectively, reflecting the vehicle's sensitivity to the state of the vehicle in front; To ensure a safe stopping distance, a value of 5m is used in this embodiment. For the full-speed vehicle spacing, a value of 30m is used in this embodiment; The speed strategy function for vehicle spacing is as follows:

[0026] The policy function is continuously differentiable, ensuring successful convergence within the MARL framework. The maximum speed of the vehicle is taken as 30 m / s in this embodiment; 2.3 Standardization of Multi-Agent Reinforcement Learning The CACC problem is formalized as a decentralized partially observable Markov decision process (Dec-POMDP), represented by tuples as follows: The elements are defined as follows: Set of intelligent agents I: contains N CAVs, each CAV acting as an independent intelligent agent; Global state space S: encompasses the number of all CAVs and the distance between them. ,speed acceleration And collision status (a collision is defined as a distance of less than 1m between vehicles); Observation space O: The observation space of agent i It includes the status information of itself and the vehicles in front and behind it. ,in To normalize the speed, The speed difference with the vehicle in front, In response to the target velocity, To normalize vehicle spacing, Normalized acceleration; Action space A: Actions of agent i Based on the OVM model, by The decision is made, and the set of actions is... ; Reward function R: The key signal guiding the agent's learning trajectory, designed as follows: ; in, , , , These are weighting coefficients used to adjust the reward percentages for vehicle spacing deviation, speed deviation, acceleration, and safety constraints, respectively. To maintain a safe distance; The reward for vehicles in a convoy with consistent spacing is subject to the following conditions: and , Triggered when the confidence level is extremely low; State transition distribution P: Constrained by the vehicle kinematics model to ensure that state transitions conform to physical laws; Discount factor Used to balance immediate rewards and future rewards, with a value of 0.99.

[0027] 3. Construction of Potential Driving Motivation Model 3.1 PM Model Building A variational inference method is used to construct a potential-driven motivation (PM) model to accurately extract potential effective information (potential motivation) in CACC scenarios. ),like Figure 2 As shown, the model includes an inference network. Generative Networks and data generator Unified parameterization .

[0028] The specific steps are as follows: 3.1.1 Definition of distribution: Define the prior distribution. With posterior distribution Both are modeled as multivariate diagonal Gaussian distributions; among them, Let i be the potential motivation of the i-th CAV at time t. Let i be the action of the i-th CAV at time t-1. Let be the global state at time t; To adapt to multi-dimensional vehicle observation information, support independent modeling of differentiated potential motivations such as safety and efficiency, avoid information coupling, and achieve fine-grained inference, a multivariate diagonal Gaussian distribution is chosen. Specifically, its parameter scale is O(d), which simplifies the calculation of determinant and inverse matrix. Combined with Monte Carlo sampling, counterfactual strategy approximation can be quickly completed, meeting the real-time decision-making requirements under a decentralized framework. Secondly, the mean and variance can be directly output through the dual fully connected layers of the generation and inference networks, compatible with the "fully connected layer + LSTM layer" architecture, ensuring stable gradient propagation. Finally, the covariance structure is constrained to reduce the risk of overfitting, avoid the singular value problem of the total covariance matrix in local observation scenarios, and improve the robustness of motivation inference.

[0029] 3.1.2 Network Construction: Both inference and generative networks employ a three-layer neural network architecture. The first layer is a fully connected layer (processing input information), the second layer is an LSTM layer (capturing the temporal dependencies of sequence data), and the third layer consists of two fully connected structures (outputting multidimensional means respectively). and variance This constitutes a multivariate diagonal Gaussian distribution. ).

[0030] Data generator: Its structure is consistent with the inference network, and its input is the latent motivation of all agents. Output global state The distribution of .

[0031] 3.1.3 Model Optimization: The model parameters are optimized based on maximizing the lower bound of evidence (ELBO), with the loss function being: ; Compared with the loss function of the latent model in existing MARL, the loss function based on the lower bound of evidence in this application has advantages in improving the accuracy of motivation inference and reducing training fluctuations: First, through the dual-objective design of reconstruction term and KL divergence, the reconstruction term ensures that the latent motivation contains effective decision-making information, and the KL divergence term constrains the consistency of prior and posterior distributions. This dual guarantee makes motivation inference more accurate and effectively captures differentiated orientations such as safety and efficiency. Second, KL divergence has a built-in regularization effect, which can suppress data noise interference, and the expected form of the loss function can reduce the influence of abnormal samples, making the fluctuations of vehicle distance and speed during training smaller and the learning curve more stable. Finally, it adapts to scenarios where local observations replace global states and can still maintain high inference accuracy in a decentralized framework, avoiding the performance degradation problem of existing loss functions in partially observable scenarios.

[0032] In actual training, observations obtained by all agents through V2V communication are used. Replace global state Input the PM model to infer the underlying motivations of each CAV. This reduces global data dependencies.

[0033] 3.2 Calculation of Potential Rewards To quantify potential motivations Derive inclusion and exclusion based on the different effects of each CAV. The strategy difference, and use it as a potential reward. Incentivize agents to explore useful information and design potential reward mechanisms: 3.2.1 First, define the strategy: Practical strategy: ,in From the posterior distribution Mid-sampling, including decision-making strategies with underlying motivations; Counterfactual strategy: a decision-making strategy based on the total probability formula and devoid of underlying motives; ; Anti-implementation strategy approximation: Using the Monte Carlo method, approximating from the posterior distribution... 1000 independent and identically distributed sampled values ​​were drawn from the middle. The counterfactual strategy is approximated by the following formula: ; in, Let n be n independent and identically distributed samples drawn from the posterior distribution, where n takes the value 1000; Sampling rules for generating latent motives based on posterior distribution parameters: ① The number of samplings is fixed at 1000. 1000 independent and identically distributed samples are drawn from the posterior distribution of latent motives to ensure the reliability of the counterfactual policy approximation; ② The sampling results are screened based on vehicle kinematic constraints, removing anomalous samples that cause the distance between vehicles to be less than 1m, the speed to exceed 0-30m / s, or the acceleration to deviate from -2.5-2.5m / s²; ③ The effective samples after screening are used to calculate the average policy, approximating the counterfactual policy without latent motives, providing a reliable basis for the quantification of potential rewards. Monte Carlo sampling sample screening rules: The Monte Carlo sampling sample screening is based on vehicle kinematic constraints as the core logic: After drawing 1000 independent and identically distributed latent motive samples from the posterior distribution, anomalous samples that cause the distance between vehicles to be less than 1m, the speed to exceed 0-30m / s, or the acceleration to deviate from -2.5-2.5m / s² are removed, and effective samples that match the vehicle state constraints are retained for the approximate calculation of the counterfactual policy.

[0034] 3.2.2 Definition of Potential Reward: The KL divergence is used to measure the difference between the actual policy and the counterfactual policy, which is then used as the potential reward. ; The potential reward is min-maximum normalized to avoid excessively large values ​​affecting the stability of the learning process. The rule for determining the extreme value of the minimum-maximum normalization of potential reward is as follows: The minimum-maximum normalization extreme value of potential reward is determined based on the statistical results of potential reward throughout the training process. The minimum potential reward of all samples during the training process is used as the lower limit of normalization, and the maximum value is used as the upper limit, forming a statistical interval that adapts to the cooperative adaptive cruise control scenario. This interval covers the fluctuation range of potential reward under typical scenarios such as chasing and deceleration, ensuring that the normalized reward value is mapped to the [0,1] interval, which not only preserves the differentiated influence of potential motivation under different scenarios, but also avoids extreme values ​​interfering with the stability of model training.

[0035] 3.2.3 Hybrid Reward Construction: ; in For environmental rewards, Normalized to 1, Based on the characteristics of the CACC scenario, this implementation method flexibly adjusts to the chasing scenario. =0.3, deceleration scenario =0.5, the environmental reward calculation formula is as follows: ; in, , , These are the weighting coefficients. To maintain a safe distance; Vehicle spacing requirement bonus, meeting the following conditions: and ; in, For the smallest confidence value, catch-up scene Deceleration scenario .

[0036] The latent motivation inference results are correlated with the hybrid reward function through quantification transformation and weight adjustment. The latent motivation is a multi-dimensional vector, with each dimension corresponding to differentiated decision orientations. Its inference results, together with local vehicle observations, construct the actual strategy. By approximating the counterfactual strategy without latent motivation through Monte Carlo sampling, and then quantifying the difference between the two using KL divergence, the latent reward is obtained. The normalized weights of each dimension of motivation, learned through training, directly adjust the contribution ratio to the KL divergence. After normalization, the latent reward is weighted and merged with the environmental reward focusing on the basic control objective to form a hybrid reward. The overall proportion of the latent reward is adjusted according to the scenario characteristics. The dual quantification adjustment mechanism not only ensures the basic driving requirements but also strengthens the guidance of motivation on strategy optimization, promoting the agent to learn the optimal collaborative strategy.

[0037] The quantitative adjustment mechanism of motivation weights for reward items: Each dimension of potential motivation corresponds to a differentiated decision-making orientation. Normalized weights are automatically learned during training, with higher contributions resulting in greater weights. Potential rewards are derived by quantifying strategy differences through KL divergence. The motivation weights of each dimension directly allocate the contribution proportion of that divergence. Strategy differences in high-weight dimensions have a more significant impact on potential rewards. Hybrid rewards are a weighted fusion of potential rewards and environmental rewards. The overall proportion of potential rewards is adjusted according to scenario characteristics, forming a dual quantitative adjustment with the motivation dimension weights. This ensures basic driving requirements while strengthening the guidance of differentiated motivations on strategy optimization. These weights are updated synchronously with model parameters and optimized through small-batch sample training, effectively improving algorithm stability and collaborative efficiency.

[0038] 4. Multi-agent reinforcement learning framework for vehicle cooperative control A vehicle cooperative adaptive cruise control algorithm based on potential driving motives is constructed, and vehicle cooperative control is realized within a decentralized framework. The algorithm framework is as follows: Figure 3 As shown, the underlying motivation for inferring the network output is... It was used for policy network execution and value function network training, and through sampling-based... Counterfactual inferences may yield potential rewards The algorithm includes an evaluator network, an actuator network, and a PM model. Each CAV is equipped with an independent network component to ensure independent decision-making and collaborative capabilities.

[0039] 4.1 Evaluator Network Structure A three-layer neural network (fully connected layer + LSTM layer + fully connected layer) is used. The first two layers use the ReLU activation function (to enhance nonlinear expressive power), and the last layer uses the linear activation function (output value function).

[0040] The parameters are updated by minimizing the loss function. The loss function is: ; in, and These are the weighting coefficients. Standardized to 1, Adjust flexibly according to the characteristics of the scenario; For environmental rewards.

[0041] 4.2 Actuator Network Structure: The actuator network structure is consistent with the evaluator network, with the last layer using the softmax activation function (adapted to the action space). Parameters are updated via gradient ascent. The gradient formula is: ; in, This is the dominant function.

[0042] 5. MAACPM Algorithm Training Process The MAACPM algorithm uses an online training method, and the training process is shown in the table below:

[0043] The specific steps are as follows: (1) Initialization: Set the number of agents Maximum trajectory duration T, total number of steps Online playback buffer D, small batch size Initialize action network parameters Evaluator network parameters and PM model parameters ; (2) Trajectory collection: For each training step, initialize the CACC random state S and receive the initial observations of each agent. At each time step t, sample the latent motivation. Select Action Obtain environmental rewards after performing the action. and the next moment of observation Calculate potential reward and combination rewards ,Will Store in playback buffer D; (3) Network update: When the replay buffer D is full, sample a small batch of data. The parameters of the evaluator network, executor network, and PM model are updated separately. The parameter update priority rules for the evaluator, executor, and PM models are as follows: The parameter update priority is based on the model convergence efficiency and the accuracy of collaborative control. Specifically, the evaluator network parameters are updated first. Based on the mixed reward signal and the advantage function, the value function output accuracy is optimized by minimizing the loss function, providing an accurate value assessment basis for the executor network's policy adjustment. Second, the executor network parameters are updated. Based on the advantage function output by the evaluator network, the action generation policy is adjusted through gradient ascent to ensure that the policy optimization fits the current value assessment and achieves the collaborative control objective. Finally, the PM model parameters are updated. As the core of potential motivation inference, the PM model is optimized by minimizing the lower bound of the evidence loss function through small batch samples to ensure the accuracy of potential motivation inference and to optimize the motivation orientation of policy adaptation. All three are updated synchronously through small batch samples in the online replay buffer to ensure the consistency of parameter optimization and the stability of algorithm convergence.

[0044] Gradient clipping threshold and setting basis during actuator and PM model parameter updates: The gradient clipping threshold for actuator and PM model parameter updates is uniformly set to 1.0. The setting basis is to adapt to the training stability requirements: This threshold can effectively suppress training oscillations caused by gradient explosion during parameter updates, and avoid large fluctuations in control indicators such as vehicle distance and speed; at the same time, it is compatible with the learning rate configuration of the actuator network and PM model, ensuring that the gradient propagates within a reasonable range, and with small batch sample iterative updates, ensures the stability and convergence efficiency of model parameter optimization.

[0045] Iterative training: Repeat the above steps until the total number of training steps is reached. The algorithm converged.

[0046] 6. Multi-dimensional experimental verification, including simulation experimental verification, real-world scenario verification, and ablation experiments; 6.1 Simulation Experimental Environment Hardware environment: Ubuntu 20.04 operating system, Intel Core i7-7700k CPU @ 4.20GHz Comparison Algorithms: The MAACPM algorithm is compared with IA2C (non-communication method), FPrint, MACACC, CommNet, and ConseNet (communication-based method) algorithms to evaluate average round reward, average vehicle distance, average vehicle speed, collision rate, and average convergence time. Meanwhile, the executor and evaluator network structures of all algorithms are consistent with MAACPM, and the hyperparameters are kept uniform. Reward function parameters: , In the chase scene In deceleration scenarios ; Training hyperparameter: Number of agents Discount factor Executor and evaluator learning rates PM model learning rate Batch size The number of hidden layer units for the actuator and evaluator is 64. The number of hidden layer units is 128.

[0047] 6.2 Typical Scenario Experiments: Two typical CACC scenarios, Catchup and Slowdown, are used to verify the algorithm's performance. (1) Chase scenario: The initial velocity of all CAVs is The initial distance between the car in front and the car in the chase scenario ,in, Random sampling is performed within the interval, and the initial distance between subsequent vehicles is... Each round lasts 60 seconds.

[0048] (2) Deceleration scenario: The initial distance between all CAVs is initial velocity ( And random sampling), target speed It linearly decreases to 15 m / s within the initial 30 seconds and then remains constant, with a round duration of 120 seconds.

[0049] (3) Experimental indicators: average reward, average vehicle distance, average vehicle speed, collision rate, average convergence time (when the vehicle distance is continuously satisfied) and (Time).

[0050] 6.3 Experimental Results and Analysis like Figure 4 As shown, the MAACPM algorithm achieved the highest training reward in both scenarios (chasing and deceleration scenarios) and exhibited good convergence, significantly outperforming the baseline algorithm. The MAACPM algorithm demonstrated the best performance in average round reward, accuracy of the average vehicle distance to the target vehicle distance, accuracy of the average vehicle speed to the target vehicle speed, collision rate control, and average convergence time. In the chasing scenario, the average convergence time of MAACPM was only 25.99 seconds, far lower than MACACC's 32.38 seconds, as shown in the table below:

[0051] In deceleration scenarios, the MAACPM collision rate is 0.00%, with an average convergence time of 58.28 seconds, significantly outperforming other algorithms; as shown in the table below:

[0052] Vehicles controlled by the MAACPM algorithm exhibit less fluctuation in distance and speed, and can quickly adjust to the target value, while vehicles controlled by the IA2C algorithm show significant fluctuations and insufficient convergence ability. Figure 5 (Chase scene) and Figure 6 As shown in the (deceleration scenario).

[0053] 6.4 Real-world scenario verification (1) Validation of autonomous driving dataset: Extract real driving data of Tesla fleet from the open source database of the European Commission Joint Research Center, introduce the sudden deceleration disturbance of the lead vehicle into a simulated fleet of 5 or 7 vehicles, and compare the distance between the CAV and the real vehicle, speed recovery time and fluctuation in the simulation.

[0054] Experimental results are as follows Figure 7 and Figure 8 As shown, vehicles controlled by the MAACPM algorithm can recover to their pre-disturbance distance and speed state more quickly, with less fluctuation, and achieve synchronous response between vehicles through V2V communication, effectively reducing control delay. (2) Human driving data validation: Using TRACKRACER's TR80Lite racing simulator combined with the IPGCarmaker simulation platform, driving data (acceleration and deceleration processes) of three drivers in high-speed scenarios were collected and used as input for the simulation test of the leading vehicle's driving behavior model; the results are as follows: Figure 9 As shown, despite slight fluctuations in the driving behavior of the lead vehicle, the MAACPM algorithm can still quickly adapt to disturbances and bring the system to converge to the target value, demonstrating strong robustness and adaptability.

[0055] 6.5 Ablation Test Potential reward impact verification: By adjusting the potential reward weights... The performance of the MAACPM algorithm under different parameters and fleet sizes will be evaluated to verify the potential rewards and the algorithm's adaptability to different fleet sizes. Set the values ​​to 1 (including potential rewards) and 0 (excluding potential rewards) respectively, and compare the performance curves in the two scenarios (e.g., Figure 10 (As shown in the figure); the results show that the algorithm performs better when potential rewards are included, verifying the incentive effect of potential rewards on the agent's exploration of effective information.

[0056] Fleet size adaptability verification: Set fleet sizes of 2, 5, and 8 vehicles, and test the vehicle spacing and speed distribution under different sizes (e.g., Figure 11 (As shown in the figure); the results show that the MAACPM algorithm can adjust the distance and speed to close to the target value under different fleet sizes, demonstrating good adaptability and flexibility.

[0057] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A vehicle cooperative adaptive cruise control method based on potential driving motives, characterized in that: Includes the following steps: S1. First, vehicle dynamics modeling is performed. Then, a decentralized multi-agent actuator-evaluator algorithm is constructed based on the actuator-evaluator framework of multi-agent reinforcement learning. Each connected autonomous vehicle is configured with an independent actuator network, evaluator network, and potential driving motivation model. Both the actuator network and the evaluator network adopt a three-layer architecture of fully connected layer + long short-term memory layer + fully connected layer. The potential driving motivation model is used to infer potential effective information in the vehicle cooperative adaptive cruise control scenario. S2. Then, each connected autonomous vehicle obtains the state information of the vehicles in front and behind through vehicle-to-vehicle communication and uses it as its own observation space. Then, it inputs the self-observation space into the potential driving motivation model, thereby inferring the potential driving motivation of each connected autonomous vehicle, thereby generating the input of the actuator network, and thus realizing the longitudinal control decision of the vehicle. S3. Finally, based on the multi-agent actuator-evaluator algorithm in S1, the speed and spacing between each connected autonomous vehicle in the fleet are coordinated.

2. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: In step S1, the vehicle dynamics modeling is as follows: First, longitudinal dynamics modeling is performed, including continuous and discrete models of vehicle longitudinal dynamics, which are used to describe the dynamic relationship between vehicle spacing, speed and acceleration; then, a vehicle behavior model is constructed based on the optimal speed model. This model accurately reflects the dynamic adjustment law of vehicles under real traffic conditions by quantitatively analyzing driving behavior; finally, multi-agent reinforcement learning normalization is performed, that is, the vehicle cooperative adaptive cruise control problem is formalized into a decentralized partially observable Markov decision process.

3. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: The local observation information of the vehicle fleet mentioned in step S2 needs to be preprocessed. Specifically, abnormal data is identified by using dynamic constraint thresholds; abnormal data is supplemented by smooth interpolation of observations at different times; multi-source state information is integrated by combining sensor fusion technology; and data temporal consistency is ensured by timestamp alignment. The preprocessed observations replace the global state input to the potential driving motivation model.

4. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: The first two layers of the evaluator network use the ReLU activation function to enhance nonlinear expressive power, and the last layer uses the linear activation function to output the value function; the last layer of the actuator network uses the softmax activation function to adapt to the action space.

5. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: The latent driving motivation model described in step S1 is constructed based on the variational inference method. The specific steps are as follows: First, define the prior distribution and the posterior distribution, both of which are modeled as multivariate diagonal Gaussian distributions; then, approximate the prior distribution through a generator network and approximate the posterior distribution through an inference network, both of which are three-layer neural network architectures; next, set a data generator, whose input is the latent motivation of all connected autonomous vehicles and whose output is the distribution of the global state; finally, construct a function based on the lower bound of evidence, and optimize the parameters of the latent driving motivation model by minimizing this loss function.

6. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 5, characterized in that: The three-layer neural network architecture of the generating network and the inference network is as follows: the first layer is a fully connected layer, the second layer is a long short-term memory (LSTM) layer to capture the time dependence of sequence data, and the third layer consists of two fully connected structures that output multivariate mean and variance respectively.

7. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: The method also includes potential reward signals, specifically: first, defining the actual strategy. The counterfactual strategy is proposed, where the actual strategy contains latent motivation, while the counterfactual strategy does not. Then, the Monte Carlo method is used to approximate the counterfactual strategy. Next, the KL divergence is used to calculate the difference between the actual and counterfactual strategies, which is defined as the potential reward. The result is then subjected to minimum-maximum normalization, i.e., the minimum-maximum normalized extreme value of the potential reward. This is determined based on the statistical results of the potential reward throughout the training process: the minimum potential reward of all samples during training is used as the lower bound of normalization, and the maximum value is used as the upper bound, forming a statistical interval suitable for the cooperative adaptive cruise control scenario. Finally, a hybrid reward signal is constructed and used to update the parameters of the evaluator network and the actuator network.

8. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 7, characterized in that: The Monte Carlo method's approximate counterfactual strategy involves drawing at least two independent, identically distributed samples from the posterior distribution and then taking their average as an approximation.

9. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 1, characterized in that: The method also includes the training process of the multi-agent executor-evaluator algorithm. The multi-agent executor-evaluator algorithm adopts an online training method, specifically: First, an online replay buffer D is established to store the transformation data. Then, when the amount of data stored in the buffer D reaches a preset threshold, a certain number of small batch samples are sampled from it. Finally, the model parameters of the evaluator network, the executor network, and the potential driving motivation model are updated using the small batch samples to realize the online training of the algorithm.

10. The vehicle cooperative adaptive cruise control method based on potential driving motives according to claim 9, characterized in that: The parameter update priority rule for the evaluator network, actuator network, and potential driving motivation model is as follows: update the evaluator network parameters first, then update the actuator network parameters, and finally update the potential driving motivation model parameters. The gradient clipping threshold for updating the actuator network and potential driving motivation model parameters is set to 1.0 to suppress gradient explosion and ensure training stability.

Images