Intelligent Driving Simulation Method and System Based on Multi-Agent Reinforcement Learning

Abstract

This invention discloses an intelligent driving simulation method and system based on multi-agent reinforcement learning, belonging to the field of intelligent driving technology. It addresses the problems of traditional path planning methods, which are typically based on static maps and rules, making them difficult to adapt to dynamically changing traffic environments and prone to local optima. Furthermore, traditional allocation strategies, based on fixed rules or simple heuristic algorithms, lead to energy waste and reduced driving range. The choice of attenuation methods significantly impacts the convergence speed and stability of the algorithm. Through multi-agent reinforcement learning, the system achieves intelligent path planning, power distribution, and speed adjustment, improving the level of driving automation. By comprehensively considering safety, efficiency, and comfort, it achieves multi-objective optimization in complex traffic environments, improving overall driving performance. It exhibits strong robustness and adaptability, dynamically adjusting battery energy consumption allocation strategies to reduce energy waste.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more specifically, to an intelligent driving simulation method and system based on multi-agent reinforcement learning. Background Technology

[0002] Intelligent control technology is primarily used for the automated control of various complex systems and processes, enabling them to operate efficiently, reliably, and intelligently. It is widely applied in various fields such as industrial production, transportation, and energy management. In rail transit turnout snow melting systems, intelligent control technology can improve snow melting efficiency and reduce track risks.

[0003] The existing technology has the following shortcomings:

[0004] Traditional route planning methods, typically based on static maps and rules, struggle to adapt to dynamically changing traffic environments, neglecting global efficiency and safety, and are prone to local optima. Allocation strategies are often based on fixed rules or simple heuristics, leading to energy waste and reduced driving range. In reinforcement learning, the choice of decay methods significantly impacts the algorithm's convergence speed and stability, making it difficult to adapt to the demands of complex tasks.

[0005] To address the above problems, this invention proposes a solution. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an intelligent driving simulation method and system based on multi-agent reinforcement learning to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method and system for intelligent driving simulation based on multi-agent reinforcement learning, including the following steps:

[0009] Step S1: Design a reward function based on the agent's safety, efficiency, and comfort, interact with the environment to obtain immediate rewards, update the state value table using the Bellman equation, and gradually find the optimal path;

[0010] Step S2: Utilize time series data combined with safety, efficiency, and comfort to predict future vehicle status, perform predictive analysis on power distribution, and dynamically adjust battery energy consumption distribution strategy based on prediction results;

[0011] Step S3: Determine whether to choose the linear decay method or the exponential decay method based on the complexity of the task and the stability of the reward level;

[0012] Step S4: Based on Q-learning, adjust vehicle speed and maintain a safe following distance by combining safety and power distribution.

[0013] In a preferred embodiment, step S1 includes the following:

[0014] Taking into account safety, efficiency, and comfort, a reward function is derived, assigning an immediate reward R to each action of the agent:

[0015] Define the safe distance threshold D safe Calculate the distance D between the agent and the nearest obstacle. current The safety factor is expressed as:

[0016] Define the remaining distance D from the agent to the target point. goal Estimate the time required for the agent to reach the target point at its current speed V.

[0017] The efficiency coefficient is expressed as: Where: k1 is an adjustment parameter, T goal It is the ideal time for the agent to reach the target point;

[0018] Calculate the rate of change of acceleration Δa and the rate of change of steering angle Δθ of the intelligent agent. The comfort coefficient is expressed as: R comfort =-(w a ·|Δa|+w θ ·|Δθ|), |Δa|: change in acceleration; |Δθ|: change in steering angle; w a and w θ It is a weight used to balance the impact of acceleration and steering angle on comfort;

[0019] The weighted sum of rewards based on safety, efficiency, and comfort forms the final immediate reward, expressed as: R = w s ·R safety +w e ·R efficiency +w c ·R comfort , where: w s ,w e ,w c These are the corresponding weights of safety, efficiency, and comfort, used to balance the importance of safety, efficiency, and comfort; R safety : is a security reward; R efficiency It's an efficiency reward; R comfort It's a comfort reward;

[0020] An agent performs actions in the environment, the environment updates its state based on the agent's actions, and returns the next state s′, instantly rewarding the data of R.

[0021] The state value table V(s) records the value of each state s, which is the expected value of the long-term cumulative reward that can be obtained according to a certain strategy.

[0022] The Bellman equation is used to update the state value table, and the process is expressed as: V(s)←V(s)+α·[R+γ·V(s′)-V(s)], where: V(s) is the value of the current state s, α is the learning rate, which controls the weight of new information, R is the immediate reward, γ is the discount factor, which controls the weight of future rewards, and V(s′) is the value of the current state s. ′ ) is the next state s ′ Value;

[0023] Initialize the state value table V(s), setting the initial value of all states to 0 or a random value. Select an action according to the current policy, randomly select an action for ∈, select the current best action with probability 1-∈, execute the action in the environment, observe the next state s′ and the immediate reward R, update the value V(s) of the current state using the Bellman equation, and repeat the above steps until the state value table converges or the maximum number of iterations is reached.

[0024] In a preferred embodiment, step S2 includes the following:

[0025] Set a threshold for the instant reward R, denoted as Ri. th When R > R th If so, the solution will be marked as an unreasonable solution;

[0026] The judgment results are obtained, and the dynamic allocation prediction analysis is carried out through time series model to further refine and optimize the initial allocation scheme.

[0027] The judgment results include the conclusion that the initial allocation scheme is unreasonable;

[0028] Furthermore, the specific steps for dynamic allocation prediction analysis using time series models are as follows:

[0029] Step B1: Obtain the data for prediction;

[0030] Step B2, create the ARIMAX model;

[0031] Step B3: Use the maximum likelihood estimation (MLE) method to estimate the ARIMAX model parameters;

[0032] Step B4: Verify the effectiveness of the fitted model by checking the goodness of fit of the model using residual analysis.

[0033] Step B5: Use the fitted model to predict the future allocation scheme, and take the maximum value of the prediction result as the highest point of energy demand in the future time period.

[0034] Power demand data refers to historical power demand data, which serves as the main variable in the time series.

[0035] Furthermore, the basic form of the ARIMAX model is:

[0036]

[0037] In the formula, y t Let α be the power demand at the current time point, and α be a constant term. Let θ be the i-th order autoregressive parameter, p be the order of the autoregressive term, and θ be the autoregressive parameter. j Let be the parameter of the j-th order moving average, and q be the order of the moving average term, ∈ t It is a white noise term, representing random error, β k It is an exogenous variable X t-k The coefficient, m is the lag order of the exogenous variable, X t-k It is an exogenous variable lagged by k periods;

[0038] Error term ∈ t Follows a normal distribution N(0,σ) 2 If ), then the likelihood function is:

[0039]

[0040] Taking the logarithm of the likelihood function yields the log-likelihood function:

[0041]

[0042] The parameter estimates μ are obtained by maximizing the log-likelihood function. θ1, β1, β2;

[0043] The initial battery energy consumption allocation strategy is dynamically adjusted based on the maximum, minimum, and average values ​​of the predicted results.

[0044] In a preferred embodiment, step S3 includes the following:

[0045] The state transition probability distribution, where the probability of transitioning to state s′ after taking action a in state s is expressed as P(s′|s,a), in a deterministic environment, is represented as:

[0046] In a stochastic environment, state transitions follow a certain probability distribution: P(s′|s,a)=[p1,p2,...,p n ], where: p1 is the probability of taking action a from state s to transition to state s1, and all probabilities satisfy ∑ i P i =1;

[0047] Environmental complexity C rand Quantified by the entropy of the state transition probability distribution, expressed as: C rand =H(P)=-∑ i p i log(p i );

[0048] The stability S of the reward level is defined as follows: Where ε is a very small constant, σ R It is the standard deviation of the reward;

[0049] Record the cumulative rewards R1, R2, ..., R of the agent over multiple rounds. n Calculate the standard deviation of the reward: Where: R i It is the cumulative reward for the i-th round. It is the average of the cumulative rewards;

[0050] Based on the complexity of the environment C rand Based on the stability S of the reward signal, a comprehensive judgment is made to select either the linear decay method or the exponential decay method, using a comprehensive score, defined as: T = w C ·C rand +w S ·(1-S), where w C w S It's a weighting factor; a threshold θ is set for the overall score. When choosing the exponential decay method: ∈ = ∈ min +(∈ max -∈ min )·e -λ·current_step Otherwise, choose the linear decay method:

[0051] In a preferred embodiment, step S4 includes the following:

[0052] The state is defined as the current vehicle's speed, distance from the vehicle in front, speed of the vehicle in front, and its own dynamic state. The state space representation discretizes the above states to form a finite set of states.

[0053] An action is defined as the action that a vehicle can take;

[0054] The reward function is designed to guide the learning process, ensuring safe following distances and fuel-efficient driving, and includes both safety and efficiency rewards.

[0055] Initialize the Q-table by assigning initial values ​​to the Q-values ​​of all state-action pairs, the learning rate α, and the discount factor γ;

[0056] Experiences are collected through interaction with the environment and stored in an experience pool;

[0057] Sampling and updating: Where: s is the current state, a is the action taken, R is the immediate reward obtained, and s ′ The next state;

[0058] After selecting an action based on the Q-learning algorithm, the power distribution is dynamically adjusted in combination with the current power state of the vehicle and external environmental conditions.

[0059] It continuously monitors the distance and relative speed to the vehicle in front, and adjusts the vehicle speed in real time based on the decisions of the Q-learning algorithm to ensure a safe following distance.

[0060] The intelligent driving simulation method and system based on multi-agent reinforcement learning includes: a data acquisition module, a data classification module, a data matching module, and a feedback adjustment module, with signal connections between the modules;

[0061] Data acquisition module: Designs reward functions based on the agent's safety, efficiency, and comfort, interacts with the environment to obtain immediate rewards, updates the state value table using the Bellman equation, and gradually finds the optimal path;

[0062] Data classification module: Utilizes time series data combined with safety, efficiency, and comfort to predict future vehicle status, performs predictive analysis on power distribution, and dynamically adjusts battery energy consumption distribution strategy based on the prediction results;

[0063] Data matching module: Determines when to choose linear decay method and when to choose exponential decay method based on the complexity of the task and the stability of the reward level;

[0064] Feedback adjustment module: Based on Q-learning, it combines safety and power distribution to adjust vehicle speed and maintain a safe following distance.

[0065] The technical effects and advantages of the intelligent driving simulation method and system based on multi-agent reinforcement learning in this invention are as follows:

[0066] Through multi-agent reinforcement learning, the system can achieve intelligent path planning, power distribution, and speed adjustment, reducing human intervention and improving the level of driving automation. Taking into account safety, efficiency, and comfort, it can achieve multi-objective optimization in complex traffic environments, improving overall driving performance. It dynamically adjusts strategies according to environmental changes, exhibiting strong robustness and adaptability, and performs excellently in diverse driving scenarios. Dynamically adjusting battery energy consumption allocation strategies improves energy utilization efficiency, reduces energy waste, and aligns with sustainable development requirements. Attached Figure Description

[0067] Figure 1This is a schematic diagram of the intelligent driving simulation method based on multi-agent reinforcement learning according to the present invention.

[0068] Figure 2 This is a schematic diagram of the intelligent driving simulation system based on multi-agent reinforcement learning according to the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example 1

[0071] This invention discloses an intelligent driving simulation method and system based on multi-agent reinforcement learning, comprising the following steps:

[0072] Step S1: Design a reward function based on the agent's safety, efficiency, and comfort, interact with the environment to obtain immediate rewards, update the state value table using the Bellman equation, and gradually find the optimal path;

[0073] Step S2: Utilize time series data combined with safety, efficiency, and comfort to predict future vehicle status, perform predictive analysis on power distribution, and dynamically adjust battery energy consumption distribution strategy based on prediction results;

[0074] Step S3: Determine whether to choose the linear decay method or the exponential decay method based on the complexity of the task and the stability of the reward level;

[0075] Step S4: Based on Q-learning, adjust vehicle speed and maintain a safe following distance by combining safety and power distribution.

[0076] In step S1, a reward function is designed based on the agent's safety, efficiency, and comfort. The agent interacts with the environment to obtain immediate rewards, and the Bellman equation updates the state value table, gradually finding the optimal path. Specific details include:

[0077] The reward function is the core of reinforcement learning, guiding the agent's behavior. It influences policy learning by quantifying the quality of the agent's actions. The reward function is derived by comprehensively considering three factors: safety, efficiency, and comfort. Each action of the agent is assigned an immediate reward R, enabling the agent to learn the optimal policy by maximizing cumulative rewards.

[0078] Safety: Avoid collisions and ensure the agent maintains a safe distance from other vehicles or obstacles: Define a safe distance threshold D. safe; Calculate the distance D between the agent and the nearest other vehicle or obstacle. current When the distance is less than the safety threshold, a larger negative reward is given, indicating an increased level of danger; when the distance is greater than the safety threshold, a smaller positive reward is given, indicating a higher level of safety. The safety coefficient is expressed as: When D current <D safe When the distance decreases, the reward decreases quadratically, indicating a rapid increase in the level of danger; when D current ≥D safe When the distance is greater, positive rewards are given smoothly, and the rewards are higher as the distance increases, but the growth gradually becomes smoother. k is an adjustment parameter used to control the growth rate of the function.

[0079] Efficiency: Reach the target point as quickly as possible and reduce travel time: Define the remaining distance D from the agent to the target point. goal Estimate the time required for the agent to reach the target point at its current speed V. When the agent approaches the target point, a larger positive reward is given; when the agent moves far from the target point or its speed is too low, a smaller reward or penalty is given. The efficiency coefficient is expressed as: When T remaining When the agent's speed is greater than 0, a positive reward is smoothly given, with higher rewards for shorter remaining time; when the agent's speed is zero, a large negative reward is given, indicating extremely low efficiency; k1 is an adjustment parameter used to control the growth rate of the function; T goal It is the ideal time for the agent to reach the target point.

[0080] Comfort: Reduce sudden acceleration and sharp turns to improve passenger experience: Calculate the rate of change of acceleration Δa and the rate of change of steering angle Δθ of the intelligent agent. A negative reward is given when the acceleration or steering angle changes are large, and a positive reward is given when the acceleration or steering angle changes are small. The comfort coefficient is expressed as: R comfort =-(w a ·|Δa|+w θ ·|Δθ|), |Δa|: change in acceleration; |Δθ|: change in steering angle; w a and w θ It is a weight used to balance the impact of acceleration and steering angle on comfort.

[0081] The weighted sum of rewards based on safety, efficiency, and comfort forms the final immediate reward, expressed as: R = w s ·R safety +w e ·R efficiency +w c ·R comfort , where: w s ,w e ,wc These are the corresponding weights of safety, efficiency, and comfort, used to balance the importance of safety, efficiency, and comfort; R safety : is a security reward; R efficiency It's an efficiency reward; R comfort It's a comfort reward.

[0082] An agent performs actions in the environment, the environment updates its state based on the agent's actions, and returns the next state s′, instantly rewarding the data of R.

[0083] The state value table V(s) records the value of each state s, representing the expected value of the long-term cumulative reward that can be obtained by following a certain strategy starting from the current state.

[0084] The Bellman equation is used to update the state value table, and the process is expressed as: V(s)←V(s)+α·[R+γ·V(s′)-V(s)], where: V(s) is the value of the current state s, α is the learning rate, which controls the weight of new information, R is the immediate reward, γ is the discount factor, which controls the weight of future rewards, and V(s′) is the value of the current state s. ′ ) is the next state s ′ Value;

[0085] Initialize the state value table V(s), setting the initial value of all states to 0 or a random value. Select an action according to the current policy, randomly select an action for ∈, select the current best action with probability 1-∈, execute the action in the environment, observe the next state s′ and the immediate reward R, update the value V(s) of the current state using the Bellman equation, and repeat the above steps until the state value table converges or the maximum number of iterations is reached.

[0086] In step S2, time series analysis is used to predict future vehicle states based on safety, efficiency, and comfort considerations. This predictive analysis of power distribution is then performed, and the battery energy consumption allocation strategy is dynamically adjusted based on the prediction results. Specific details include:

[0087] Set a threshold for the instant reward R, denoted as Ri. th When R > R th If so, the solution will be marked as an unreasonable solution;

[0088] The judgment results are obtained, and the dynamic allocation prediction analysis is carried out through time series model to further refine and optimize the initial allocation scheme.

[0089] The judgment results include the conclusion that the initial allocation scheme is unreasonable;

[0090] It should be noted that the time series model used in this embodiment is the ARIMAX model. Refining and optimizing the initial allocation scheme refers to making the power allocation scheme more scientific and reasonable through further analysis and adjustment when it is determined that the current initial allocation scheme is unreasonable, thereby achieving control over the internal energy consumption of the battery.

[0091] Furthermore, the specific steps for dynamic allocation prediction analysis using time series models are as follows:

[0092] Step B1: Obtain the data for prediction;

[0093] Step B2, create the ARIMAX model;

[0094] Step B3: Use the maximum likelihood estimation (MLE) method to estimate the ARIMAX model parameters;

[0095] Step B4: Verify the effectiveness of the fitted model by checking the goodness of fit of the model using residual analysis.

[0096] Step B5: Use the fitted model to predict future allocation schemes, and take the maximum value of the prediction results as the highest point of energy demand in the future time period.

[0097] Specifically, the predictive data includes safety, efficiency, comfort, and power demand data for each time point; examples of safety, efficiency, and comfort for each time point have been given in Example 1 and will not be repeated here.

[0098] Power demand data refers to historical power demand data, which serves as the main variable in the time series.

[0099] Furthermore, the basic form of the ARIMAX model is:

[0100]

[0101] In the formula, y t Let α be the power demand at the current time point, and α be a constant term. Let θ be the i-th order autoregressive parameter, p be the order of the autoregressive term, and θ be the autoregressive parameter. j Let be the parameter of the j-th order moving average, and q be the order of the moving average term, ∈ t It is a white noise term, representing random error, β k It is an exogenous variable X t-k The coefficient, m is the lag order of the exogenous variable, X t-k It is an exogenous variable lagged by k periods;

[0102] It should be noted that the exogenous variable section can integrate the impact of other relevant variables (such as safety, efficiency, and comfort) on power demand;

[0103] It should be noted that in step B3, α, θ1, β1, and β2 are obtained by the maximum likelihood estimation (MLE) method. The specific steps are as follows:

[0104] Error term ∈ t Follows a normal distribution N(0,σ) 2 If ), then the likelihood function is:

[0105]

[0106] Taking the logarithm of the likelihood function yields the log-likelihood function:

[0107]

[0108] The parameter estimates μ are obtained by maximizing the log-likelihood function. θ1, β1, β2;

[0109] The initial battery energy consumption allocation strategy is dynamically adjusted based on the maximum, minimum, and average values ​​of the predicted results.

[0110] The following is an example of this embodiment:

[0111] Maximum value of the prediction result This represents the peak of energy demand within a future timeframe. A peak-period allocation strategy is formulated, targeting the peak energy demand period (i.e., close to or equal to the predicted peak). (At the designated time point), increase the battery output power by 20% to ensure smooth vehicle operation and avoid insufficient power;

[0112] Based on the lowest value of the prediction results Or, during periods of low demand, formulate a strategy for allocating resources during these periods, specifically during periods of low or below [the threshold for demand]. (At a certain point in time), reduce the battery's output power by 15% to extend battery life and efficiency;

[0113] Based on the average of the prediction results Develop a balanced power allocation strategy to maintain normal battery output during periods when power demand is close to average, ensuring continuous and stable vehicle operation. During this period, maintain the normal output of the battery to ensure the continuous and stable operation of the vehicle.

[0114] In step S3, the choice between linear decay and exponential decay is determined based on the complexity of the task and the stability of the reward level. The exploration rate is dynamically adjusted using the selected screening subtraction method. Specific details include:

[0115] The state transition probability distribution, where the probability of transitioning to state s′ after taking action a in state s is expressed as P(s′|s,a), is completely deterministic in a deterministic environment, and is expressed as:

[0116] In a stochastic environment, state transitions follow a certain probability distribution: P(s′|s,a)=[p1,p2,...,p n ], where: p1 is the probability of taking action a from state s to transition to state s1, and all probabilities satisfy ∑ i P i =1;

[0117] Environmental complexity C rand Quantified by the entropy of the state transition probability distribution, expressed as: C rand =H(P)=-∑ i p i log(p i The higher the entropy, the more complex the environment; when the entropy is 0, the environment is completely deterministic.

[0118] The stability S of the reward level is defined as follows: Where ε is a very small constant used to avoid the denominator being zero. σ R A larger value indicates greater fluctuation in the reward signal, and a smaller value for S; σ R A smaller value indicates a more stable reward signal, while a larger value indicates a larger reward signal.

[0119] During training, the cumulative rewards R1, R2, ..., Rn obtained by the agent over multiple rounds are recorded. n Calculate the standard deviation of the reward: Where: R i It is the cumulative reward for the i-th round. It is the average of the cumulative rewards.

[0120] Based on the complexity of the environment C rand Based on the stability S of the reward signal, a comprehensive judgment is made to select either the linear decay method or the exponential decay method, using a comprehensive score, defined as: T = w C ·C rand +w S ·(1-S), where w C w S These are weights used to balance the effects of complexity and stationarity, C rand The larger the value of T, the higher the task complexity; the smaller the value, the steeper the reward curve. A larger T value indicates a more complex task or a steeper reward curve. A threshold θ is set for the overall score. When choosing the exponential decay method: ∈ = ∈ min +(∈ max -∈min )·e -λ·current_step Otherwise, choose the linear decay method:

[0121] In step S4, based on Q-learning and combining safety and power distribution, the updated ∈ value is used to adjust the vehicle speed to maintain a safe following distance. Specifically, this includes:

[0122] Step D1: State Definition: Defined as the current vehicle's speed, distance to the vehicle in front, speed of the vehicle in front, and its own dynamic state. State Space Representation: Discretizes the above states to form a finite set of states;

[0123] Step D2: Action Definition: Defined as the actions that the vehicle can take, such as accelerating, decelerating, or maintaining the current speed. The action space includes all possible action options.

[0124] Step D3: Reward Function Design: Design a reward function to guide the learning process, ensuring safe following distance and energy-efficient driving. Safety Reward: A positive reward is given when maintaining an appropriate distance from the vehicle in front; a larger negative reward is given if the distance is too close. Efficiency Reward: A positive reward is given when approaching the target point at an appropriate speed, provided safety is ensured; a negative reward is given for excessive deceleration or acceleration.

[0125] Step D4: Q-learning algorithm application: Initialize the Q-table, assign initial values ​​to the Q-values ​​of all state-action pairs, learning rate α and discount factor γ: Set appropriate learning rate and discount factor to balance the importance of immediate rewards and future rewards.

[0126] Step D5: Training process: Collect experience (state, action, reward, next state) through interaction with the environment and store it in the experience pool;

[0127] Sampling and updating: Where: s is the current state, a is the action taken, R is the immediate reward obtained, and s ′ This is the next state.

[0128] Step D6: Power Distribution Adjustment: After selecting an action based on the Q-learning algorithm, the power distribution is dynamically adjusted in combination with the current power status of the vehicle and external environmental conditions to optimize energy consumption and driving efficiency.

[0129] Step D7: Maintaining a safe following distance: During driving, the system continuously monitors the distance and relative speed to the vehicle in front, and adjusts the vehicle speed in real time based on the decision of the Q-learning algorithm to ensure a safe following distance.

[0130] The intelligent driving simulation method and system based on multi-agent reinforcement learning includes: a data acquisition module, a data classification module, a data matching module, and a feedback adjustment module, with signal connections between the modules;

[0131] Data acquisition module: Designs reward functions based on the agent's safety, efficiency, and comfort, interacts with the environment to obtain immediate rewards, updates the state value table using the Bellman equation, and gradually finds the optimal path;

[0132] Data classification module: Utilizes time series data combined with safety, efficiency, and comfort to predict future vehicle status, performs predictive analysis on power distribution, and dynamically adjusts battery energy consumption distribution strategy based on the prediction results;

[0133] Data matching module: Determines when to choose linear decay method and when to choose exponential decay method based on the complexity of the task and the stability of the reward level;

[0134] Feedback adjustment module: Based on Q-learning, it combines safety and power distribution to adjust vehicle speed and maintain a safe following distance.

[0135] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0136] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0137] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent driving simulation method based on multi-agent reinforcement learning, characterized in that; The method comprises the steps of: Step S1: Designing a reward function according to the safety, efficiency and comfort of the agent, interacting with the environment to obtain an immediate reward, updating the state value table through the Bellman equation, and gradually finding the optimal path; Step S2: Predicting the future state of the vehicle by using time series combined with safety, efficiency and comfort, and predicting and analyzing the power distribution based on the prediction results to dynamically adjust the battery energy consumption distribution strategy; Step S3: Deciding when to choose linear decay method or exponential decay method according to the complexity of the task and the smoothness of the reward degree; Step S4: Adjusting the vehicle speed to maintain a safe distance based on Q-learning combined with safety and power distribution; Obtaining a judgment result, predicting and analyzing the power distribution through a time series model, and further refining and optimizing the initial distribution scheme; The judgment result includes an unreasonable initial distribution scheme; The specific steps of predicting and analyzing the power distribution through the time series model are as follows: Step B1, obtaining prediction data; Step B2, establishing an ARIMAX model; Step B3, using the maximum likelihood estimation (MLE) method to estimate the ARIMAX model parameters; Step B4, verifying the effect of the fitted model, and checking the goodness of fit of the model through residual analysis method; Step B5, using the fitted model to predict the future distribution scheme, and taking the maximum value of the prediction result as the highest point of power demand in the future time period; The power demand data refers to the historical power demand data, which is used as the main variable of the time series; Further, the basic form of the ARIMAX model is: In the formula, For the current point in time, For constant terms, Let be the i-th order autoregressive parameter. It is the order of the autoregressive term. Let be the parameters of the j-th moving average. It is the order of the moving average term. It is a white noise term, representing random error. It is an exogenous variable coefficient, It is the lag order of the exogenous variable. It is an exogenous variable lagged by k periods; error term subjection to normal distribution The likelihood function is then Take the logarithm of the likelihood function to get the log-likelihood function: The parameter estimates are obtained by maximizing the log-likelihood function , , , , ; Through the maximum value, the minimum value and the average value of the prediction result, the initial battery energy consumption distribution strategy is dynamically adjusted.

2. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 1, wherein: The reward function is obtained by comprehensively considering safety, efficiency and comfort, and each action of the agent is assigned an immediate reward R: Defining a safety distance threshold , calculating the distance of the agent from the nearest obstacle , the safety coefficient is expressed as: ; defining a remaining distance of the agent to the target point ; estimating a time required for the agent to reach the target point at the current speed V ; The efficiency coefficient is expressed as: wherein: is a tuning parameter, is the ideal time for the agent to reach the target point; a rate of change of acceleration of the computing agent a rate of change of steering angle a comfort coefficient is expressed as: a: a change amount of acceleration; : a change amount of steering angle; and is a weight for balancing the effects of acceleration and steering angle on comfort; The final instant reward is expressed as a weighted sum of the rewards for safety, efficiency, and comfort: where: , , are corresponding weights for safety, efficiency, and comfort, used to balance the importance of safety, efficiency, and comfort; : is the safety reward; is the efficiency reward; is the comfort reward.

3. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 2, wherein: The agent performs actions in the environment, and the environment updates the state according to the actions of the agent and returns the next state s', the data of the immediate reward R; The state value table is updated through the Bellman equation, which records the value of each state s, indicating the expected value of the long-term cumulative reward that can be obtained from the current state according to a certain strategy; The update of the state-value table relies on the immediate reward and the value of the next state, specifically through the following equation: where: V(s) is the value of the current state s, a is the learning rate, controlling the weight of the new information, R is the immediate reward, is the discount factor, controlling the weight of future rewards, is the value of the next state . Initialize the state-value table V(s) with all states having an initial value of 0 or a random value, select an action according to the current policy, randomly select an action with probability select the action that is currently optimal, perform the action in the environment, observe the next state s' and immediate reward R, update the value of the current state V(s) using the Bellman equation, repeat the above steps until the state-value table converges or a maximum number of iterations is reached.

4. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 3, wherein: A threshold is set for the immediate reward R denoted as When R > Rth then the scheme is marked as an unreasonable scheme.

5. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 1, wherein: The state transition probability distribution, the probability of transitioning to state s' after taking action a in state s, is denoted as: P(s' | s, a). In a deterministic environment, the state transition is denoted as: In a stochastic environment, state transitions follow a certain probability distribution: P(s′|s, a) = [ , ,..., ],in: From state s, take action a to transition to state s. The probability of all probabilities satisfies =1; Complexity of the environment Quantified by the entropy of the state transition probability distribution, denoted as: ; The smoothness S of the degree of the computation of the reward is defined by the smoothness score S = 1 - ε / σ where ε is a very small constant, is the standard deviation of the reward; record the cumulative rewards obtained by the agent over multiple episodes , ,…, , compute the standard deviation of the rewards: where: is the cumulative reward of the ith episode, is the average of the cumulative rewards.

6. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 5, wherein: According to the complexity of the environment and the smoothness S of the reward signal, a comprehensive judgment is made to select the linear decay method or the exponential decay method, which is expressed by a comprehensive score, defined as: wherein , is a weight, and a threshold value is set for the comprehensive score , When the comprehensive score is less than the threshold value, the exponential decay method is selected: ; Otherwise, select linear falloff: .

7. The intelligent driving simulation method based on multi-agent reinforcement learning according to claim 6, wherein: The state is defined as the current vehicle speed, distance to the front vehicle, front vehicle speed, and power state of the vehicle itself. The state space is discretized to form a finite set of states. The action is defined as the actions that the vehicle can take. The reward function is designed to guide the learning process and ensure safe following distance and energy-efficient driving, including safety rewards and efficiency rewards. The Q-table is initialized, and the Q-values for all state-action pairs are assigned initial values, learning rate α, and discount factor γ. Experience is collected through interaction with the environment and stored in the experience pool. After selecting an action based on the Q-learning algorithm, the power distribution is dynamically adjusted based on the current vehicle power state and external environmental conditions. The distance to the front vehicle and relative speed are continuously monitored, and the vehicle speed is adjusted in real time based on the decision of the Q-learning algorithm to ensure safe following distance. The system includes: a data acquisition module, a data classification module, a data matching module, and a feedback adjustment module, with signal connections between the modules. The data acquisition module designs a reward function based on the safety, efficiency, and comfort of the agent, interacts with the environment to obtain immediate rewards, updates the state value table using the Bellman equation, and gradually finds the optimal path. The data classification module uses time series combined with safety, efficiency, and comfort to predict future vehicle states, performs predictive analysis on power distribution, and dynamically adjusts the battery energy consumption distribution strategy based on the prediction results. The data matching module determines when to choose linear decay or exponential decay based on the complexity of the task and the smoothness of the reward level. The feedback adjustment module adjusts the vehicle speed based on Q-learning to maintain a safe following distance. ​ ​ Sampling and update: where: s is the current state, a is the action taken, R is the immediate reward obtained, is the next state; ​ ​ 8. An intelligent driving simulation system based on multi-agent reinforcement learning, for implementing the intelligent driving simulation method based on multi-agent reinforcement learning according to any one of claims 1-7, characterized in that, ​ ​ ​ ​ ​ ​

Images