Modeling and flow control method of traffic flow system

Abstract

The invention generally relates to establishment of a traffic flow system model and a reinforcement learning traffic flow control method based on a Lyapunov value function. Comprises: establishing a mathematical model of a traffic flow system; constructing an environment simulator, and converting a boundary control problem of the system into a Markov process; and designing an SAC reinforcement learning algorithm based on a Lyapunov value function, training an Actor-Critic network through interaction with an environment simulator, and realizing flow control of the traffic flow system through an optimal controller output by the Actor network.

Description

Technical Field

[0001] The present invention generally relates to the establishment of a traffic flow system model and a reinforcement learning traffic flow control method based on a Lyapunov value function. Background Art

[0002] In traffic flow, the driving state of vehicles is affected by many factors, such as the speed change of the vehicle in front, the specific conditions of the road, and the personal behavior of the driver. In order to predict the changes in traffic flow in the future, we can use the KdV-Burgers (KdVB) equation model, which can capture the nonlinear effects and fluctuation characteristics in traffic flow, so as to more accurately describe the flow process within the traffic flow. At the same time, it provides strong data support for traffic management and control measures. Traditional traffic flow control methods usually rely on accurate modeling of traffic systems. Based on the model, engineers can design controllers to optimize traffic flow, reduce congestion, and improve road capacity.

[0003] However, the uncertainty or difficult-to-quantify factors that may exist in practice make model building for some problems more difficult. In contrast, reinforcement learning algorithms have unique advantages. Through interaction with the environment, reinforcement learning algorithms learn from actual experience and gradually improve their own strategies through exploration and utilization without prior knowledge of the specific model of the environment. In the scenario of traffic flow control, the reinforcement learning algorithm can be regarded as an intelligent agent that observes traffic conditions (such as vehicle density, speed, etc.) and selects the optimal action based on the current state (such as adjusting the duration of traffic lights, limiting the speed of entrances and exits, etc.), and learns the best strategy through actual experience. Therefore, reinforcement learning, as a method that does not rely on model knowledge, has great advantages in practical problems.

[0004] In summary, reinforcement learning, as a method that does not rely on model knowledge, has great advantages in practical problems such as traffic flow control. It can learn the best strategy through interaction with the environment and adapt to complex and uncertain environments. Therefore, it is necessary to establish a traffic flow model and design a controller through reinforcement learning algorithm to control the stability of the traffic flow system. Summary of the invention

[0005] The present invention generally relates to the establishment of a traffic flow system model and a reinforcement learning traffic flow control method based on a Lyapunov value function, so as to solve the system modeling and stabilization problems of partially unknown traffic flow system models in practical engineering problems.

[0006] The technical solution of the present invention is:

[0007] 1) Establishment of traffic flow system model based on KdVB equation.

[0008] Specifically, the one-dimensional KdVB equation is used to model the traffic flow system, and the traffic flow at the boundary is adjusted by setting a controller at the boundary to control the stability of the traffic flow system. The traffic flow problem is converted into a boundary control problem of the KdVB equation.

[0009] 2) Design a reinforcement learning traffic flow control method based on Lyapunov value function.

[0010] In order to overcome the defects of the prior art and further solve the boundary control problem of the traffic flow system, the present invention provides a reinforcement learning traffic flow control method based on the Lyapunov value function. Specifically, it includes:

[0011] Step 1. Represent the boundary control problem of the traffic flow system as a Markov decision process. And deal with the continuous space and time domains through discretization approximation. This ensures the rationality of the reinforcement learning algorithm;

[0012] Step 2. Establish a simulation environment simulator for the traffic flow model and collect training data. Specifically, by discretizing the model in step 1, the environment is established as the state s at time t+1 t+1 The state vector s at time t t and control input u t and continuously obtains the system state at each moment as the controller interacts with the environment.

[0013] Step 3. Train the Actor-Critic network. Use the Critic network to approximate the value function and evaluate the quality of the current strategy. Based on the evaluation results, update the parameters of the Critic network to improve the approximation accuracy of the value function. According to the updated value function, adjust the parameters of the Actor network and optimize the strategy. The controller interacts with the system environment under the optimized strategy and collects new data. Repeat the above steps to update the network parameters until the predetermined number of updates is reached and the parameters are updated. Finally, the fully trained Actor network will be able to output the optimal control strategy for the controller.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1) The use of reinforcement learning algorithms to design controllers does not require complete model knowledge, reducing the requirement for prior knowledge of the system. Therefore, the controller design process becomes more flexible and can better adapt to various uncertain factors. Through continuous trial and error and learning, reinforcement learning algorithms can gradually optimize control strategies, thereby achieving effective control of complex systems.

[0016] 2) Design a SAC algorithm based on the Lyapunov value function to stabilize the system more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 .Traffic flow system images;

[0018] Figure 2 .Block diagram of KdVB equation stabilized by SAC algorithm based on Lyapunov value function;

[0019] Figure 3 .A diagram of the framework for designing a border controller using a reinforcement learning algorithm;

[0020] Figure 4 .Flowchart of SAC algorithm based on Lyapunov value function;

[0021] Figure 5 . The image of the stabilized KdVB equation when the parameters are set to a=1, b=0.5, c=0.2. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings.

[0023] 1. Establishment of traffic flow model based on KdVB equation

[0024] Figure 1 The image shows a simple traffic flow system. Vehicles travel from x = 0 to x = L. u(x, t) represents the flow rate of the traffic flow system, u0 represents the expected flow rate of the traffic flow system, z(x, t) = u(x, t) - u0, represents the difference between the traffic flow rate and the expected flow rate, where t is the time. Our goal is to stabilize the traffic flow to the expected flow rate u0.

[0025] The model of the error system can be expressed by the following KdVB equation:

[0026]

[0027] Where x∈[0,L], t>0, at x=0, it means setting a ramp and controlling traffic flow by adjusting the ramp. At x=L, the boundary condition z is set x (L,t)=z xx (L, t) = 0 indicates the restriction on vehicles at the exit. In addition, the initial condition z(x, 0) = z0(x) is set to indicate the system state at time 0. a, b, c are all constant coefficients.

[0028] 2. Deep Reinforcement Learning Controller Design

[0029] The present invention adopts a reinforcement learning algorithm. The training goal of reinforcement learning is to maximize the cumulative discounted reward of the strategy. In the present invention, when the system state is s t Select control input u t, the corresponding instantaneous reward function is defined as r t (s t ,u t )=-||s t -s * || 2 , where s * is the desired system state. At this time, the cumulative discounted reward is maximized to obtain the maximum value of a non-positive function. After the neural network outputs the control signal, the cumulative discounted reward is fed back to the neural network, and the above steps are repeated to train the neural network to maximize the cumulative reward. Control is performed based on the final optimal strategy.

[0030] The reinforcement learning speed control method based on Lyapunov value function provided by the present invention uses the SAC algorithm in the reinforcement learning algorithm. The algorithm aims at the problem of over-estimation of action value function and adds the double Q network idea of ​​DoubleDQN to make the algorithm more stable.

[0031] The specific implementation steps are as follows:

[0032] Step 1. Convert the boundary control problem into a Markov decision process.

[0033] Figure 3 A framework for designing boundary controllers using reinforcement learning algorithms is described. The boundary control problem for the system can be expressed as a Markov decision process. The system is discretized and the discretization steps dx = L / (M-1) and dt = T / (N-1) are chosen to discretize the spatial domain [0,L] and the temporal domain [0,T]. This allows us to express the solution as a piecewise constant function in each interval. The discretized system model can be expressed in the following way:

[0034] s t+1 =f(s t ,u t ),

[0035] where s t represents the system state vector at time t

[0036]

[0037] u t Represents boundary control input

[0038]

[0039] The function f describes the dynamics of the system in discrete form.

[0040] Therefore, the process of system changes over discrete time can be described as a Markov decision process

[0041] s t+1 ~P(s t+1 |s t ,u t ),

[0042] Where P(s t+1 |s t ,u t ) represents the state s t In action t Under the action of t+1 The transition probability.

[0043] Step 2. Establish a simulation environment simulator for the traffic flow model based on the KdVB equation and collect training data.

[0044] Specifically, by discretizing the model in step 1, the environment is established as the state s at time t+1 t+1 The state vector s at time t t and control input u t and continuously obtains the system state at each moment as the controller interacts with the environment.

[0045] Here we need to explain the parameter settings in the environment: ① The main purpose of setting the model parameters is to build a complete system dynamics model, and then provide a realistic environment simulator for algorithm training. This simulator can simulate the situation in the actual scene, that is, we can only obtain the measurement data of the system, but cannot directly know the model parameters of the system. In practical applications, these model parameters are often unknown, and the method proposed in the present invention does not need to rely on these known model parameters to train the controller. This means that even if we do not understand the internal mechanism of the system, we can train an effective controller through our method. ② The simulation step size can be set to dx=0.2, dt=0.001, and can also be adjusted according to actual needs. It should be noted that the step size parameter should not be too large to prevent the algorithm from becoming unstable and the estimation accuracy from decreasing.

[0046] In the current strategy Under the condition of u, the training trajectory data is collected through the interaction between the controller and the environment simulator. t From the normal distribution N(μ(s t ),σ 2 (s t )) is generated, the mean μ(s t ) and standard deviation σ 2 (s t ) is calculated by a deep neural network (DNN) parameterized by θ. t After being applied to the system environment, the state s of the next time step is obtainedt+1 And instant rewards t+1 When a round of interaction between the controller and the environment simulator ends, the data pairs corresponding to all time steps in this round (s t ,u t ,s t+1 ,r t+1 ) is stored in buffer D j Buffer D j will include k rounds of data.

[0047] Step 3. Train the Actor-Critic Network

[0048] Reference Figure 2 , the algorithm contains two neural networks: Actor network (strategy network) and Critic network (value network).

[0049] The Actor network receives the state data of the environment simulator at a certain moment and generates the corresponding control action; the Critic network estimates the value function and evaluates the current strategy learned by the Actor network.

[0050] The Critic network includes two main Q-value function networks, Q-network1 (Q w1 ): used to estimate the Q value of the current state and action; Q-network2(Q w2 ): As Q w1 The auxiliary network is used to provide additional Q-value estimation. These two main Q-value function networks are used to evaluate the Q-value of the current state and action to guide the update of the policy network and make the training of the Actor more stable; two target Q-value function networks, TargetQ-network1 (Q wt1 ): used to estimate the target Q value of the next state; TargetQ-network2(Q wt2 ): As Q wt1 The auxiliary network is used to provide additional target Q value estimates. The two target Q value function networks are used to calculate the dual Q learning objectives in the SAC algorithm. Each time the main Q value function network is updated, the parameters of the main Q value function network are gradually updated to the target Q value function network by soft update to provide a more stable and reliable target value. By using the above four Q networks, the SAC algorithm can reduce the estimation error of the target value and improve the performance and convergence speed of the algorithm.

[0051] The network objective function of the main Q-value function in the updated Critic network is defined as:

[0052]

[0053] Among them, Q ω is the main Q-value function network, R(s t ,u t ) is the cumulative discount reward, is the target Q-value function network, π(·|s t+1 ) is the policy function.

[0054] The Actor network is iteratively updated based on the Critic network's estimate of the value function. The objective function for updating the Actor network parameters is defined as:

[0055]

[0056] Among them, Q ω is the main Q-value function network, the policy function π θ (f θ (∈ t ;s t )|s t ),∈ t ~N(0,1).

[0057] The Actor-Critic network model is trained using the collected training data.

[0058] After the parameters of the Actor-Critic network have been updated N times, the optimal strategy learned is the optimal control strategy of the controller. The control strategy is applied to the actual system to control the system by controlling the behavior of the actuator. According to the feedback information of the system, the output of the controller is adjusted in real time to adapt to the changes of the system and optimize the control effect. The specific algorithm flow can be found in Figure 4 .

[0059] The effect of the temperature control method provided by the present invention is verified below.

[0060] Assume the initial condition z(x,0) = z0(x) = sin((x+0.5π) / L), and use the following data as the initialization conditions of the experimental setting: s(0) = [z(0,t), z(dx,t), ..., z(L,t)] = [sin(0.5π / L), sin((dx+0.5π) / L), ..., sin((L+0.5π) / L)], and set the following parameters: total time length T = 200, steady state s * =0, and set the controller to interact with the system environment for 2,000,000 steps, with every 400 steps as a sample to ensure the convergence of the RL algorithm.

[0061] After conducting experiments with the above data, Figure 5The graph depicts the change of the system state over time when the parameters are set to a=1, b=0.5, c=0.2. It can be observed that the controllers proposed in the present invention successfully stabilize the state to a steady-state value. The above two sets of implementation examples both prove that the reinforcement learning method based on the Lyapunov value function proposed in the present invention can effectively stabilize the nonlinear KdVB system, thereby achieving the stabilization of the traffic flow system.

Claims

1. A traffic flow system modeling and control method, comprising: Establish a traffic flow system model based on the nonlinear KdVB equation; Build a system environment simulator to transform the system boundary control problem into a Markov process; Based on the reinforcement learning algorithm of Lyapunov value function, a controller is designed. The Actor-Critic network is trained through interaction with the environment simulator, and the traffic flow is controlled by the optimal controller output by the Actor network.

2. The traffic flow system modeling method according to claim 1, characterized in that: The nonlinear KdVB mathematical model is used to describe the flow process within the traffic flow. The model can capture the nonlinear effects and fluctuation characteristics in the traffic flow. The model includes: z t (x,t)+az xx (x,t)+bz xxx (x,t)+cz(x,t)z x (x,t)=0, Initial condition z(x,0)=z0(x), boundary condition z x (L,t)=z xx (L, t) = 0, controller z(0, t) = U(t); model variables: system state z(x, t); model parameters: constant coefficients a, b, c.

3. The speed control method of the traffic flow system according to claim 1, characterized in that: Using a SAC reinforcement learning algorithm based on the Lyapunov value function, a controller is designed to stabilize the nonlinear KdVB system of claim 1, comprising the following steps: Step 1. Convert the system boundary control problem into a Markov decision process; Step 2. Establish a simulation environment simulator for the nonlinear KdVB model, and use the current strategy to make the controller interact with the system environment simulator to collect training data, including applying actions to the system through the actuator and observing the state and reward signal of the system; Step 3. Train the Actor-Critic network, use the Critic network approximate value function, evaluate the current strategy to measure the quality of the strategy, and update the parameters of the Critic network to improve the approximation effect of the value function; according to the value function, update the parameters of the Actor network, update and optimize the strategy; the controller interacts with the system environment again under the updated strategy and re-collects data; repeatedly update the network parameters until the predetermined number of parameter updates is reached; at this time, the strategy learned by the trained Actor network is the optimal control strategy of the controller.

4. The speed control method of the traffic flow system according to claim 3, characterized in that: The reinforcement learning algorithm includes a SAC algorithm based on the Lyapunov value function. The neural network used in the algorithm includes an Actor network and a Critic network. The instant reward function used in the algorithm is a negative L 2 norm, defined as r t (s t ,u t )=-||s t -s * || 2 , where s * The desired system state.

5. The discretization system according to claim 3, characterized in that: Discretize the system and select the discretization step size dx = L / (M-1) and dt = T / (N-1) to discretize the spatial domain [0, L] and the time domain [0, T]. This allows us to express the solution as a piecewise constant function in each interval. The discretized system model can be expressed as follows: t+1 =f(s t ,u t ), including the characteristics of: s t represents the system state vector at time t u t Represents boundary control input And the function f(s t ,u t ), which describes the dynamics of the system in discrete form.

6. The discretization system according to claim 3, characterized in that: Establish the environment as the state s at time t+1 t+1 The state vector s at time t t and control input u t The relationship between the system and discrete time can be described as a Markov decision process s t+1 ~P(s t+1 |s t ,u t ), where P(s t+1 |s t ,u t ) represents the state s t In action t Under the action of t+1 The transition probability.

7. The Critic network according to claim 3, characterized in that: The gradient descent method is used to minimize the difference between the estimate of the value function and the cumulative discounted reward, and the main Q-value function in the Critic network is updated. The network objective function is defined as: Among them, Q ω is the main Q-value function network, R(s t ,u t ) is the cumulative discount reward, is the target Q-value function network, π(·|s t+1 ) is the policy function.

8. The Actor network according to claim 3, characterized in that: The gradient ascent algorithm is used to maximize the policy gradient, and the objective function for updating the Actor network parameters is defined as: Among them, Q ω is the main Q-value function network, the policy function π θ (f θ (∈ t ;s t )|s t ),∈ t ~N(0,1).

Images