A bus on-the-go reinforcement learning speed control method for epidemic prevention and control

Abstract

The present invention discloses a method for controlling the speed of buses in transit by reinforcement learning for epidemic prevention and control, the method comprising: sensing real-time road traffic information of bus lines through roadside facilities, dynamically acquiring real-time operation information of buses, establishing a time-continuous bus operation simulation system according to real bus lines, and providing a simulated environment for reinforcement learning training; establishing a bus in transit driving state transfer model, establishing a reinforcement learning framework based on the simulation system, using a neural network to output specific actions in a continuous space in the reinforcement learning framework, the neural network calculates the driving speed of the bus to the next stop according to the running state of the bus, and effectively controls the running speed of the bus in transit, ultimately making the headway time distance between buses more uniform, and further making the distribution of passengers in multiple buses more balanced, thereby reducing the congestion in the bus compartment, balancing the vehicle load rate, and having a better operating effect under the epidemic prevention and control environment.

Description

Technical Field

[0001] The present invention relates to the fields of traffic control, simulation and reinforcement learning, and in particular to an on-the-way reinforcement learning speed control method for buses for epidemic prevention and control. Background Art

[0002] Public transportation is an important mode of transportation. If it cannot be fully utilized, it will have a great impact on meeting people's travel needs. The number of passengers on the bus reflects the risk of epidemic transmission to a certain extent. By balancing the number of passengers on different buses and making the distribution of passengers more even, it can provide passengers with satisfaction with public transportation, increase passengers' willingness to take the bus, balance the full load rate of different buses, and objectively reduce the risk of epidemic transmission.

[0003] Reinforcement learning is an important area of ​​artificial intelligence research in recent years, and it has universal applicability to research problems. The model is trained by the agent interacting with the environment. Computer simulation programs provide a convenient environment for reinforcement learning to solve practical problems. The trained model has a high feedback speed in actual use, so it has a unique advantage in solving the on-the-way speed control of buses for epidemic prevention and control. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method for real-time control of bus running speed that is helpful for epidemic prevention and control.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A bus on-the-way reinforcement learning speed control method for epidemic prevention and control includes the following steps:

[0007] S1. Set optimization goals for epidemic prevention and control: make the number of passengers on each bus average and reduce crowding;

[0008] S2. Obtain real-time bus operation information and road traffic conditions through traffic information detectors; establish a time-continuous bus operation simulation system based on the real bus operation network;

[0009] S3. Establish a Markov model, determine the objective function, and build a deep deterministic policy gradient reinforcement learning framework;

[0010] S4. Perform training in the bus operation simulation system to optimize the neural network parameters in the reinforcement learning framework. Whenever a bus on the road enters a bus station, select the 5 vehicles in front and behind the bus to form a group, select the bus information in the group as the state parameter and pass it into the reinforcement learning framework to calculate the running speed of the next section of the bus. When the bus enters the next station, calculate the variance of the headway time of all vehicles in the group as the reward feedback of the last control result.

[0011] S5. Repeat the process of training and optimizing the parameters of the neural network in the reinforcement learning framework in S4 in the bus operation simulation system; save the neural network structure and parameters after the training; evaluate the control effect of the neural network trained in S4 in the bus operation simulation system: when the controlled bus enters the station, select a group of 5 vehicles before and after the bus, and transmit the bus status information in each group as input to the trained and saved neural network. The running speed of the bus before the next stop is calculated by the neural network, and compared with the case without control, it is obtained that the number of passengers in all buses is more evenly distributed.

[0012] Furthermore, in step S2, based on the intelligent on-board system technology and the intelligent roadside system technology, a real bus route is simulated through a bus operation simulation system, and the bus route is divided into several sections according to bus stops; a statistical regression analysis is performed on the bus operation data collected by the traffic detector, and different maximum bus operating speed limits are set for different sections in the bus operation simulation system according to the historical traffic congestion conditions and road traffic conditions of different sections. In the bus operation simulation system, the maximum operating speed limits are randomly set in different sections according to the distribution obtained from the statistical regression analysis, and the random maximum operating speed limits can reflect the differences in real road traffic conditions in different sections.

[0013] Furthermore, the bus operation simulation system updates the information related to the buses on the controlled route every second, including the operation status of the buses, the bus positions and their cruising speeds.

[0014] Furthermore, in step S3, a deep deterministic policy gradient reinforcement learning framework is selected. On the one hand, the continuity of the final output result of the neural network is guaranteed, which is more in line with the actual situation that the running speed of the bus is continuous rather than discrete; on the other hand, the actor-critic framework is adopted, and the experience replay mechanism is introduced to reduce the continuity after several training data, thereby improving the robustness of the training results.

[0015] Furthermore, the reinforcement learning framework established in step S3 includes four networks, namely, the Actor reality network, the Actor estimation network, the Critic reality network, and the Critic estimation network; the Actor reality network and the Actor estimation network are responsible for obtaining a certain action by inputting state variables, and the established neural network structure is as follows: the input of the neural network is the state of the environment; the output of the neural network is the action to be taken by the agent: the input accepted by the Actor reality network is s_, which represents the state of the next environment; the input accepted by the Actor estimation network is s, which represents the state of the current environment. During the training process, the parameters of the Actor estimation network are optimized, and the parameters of the Actor reality network are updated after several steps of training;

[0016] The Critic reality network and the Critic estimation network are responsible for evaluating the current state and the actions taken; the inputs of the Critic reality network and the Critic estimation network are the state variables of the simulation environment and the actions taken in the last decision, and the output is q, which refers to the score value obtained under the corresponding simulation environment state and action conditions; the input accepted by the Critic estimation network is [s, a], which represents the state variables and corresponding actions of the simulation environment at the current moment; the input accepted by the Critic reality network is [s_, a_], which represents the state variables and corresponding actions of the simulation environment at the next moment;

[0017] The parameter optimization process of the Actor Estimation Network is carried out in combination with the Critic Real Network and the Critic Estimation Network. The parameter optimization process of the Actor Estimation Network first obtains the action predicted by the Actor Estimation Network through the current environment, then adds random noise interference to the predicted action to prevent overfitting, and passes the action with added noise to the Critic Estimation Network to obtain its score Value. The gradient back propagation optimization direction is to maximize the score Value.

[0018] Furthermore, in step S4, when the bus to be controlled enters any station, the judgment of whether to conduct training is started. When the number of buses before and after the bus is ≥ 5, the 5-bus formation closest to the bus is selected, and the running status of the vehicles in the group is counted, including: the running speeds of the 5 buses: speed_1, speed_2, speed_3, speed_4, speed_5, the headway time between the 5 buses: headway_1, headway_2, headway_3, headway_4; and the number of stops between the 5 buses: stopnum_1, stopnum_2, stopnum_3, stopnum_4, the position num of the bus in the group, and the above 14 data are counted. It is converted into a one-dimensional vector and passed into the reinforcement learning framework as the initial state parameter state, where the Actor estimation network in the reinforcement learning framework calculates the optimal driving speed of the bus before it departs from the station and arrives at the next station. Under the premise of not exceeding the maximum allowable driving speed of the section, the bus is ordered to go to the next station at the optimal driving speed calculated by the actor estimation network; when the bus arrives at the next station at the above driving speed, the 5 buses closest to the bus are selected to form a group, and the variance of the estimated headway time of the 5 buses in the group is calculated as the evaluation result of the last driving speed control behavior. At the same time, [initial state state, driving speed a calculated by the Actor estimation network, evaluation result r] is saved as a set of training data in the experience pool for subsequent neural network parameter training.

[0019] Furthermore, in step S1, the epidemic prevention and control objective function is set as:

[0020]

[0021] Among them, minx is to minimize the sum of the squares of the number of passengers of all buses on each section; n aij is the number of passengers in bus a on road section ij; in view of the different risk levels in different regions, the full load rate limit in the bus operation simulation system is increased according to the actual situation; by fine-tuning the parameters in the bus operation simulation system and the internal parameters of the reinforcement learning framework, the on-the-way reinforcement learning speed control method for buses is suitable for speed control of different types of buses in areas with different risk levels.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the in-transit reinforcement learning speed control method for buses for epidemic prevention and control.

[0023] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the in-transit reinforcement learning speed control method for buses for epidemic prevention and control are implemented.

[0024] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0025] 1. The present invention uses a computer to operate a bus operation simulation system to learn and optimize the neural network parameters in a reinforcement learning framework, ultimately making the headway time between buses more uniform. At the same time, it also makes the distribution of passengers in multiple buses more balanced, thereby reducing congestion in the bus compartment and balancing the vehicle load rate. It has better operating results under the epidemic prevention and control environment, and creatively proposes quantitative indicators for evaluating the spread of the epidemic in buses. The real-time control method of operating speed proposed in the present invention can reduce the quantitative indicators by 15%.

[0026] 2. The parameters of the neural network in the reinforcement learning framework are softly replaced by the real network and the estimated network during the learning process: Since the real network will not be directly updated by the optimizer, the estimated network is updated to the real network at a certain ratio. The information generated by the bus operation simulation system in the past period of time is saved in the memory bank. By randomly obtaining data from the memory bank for training, the continuity of the data is broken, which has a good training effect. After the training is completed, the network structure and parameters of the Actor estimation network after training are retained. In the final experimental result verification process, the state variable state of the bus that needs to be controlled is passed to the Actor estimation network, and the Actor estimation network outputs the optimal driving speed of the bus in the next section. When all buses on the bus route are controlled, the objective function value will be significantly improved compared to when no control is performed.

[0027] 3. The present invention selects a deep deterministic policy gradient reinforcement learning framework, which, on the one hand, ensures the continuity of the output results of the neural network, making it more in line with the actual operating speed of the bus; on the other hand, the Actor and Critic dual network structure within the framework, as well as the experience replay mechanism, can reduce the continuity in multiple trainings, improve the effectiveness of the training results, and have a better training effect.

[0028] 4. In the method of the present invention, different maximum allowable speeds are set in the bus operation simulation system to represent the randomness of road traffic conditions, which can improve the robustness and applicability of the algorithm and make the training environment more consistent with the actual bus operation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the implementation of the on-the-go reinforcement learning speed control method for buses for epidemic prevention and control.

[0030] Figure 2 This is a statistical chart of the example test of the on-the-go reinforcement learning speed control method for buses for epidemic prevention and control. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] like Figure 1 As shown, an embodiment of the present invention provides a bus on-the-way reinforcement learning speed control method for epidemic prevention and control, comprising the following steps:

[0033] S1. Average number of passengers on each bus, reduce crowding, and set optimization goals for epidemic prevention and control;

[0034] The objective function determined as the epidemic prevention and control indicator is:

[0035]

[0036] Where minx is the sum of the squares of the number of passengers on each section of the bus that minimizes the service; n aij is the number of passengers in bus a on section ij. In the setting of the objective function, the bus route is segmented according to the bus stops on the entire route. By summing the square of the number of passengers on all sections of the vehicles that have completed the service, the number of passengers on each bus on each section is guaranteed to be more uniform. The square of the number of passengers in the car is used to characterize the real-time congestion inside the vehicle, and the congestion directly corresponds to the risk of virus transmission in epidemic prevention and control work. Therefore, setting this objective function as the optimization target can average the distribution of all passengers in different buses and reduce the risk of epidemic transmission by reducing the congestion in the bus.

[0037] S2. Obtain real-time bus operation information and road traffic conditions through traffic information detectors. Establish a time-continuous bus operation simulation system based on the real bus operation network;

[0038] The bus operation simulation system is time-continuous and can simulate the operation of buses on a single bus route. The historical data of bus road driving is collected through roadside information collection sensors and on-board intelligent sensors. The bus operation simulation system is established based on the collected data, and the bus is controlled on the bus route set in the simulation system. The simulation system updates the information related to the buses on the controlled route every second, including the operation status of the bus ("service in the station" or "driving on the road"), the location of the bus and its cruising speed. In the simulation system, it is assumed that the fixed passenger capacity of each bus is 70 people, that is, the passenger capacity of a bus at any time cannot exceed 70 people. According to the epidemic prevention level of different regions, the maximum full load rate of the bus is set to ensure the safe distance of passengers in the car. In the simulation system, the boarding and alighting of the bus passengers occur at the same time, and the service time is equal to the maximum boarding and alighting time. Passengers arrive at each station according to the Poisson process and are served according to the FCFS (first come first served) rule. The number of passengers getting off a bus at any station is a random integer, which is less than the number of existing passengers on the bus. In the calculation example of this embodiment, a controlled route is established according to the actual bus operation route. There are 16 stations on the controlled route, including a starting station and a terminal station. According to the collected historical data, the bus line departure frequency is set to 16 vehicles / hour, that is, the standard driving interval is 225 seconds. According to the data collected by the roadside sensors, the traffic conditions of different sections are regressed and analyzed. By setting different maximum allowable speeds between different stations on the route, the road conditions of each section in the actual situation are reflected. The maximum allowable speed of each section is taken according to an independent random distribution. The random distribution is a normal distribution with a mean of 7.25 and a standard deviation of 4.5, and the minimum value is set to 3.25 and the maximum value is set to 11.25. The randomness of the road traffic conditions represented by different maximum allowable speeds can improve the robustness and applicability of the algorithm.

[0039] S3. Establish a Markov model, determine the objective function, and build a deep deterministic policy gradient reinforcement learning framework;

[0040] In step S3, the state of the bus operation simulation system at the next moment depends only on the state of the system at the previous moment, and the state transfer conforms to the Markov property. The deep deterministic policy gradient reinforcement learning framework is selected. On the one hand, it ensures that the output behavior decision remains numerically continuous after the reinforcement learning framework training is completed, which is more in line with the actual bus running speed. On the other hand, the Actor-Critic neural network structure and experience replay mechanism in the framework reduce the continuity in multiple trainings, improve the effectiveness of the training results, and have a good training effect.

[0041] The reinforcement learning framework built in step S3 includes four neural networks, namely the actor's real network, the actor's estimated network, the critic's real network, and the critic's estimated network.

[0042] The two neural networks of the Actor are mainly responsible for obtaining a certain action by inputting state variables. The established neural network structure is as follows: the input of the neural network is the state of the environment; the output of the neural network is the action that the agent will take: the input accepted by the Actor reality network is s_, which represents the state of the next environment; the input accepted by the Actor estimation network is s, which represents the state of the current environment. During the training process, the parameters of the Actor estimation network are optimized, and the parameters of the Actor reality network are updated after multiple steps of training.

[0043] The two neural networks of Critic are mainly responsible for evaluating the current state and the actions taken. The input of the two neural networks of Critic is the state variables of the simulation environment and the actions taken in the last decision. The output is q, which refers to the score value obtained by the state variables of the simulation environment and the actions taken in the last decision. The input accepted by the Critic estimation network is [s, a], which represents the state variables and corresponding actions of the simulation environment at the current moment; the input accepted by the Critic reality network is [s_, a_], which represents the state variables and corresponding actions of the simulation environment at the next moment.

[0044] The parameter optimization process of the Actor estimation network needs to be combined with the two neural networks of the Critic. The parameter optimization process first obtains the action predicted by the Actor estimation network through the current state variable, then adds random noise interference to the predicted action to prevent overfitting, and passes the action after adding noise and the state variable to the Critic estimation network to obtain its score Value. The direction of gradient backpropagation optimization is to maximize the score Value.

[0045] S4. Perform training in the bus operation simulation system to optimize the neural network parameters in the reinforcement learning framework;

[0046] In the bus operation simulation system, the operation status of all buses currently running on the line is monitored. When the bus to be controlled enters any station, the judgment of whether to perform training is started. When the number of buses before and after the bus is greater than or equal to 5 vehicles, the 5 vehicles closest to the bus are selected to form a group, and the operation status of the vehicles in the group is counted, including: the running speed of the 5 buses (speed_1, speed_2, speed_3, speed_4, speed_5), the headway time between the 5 buses (headway_1, headway_2, headway_3, headway_4) and the number of stops between the 5 buses (stopnum_1, stopnum_2, stopnum_3, stopnum_4), the position of the bus in the group (num), and the above 14 data are converted into a one-dimensional vector and passed into the reinforcement learning framework as the initial state variable s, wherein the Actor estimation network in the reinforcement learning framework calculates the optimal driving speed of the bus from the station to the next station, and under the premise of not exceeding the maximum allowable driving speed of the section, the bus is ordered to go to the next station according to the optimal driving speed calculated by the Actor network.

[0047] When the bus arrives at the next stop, the 5 vehicles closest to the bus are selected to form a group, and the running status of the vehicles in the group is counted. The above data is converted into a one-dimensional vector as a new state variable s_, and the variance r of the estimated headway of the 5 vehicles in the group is calculated as an evaluation index of the last driving speed control result. At the same time, [initial state variable s, driving speed a calculated by the Actor estimation network, variance r of the estimated headway of the vehicles in the group, new state variable s_] is saved as a set of data to the experience pool for subsequent neural network parameter training. The estimated headway is calculated using the distance between different buses and the maximum operating speed limit of the road section.

[0048] Set the maximum capacity of the experience pool. When the experience pool reaches the maximum capacity, the newly added data randomly replaces the historical data in the experience pool. And disrupt the continuity of the original data, randomly select data from the experience pool for training, use the mean square error loss function, use gradient back propagation to update all parameters of the Critic estimation network, use the output of the Critic estimation network as the loss function, and use gradient back propagation to update all parameters of the Actor estimation network. After training a certain number of times, refer to the parameters of the Actor and Critic estimation networks, and perform soft replacement updates on the parameters of the Actor and Critic real networks in a certain proportion.

[0049] The training process is as follows:

[0050] 1) Initialize the Actor estimated network parameter θ, the Critic estimated network parameter ω, set the Actor real network parameter θ′=θ, the Critic real network parameter ω′=ω, the attenuation factor γ, the soft update coefficient τ, and the constant m

[0051] Clear experience pool D

[0052] 2) for i from 1 to T, iterate

[0053] 2.1) Initialize the simulation system state. After the simulation system is preheated and training begins, obtain the feature vector S

[0054] 2.2) In the Actor estimation network, the action A is obtained based on the current state variable S

[0055] 2.3) Execute action A, get the new state S′, and calculate the immediate reward R. R is calculated by the variance of the headway of buses in the group. The smaller the variance of the headway of buses in the group, the larger R is.

[0056] 2.4) Store [S, A, R, S′] into the experience pool D, and let S = S′

[0057] 2.5) Sample m samples from the experience pool D and use m samples to calculate the current target value y j (Q is the evaluation value calculated by the Critic network)

[0058]

[0059] 2.6) Using mean square error loss function

[0060]

[0061] Update the Critic estimation network parameters ω through the gradient back propagation of the neural network

[0062] 2.7) Update the parameters θ of the Actor estimation network through the gradient back propagation of the neural network

[0063] 2.8) According to the soft update coefficient, assign ω, θ to ω′, θ′ in a certain proportion, and update the Actor and Critic real network parameters

[0064] 2.9) When i=T, this iteration is complete, otherwise jump to 2.2, i=i+1

[0065] S5. Repeat the S4 process in the bus operation simulation system to train the neural network parameters in the reinforcement learning framework, and save the Actor estimation network structure and parameters after the training.

[0066] The simulation system runs for 18,000 seconds as one generation, and the above training process is repeated for multiple generations to train the neural network parameters in the reinforcement learning framework. After the training is completed, the Actor estimation network structure and all parameters are retained as the training results. In the application stage, when the bus to be controlled enters any station, it is judged whether the speed control conditions are met. When the number of buses before and after the bus is greater than or equal to 5, the 5-bus formation closest to the bus is selected, and the running status of the vehicles in the group is counted, including: the running speed of the 5 buses (speed_1, speed_2, speed_3, speed_4, speed_5), the headway time between the 5 buses (headway_1, headway_2, headway_3, headway_4) and the number of stops between the 5 buses (stopnum_1, stopnum_2, stopnum_3, stopnum_4), the position of the bus in the group (num), and the above 14 data are converted into a one-dimensional vector as the input S of the Actor network. The running speed V of the bus before the next stop is calculated through the Actor estimation network. Under the premise of not exceeding the maximum allowable driving speed of the section, the bus is ordered to go to the next stop according to the optimal driving speed calculated by the Actor network. After testing the control effect, it was found that the distribution of passengers in all buses was more even, which alleviated the overcrowding inside individual vehicles and reduced the risk of epidemic transmission. The experimental results are as follows: Figure 2 As shown, the optimization effect of the objective function value is close to 15%.

[0067] Finally, it should be pointed out that the above examples are only used to illustrate the calculation process of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above examples, a person skilled in the art should understand that the calculation process described in the above examples can still be modified, or some parameters can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding calculation method deviate from the spirit and scope of the calculation method of the present invention.

[0068] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A bus on-the-go reinforcement learning speed control method for epidemic prevention and control, characterized in that: The steps include: S1. Set optimization goals for epidemic prevention and control: make the number of passengers on each bus average and reduce crowding; S2. Obtain real-time bus operation information and road traffic conditions through traffic information detectors; establish a time-continuous bus operation simulation system based on the real bus operation network; S3. Establish a Markov model, determine the objective function, and build a reinforcement learning framework for deep deterministic policy gradients; S4. Perform training in the bus operation simulation system to optimize the neural network parameters in the reinforcement learning framework. Whenever a bus on the road enters a bus station, select the 5 vehicles in front and behind the bus to form a group, select the bus information in the group as the state parameter and pass it into the reinforcement learning framework to calculate the running speed of the next section of the bus. When the bus enters the next station, calculate the variance of the headway time of all vehicles in the group as the reward feedback of the last control result. S5, repeating the process of training and optimizing the parameters of the neural network in the reinforcement learning framework in S4 in the bus operation simulation system; saving the neural network structure and parameters after the training; The control effect of the neural network trained by S4 was evaluated in the bus operation simulation system: when the controlled bus entered the station, 5 vehicles were selected in front and behind the bus, and the bus status information in each group was passed into the trained and saved neural network as input. The running speed of the bus before the next stop was calculated by the neural network and compared with the case without control, which showed that the number of passengers in all buses was more evenly distributed.

2. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: In step S2, based on the intelligent vehicle system technology and the intelligent roadside system technology, the bus operation simulation system simulates the actual bus operation route, and divides the bus operation route into several sections according to the bus stops; A statistical regression analysis is performed on the bus operation data collected by traffic detectors. Different maximum bus operating speed limits are set for different sections in the bus operation simulation system according to the historical traffic congestion and road conditions of different sections. In the bus operation simulation system, the maximum operating speed limits are randomly set in different sections according to the distribution obtained from the statistical regression analysis. The random maximum operating speed limits can reflect the differences in real road traffic conditions in different sections.

3. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: The bus operation simulation system updates the bus-related information on the controlled line every second, including the bus's operating status, bus location and cruising speed.

4. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: In step S3, a deep deterministic policy gradient reinforcement learning framework is selected. On the one hand, the continuity of the final output result of the neural network is guaranteed, which is more in line with the actual situation that the bus running speed is continuous rather than discrete; on the other hand, the actor-critic framework is adopted, and the experience replay mechanism is introduced to reduce the continuity after several training data and improve the robustness of the training results.

5. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: The reinforcement learning framework established in step S3 includes four networks, namely, the Actor reality network, the Actor estimation network, the Critic reality network, and the Critic estimation network; the Actor reality network and the Actor estimation network are responsible for obtaining a certain action by inputting state variables, and the established neural network structure is as follows: the input of the neural network is the state of the environment; the output of the neural network is the action to be taken by the agent: the input accepted by the Actor reality network is s_, which represents the state of the next environment; the input accepted by the Actor estimation network is s, which represents the state of the current environment. During the training process, the parameters of the Actor estimation network are optimized, and the parameters of the Actor reality network are updated after several steps of training; The Critic reality network and the Critic estimation network are responsible for evaluating the current state and the actions taken; the inputs of the Critic reality network and the Critic estimation network are the state variables of the simulation environment and the actions taken in the last decision, and the output is q, which refers to the score value obtained under the corresponding simulation environment state and action conditions; the input accepted by the Critic estimation network is [s, a], which represents the state variables and corresponding actions of the simulation environment at the current moment; the input accepted by the Critic reality network is [s_, a_], which represents the state variables and corresponding actions of the simulation environment at the next moment; The parameter optimization process of the Actor Estimation Network is carried out in combination with the Critic Real Network and the Critic Estimation Network. The parameter optimization process of the Actor Estimation Network first obtains the action predicted by the Actor Estimation Network through the current environment, then adds random noise interference to the predicted action to prevent overfitting, and passes the action with added noise to the Critic Estimation Network to obtain its score Value. The gradient back propagation optimization direction is to maximize the score Value.

6. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: In step S4, when the bus to be controlled enters any station, the judgment of whether to conduct training is started. When the number of buses before and after the bus is ≥ 5, the 5 buses closest to the bus are selected to form a group, and the running status of the vehicles in the group is counted, including: the running speed of the 5 buses: speed_1, speed_2, speed_3, speed_4, speed_5, the headway time between the 5 buses: headway_1, headway_2, headway_3, headway_4; and the number of stops between the 5 buses: stopnum_1, stopnum_2, stopnum_3, stopnum_4, the position num of the bus in the group, and the above 14 data are converted into A one-dimensional vector is passed into the reinforcement learning framework as the initial state parameter state, where the Actor estimation network in the reinforcement learning framework calculates the optimal driving speed of the bus before it leaves the station and reaches the next station. Under the premise of not exceeding the maximum allowable driving speed of the section, the bus is ordered to go to the next station at the optimal driving speed calculated by the actor estimation network; when the bus arrives at the next station at the above driving speed, the 5 buses closest to the bus are selected to form a group, and the variance of the estimated headway time of the 5 buses in the group is calculated as the evaluation result of the last driving speed control behavior. At the same time, [initial state state, driving speed a calculated by the Actor estimation network, evaluation result r] is saved as a set of training data in the experience pool for subsequent neural network parameter training.

7. The bus on-the-way reinforcement learning speed control method for epidemic prevention and control according to claim 1 is characterized in that: In step S1, the epidemic prevention and control objective function is set as: Among them, minx is to minimize the sum of the squares of the number of passengers of all buses on each section; n aij is the number of passengers in bus a on road section ij; in view of the different risk levels in different regions, the full load rate limit in the bus operation simulation system is increased according to the actual situation; by fine-tuning the parameters in the bus operation simulation system and the internal parameters of the reinforcement learning framework, the on-the-way reinforcement learning speed control method for buses is suitable for speed control of different types of buses in areas with different risk levels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the on-the-way reinforcement learning speed control method for buses for epidemic prevention and control as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the on-the-way reinforcement learning speed control method for buses for epidemic prevention and control as described in any one of claims 1 to 7 are implemented.

Images