Bus resource dynamic scheduling method based on spatial graph convolution and near-end strategy optimization

Abstract

The invention discloses a bus resource dynamic scheduling method based on spatial graph convolution and near-end strategy optimization in the technical field of bus resource scheduling, and the method comprises the following specific steps: employing a weight graph and AggreGatE to carry out the modeling of a road network structure, and carrying out the sampling and aggregation operation through a weighted GraphSAGE model, according to the method, relevance and edge feature capture between nodes of a vehicle and a station are realized, and interaction with the environment is carried out by utilizing a multi-output strategy network architecture and an objective function of a reward mechanism through a reinforcement learning near-end strategy optimization algorithm; according to the invention, dynamic and multi-line departure interval adjustment and passenger travel route planning are realized, and a Dijkstra algorithm is utilized to calculate an initial route of a bus from a current position to a passenger starting station, and the like. The method adapts to the change of traffic conditions, and generates a more dynamic and flexible scheduling decision.

Description

Technical Field

[0001] The present invention relates to the technical field of public transport resource scheduling, and in particular to a public transport resource dynamic scheduling method based on spatial domain graph convolution and proximal strategy optimization. Background Art

[0002] The dispatching effect of urban public transportation resources is affected by factors such as road conditions, traffic flow and passenger travel demand. This requires the dispatching method of public transportation resources to be able to adapt to complex dynamic environmental changes and adjust the dispatching strategy in real time. The construction and design of public transportation resource dispatching methods and models require a large amount of data support. How to fully explore the road network structure, vehicle operation status and passenger behavior in the data to design a reasonable dispatching plan generation method in view of the characteristics of large data volatility, strong heterogeneity and high real-time requirements is a difficult problem. On the other hand, in actual scenarios, passengers' public transportation travel needs are changing. The existing mainstream technical solutions (Random strategy, Deep Deterministic Policy Gradient, DDPG, Advantage Actor-Critic, A2C) are not suitable for public transportation.

[0003] 1) Random strategy: This method makes random decisions regardless of external environmental factors such as traffic conditions and passenger demand. It randomly selects an action at each decision point and randomly selects bus stops and routes at each moment, without considering the actual situation and optimization goals. The model does not consider the coordinated scheduling between vehicles and changes in the traffic environment, but only operates according to randomly generated decisions, resulting in unstable scheduling results and large differences in the decision-making process.

[0004] 2) Deep Deterministic Policy Gradient (DDPG): This method is based on deep reinforcement learning. It uses a deterministic strategy to output a certain action in a given traffic environment to solve the problem of continuous action space. It uses a neural network approximate value function and learns the strategy through experience replay and target network methods. It uses experience replay for training, so it has certain disadvantages in training efficiency.

[0005] 3) Advantage Actor-Critic (A2C): This method is a reinforcement learning method based on policy gradients. It combines the gradient strategy and the value function method, uses a probabilistic strategy, and outputs actions represented by probability distribution. It can significantly demonstrate the advantage of training speed in dealing with large-scale continuous control problems such as dynamic bus scheduling. It can directly interact with the environment to update parameters and train multiple agents on multiple environment instances, thereby improving training efficiency. However, since it is a policy gradient-based method, the training process may be accompanied by a high increment, resulting in instability in the model. In addition, A2C itself does not have a dedicated and reasonable exploration strategy, which is one of the biggest drawbacks of this model.

[0006] In summary, these algorithms find it difficult to cope with complex environments such as fluctuations in passenger travel demand, changes in traffic flow, and uncertain road conditions. In addition, there are difficulties in obtaining accurate and high-quality data and fully extracting the features of graph-structured data. Summary of the invention

[0007] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0008] Therefore, the purpose of the present invention is to provide a bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization to solve the problems raised in the above background technology.

[0009] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0010] The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization is characterized by the following steps:

[0011] S1. Use weighted graph and AggreGatE to model the road network structure, and perform sampling and aggregation operations through the weighted GraphSAGE model to achieve the correlation and edge feature capture between the nodes of vehicles and stations;

[0012] S2, through the reinforcement learning proximal policy optimization algorithm, using its multi-output policy network architecture and the objective function of the reward mechanism to interact with the environment, to achieve dynamic, multi-line departure interval adjustment and passenger travel route planning;

[0013] S3. The Dijkstra algorithm is used to calculate the initial path of the bus from the current location to the passenger's starting station. Then, the node and edge feature representations learned by the weighted GraphSAGE model are combined with the reinforcement learning proximal strategy optimization algorithm to adjust the station selection in the path and the order of passengers getting on and off the bus. The path is optimized through continuous iteration until the predetermined stopping condition is reached to maximize passenger satisfaction and reduce the overall travel time of passengers.

[0014] As a preferred solution of the bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization described in the present invention, in step S1, the GraphSAGE model uses a uniform sampling method to sample fixed neighborhood nodes, and the sampling formula is as follows:

[0015]

[0016] in, For nodes, Represents a collection of nodes.

[0017] As a preferred solution of the bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization described in the present invention, in step S1, an aggregation function is used to aggregate the information of neighbor nodes to obtain the embedding of the target node, and then the information aggregated on the node is used to predict the node label. The aggregation function formula is as follows:

[0018]

[0019] in, and They represent the features of the current node and neighboring nodes respectively, and are both weight matrices of the neural network. e refers to the weight assigned to the edge, and mean represents the average of the features of the neighboring nodes. Refers to the characteristic information of the target node after the aggregation operation. Represents the characteristic information of the current node. Represents the characteristic information of the neighboring nodes of the current node, Represents the neighborhood set of the current node i;

[0020] After the aggregation operation, the characteristic information of the central node and the domain node are combined together, as shown below:

[0021]

[0022] Among them, σ is a nonlinear activation function, k is the depth of the neural network, is the parameter of the k-th layer linear transformation, CONCAT refers to the feature aggregation of the central node and the neighboring nodes, represents the feature vector of the central node of the kth layer, represents the feature vector of the central node of the k-1th layer, Represents the feature vector of the neighborhood nodes of the central node in the kth layer;

[0023] Finally, the normalization operation is performed, and the formula is as follows:

[0024] ;

[0025] in, The feature vector representing the central node of the kth layer is modulo-operated, and then the vector is normalized to a unit vector.

[0026] As a preferred solution of the bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization described in the present invention, in step S2, the multi-output strategy network architecture is an Actor-Critic architecture.

[0027] As a preferred solution of the bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization described in the present invention, in step S2, the objective function of the Actor-Critic architecture is as follows:

[0028]

[0029]

[0030] in, Represents the empirical expected value, which is used to determine the update direction of the Actor network. Represents the parameters of the current policy function, , and They correspond to the state of the agent at a certain moment, the action performed, and the reward obtained. ε is an adjustable parameter. Represents the parameters of the strategy, through optimization Improve the performance of your strategy, is the advantage function, which is used to evaluate the quality of the action and guide the update direction of the strategy. is a cutting function that ensures the stability of the strategy update. It is the ratio of the probability of taking an action under a given state between the new strategy and the old strategy, reflecting the magnitude of the strategy update. , It means limiting the amplitude of policy update to a given range, where is a tunable hyperparameter.

[0031] As a preferred solution of the bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization described in the present invention, the proximal strategy optimization algorithm modifies and updates the objective function, and the specific update formula is as follows:

[0032] ;

[0033] in, A parameterized policy network representing the old policy.

[0034] As a preferred solution of the method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization described in the present invention, in step S2, in the reinforcement learning proximal strategy optimization algorithm, the agent The data obtained by interacting with the environment is stored in the data container buffer, where is a state set, is a collection of actions. is the reward function, P is the state transition probability, is the discount factor. When the buffer is full, GAE is used to calculate the advantage value of each time step and the optimization target of Critic. Then the data in the container is taken out in sequence according to the batch size, and the entropy loss, policy loss and value loss of each batch are calculated. Then, gradient descent is used to update the parameters of Actor and Critic in the output policy network architecture.

[0035] Compared with the prior art, the present invention has the following beneficial effects: the present invention adopts weight graph and AggreGatE to model the road network structure, and performs multi-level information dissemination through sampling and aggregation operations, so as to realize the correlation and edge feature capture between nodes such as vehicles and stations, and obtain the bus resource status with a wide coverage. In addition, through the proximal strategy optimization algorithm, the multi-output strategy network architecture and reasonable reward function design are utilized to realize dynamic, multi-line departure interval adjustment and passenger travel route planning. The present invention fully exploits the dependency and characteristics of information such as vehicle location, traffic conditions, and number of passengers during traffic operation, adapts to changes in traffic conditions, and generates more dynamic and flexible scheduling decisions. The application of this method provides a good solution for the construction of smart cities, changes in traffic flow, uncertainty in road conditions, meeting passenger travel needs, and providing a good passenger experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0037] Figure 1 A model diagram of a multi-output strategy network architecture provided by the present invention;

[0038] Figure 2 A flowchart of the reinforcement learning proximal strategy optimization algorithm provided by the present invention;

[0039] Figure 3 The entropy loss graph provided by the present invention;

[0040] Figure 4 A loss graph of the strategy provided by the present invention;

[0041] Figure 5 A value loss graph provided for the present invention;

[0042] Figure 6 A comparison chart of the method provided by the present invention with the random strategy method, the deep deterministic strategy gradient method, and the dominant action review method. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] Secondly, the present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] The present invention provides a bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization, which realizes the multi-objective optimization of passenger travel time and bus operating cost. Through the bus dynamic scheduling model, the dependency and characteristics of information such as vehicle position, traffic conditions, and number of passengers in the traffic operation process are fully explored, and the changes in traffic conditions are adapted to generate more dynamic and flexible scheduling decisions.

[0047] The present invention is based on the method of dynamic dispatching of public transportation resources based on spatial graph convolution and proximal strategy optimization, and the specific implementation steps are as follows:

[0048] S1. Use weighted graph and AggreGatE to model the road network structure, and perform sampling and aggregation operations through the weighted GraphSAGE model to achieve the correlation and edge feature capture between the nodes of vehicles and stations;

[0049] S2, through the reinforcement learning proximal policy optimization algorithm, using its multi-output policy network architecture and the objective function of the reward mechanism to interact with the environment, to achieve dynamic, multi-line departure interval adjustment and passenger travel route planning;

[0050] S3. The Dijkstra algorithm is used to calculate the initial path of the bus from the current location to the passenger's starting station. Then, the node and edge feature representations learned by the weighted GraphSAGE model are combined with the reinforcement learning proximal strategy optimization algorithm to adjust the station selection in the path and the order of passengers getting on and off the bus. The path is optimized through continuous iteration until the predetermined stopping condition is reached to maximize passenger satisfaction and reduce the overall travel time of passengers.

[0051] In the embodiment of the present invention, a weighted GraphSAGE model is used to extract features from the graph, and each node and edge is mapped to a low-dimensional vector space. Then, a reinforcement learning proximal policy optimization algorithm is used to improve the bus scheduling strategy, and the vehicle scheduling problem is converted into a Markov decision process. The model is trained with the help of the proximal policy optimization algorithm. The model learns the optimal scheduling strategy by interacting with the environment to maximize the overall return. The multi-output strategy network architecture is as follows: Figure 1 shown.

[0052] In the embodiment of the present invention, GraphSAGE uses a uniform sampling method to sample fixed neighborhood nodes, and the sampling function is represented by formula (1).

[0053] (1)

[0054] in, For nodes, Represents a collection of nodes.

[0055] Use the aggregation function to aggregate the information of neighbor nodes to obtain the embedding of the target node, and then use the information aggregated on the node to predict the node label. The aggregation function is shown in formula (2).

[0056] (2)

[0057] in, and They represent the features of the current node and neighboring nodes respectively, and are both weight matrices of the neural network. e refers to the weight assigned to the edge, and mean represents the average of the features of the neighboring nodes. Refers to the characteristic information of the target node after the aggregation operation. Represents the characteristic information of the current node. Represents the characteristic information of the neighboring nodes of the current node, Represents the neighborhood set of the current node i. The improved GraphSAGE mainly assigns weights to each neighbor node according to the edge and performs weighted average to represent the embedding updated by the previous convolutional layer of the node. The features of the neighbor nodes are updated through formula (3).

[0058] (3)

[0059] in, is the weight matrix, σ is the nonlinear activation function, and b is the bias term of the neural network. After the aggregation operation, the feature information of the central node and the domain node are combined together, as shown in formula (4).

[0060] (4)

[0061] Among them, σ is a nonlinear activation function, k is the depth of the neural network, is the parameter of the k-th layer linear transformation, CONCAT refers to the feature aggregation of the central node and the neighboring nodes, represents the feature vector of the central node of the kth layer, represents the feature vector of the central node of the k-1th layer, Represents the feature vector of the neighborhood nodes of the central node in the kth layer. Finally, a normalization operation is performed, as shown in formula (7).

[0062] (5)

[0063] in, The feature vector representing the central node of the kth layer is modulo-operated, and then the vector is normalized to a unit vector.

[0064] In an embodiment of the present invention, the proximal policy optimization algorithm is a deep reinforcement learning strategy based on the Actor-Critic architecture. The proximal policy optimization is used to update the policy function, thereby improving the performance of the model. The Actor policy network is used to control the movement of the intelligent body and perform a series of actions. Because it is necessary to output a discrete action probability distribution, the number of neurons in the output head is equal to the number of optional actions. The Critic value network is used to evaluate the quality of the action and estimate the state value, so its output head is one neuron. The proximal policy optimization algorithm proposes a pruned objective function, as shown in formulas (6) and (7).

[0065] (6)

[0066] (7)

[0067] in, Represents the empirical expected value, which is used to determine the update direction of the Actor network. Represents the parameters of the current policy function, , and They correspond to the state of the agent at a certain moment, the action performed, and the reward obtained. ε is an adjustable parameter. Represents the parameters of the strategy, through optimization Improve the performance of your strategy, is the advantage function, which is used to evaluate the quality of the action and guide the update direction of the strategy. is a cutting function that ensures the stability of the strategy update. It is the ratio of the probability of taking an action under a given state between the new strategy and the old strategy, reflecting the magnitude of the strategy update. , It means limiting the amplitude of policy update to a given range, where is an adjustable hyperparameter. The first term of min in formula (6) is , which is a conservative policy iteration method. In the absence of any constraints, The maximization of will make the policy update too large, so the proximal policy optimization algorithm modifies and updates the objective function, as shown in formula (8).

[0068] (8)

[0069] in, The parameterized policy network represents the old policy. The second term of min in formula (6) modifies the alternative objective function by clipping the probability ratio, which controls the movement range of within the interval, and then takes the minimum of the clipped and unclipped objectives, so that the final objective function is the lower bound of the unclipped objective function. The clipped objective function is only the first part of the objective function of the proximal policy optimization algorithm. The second part is the optimization target of the state value function. The third part is to introduce entropy to encourage the policy to explore unknown states and state-action combinations.

[0070] In the embodiment of the present invention, at each step, the proximal policy optimization algorithm estimates the state value of the current state, selects an action and estimates the probability of executing the action, and limits the magnitude of each update to ensure the stability of the strategy. The data obtained by interacting with the environment are stored in the data container buffer. When the buffer is full, GAE is used to calculate the advantage value of each time step and the optimization target of the Critic. Then the data in the container is taken out in sequence according to the batch size, and the entropy loss, policy loss and value loss of each batch are calculated. Then, gradient descent is used to update the parameters of the Actor and Critic.

[0071] The workflow diagram of the proximal strategy optimization algorithm is as follows: Figure 2 As shown in the figure, the specific steps are as follows: a batch of data is randomly sampled from the historical data of the interaction between the agent and the environment for training. Then, these data are sent to the Actor Network (policy network), which outputs the probability of action a based on the current strategy and the environment state s, and calculates the ratio between the current strategy and the old strategy. and the objective function using clipping To stabilize the learning process, at the same time, the Critic Network (value network) receives state s and action a as input, outputs the value estimate of state s, and returns the new state and reward. This information is used to calculate the advantage function , which measures the relative value of executing action a in state s, where the advantage function A is calculated based on the temporal difference error TD-error. By continuously iterating this process, the PPO algorithm can simultaneously learn the strategy and value function, so that the agent can gradually improve its behavior strategy in the interaction with the environment.

[0072] In the embodiment of the present invention, due to the relationship between the number of output actions, the present invention considers the graph network model of the Actor in the multi-output strategy network architecture model as a Node-level task, and considers the Critic as a Graph-level task. The Critic is similar to the above-mentioned bus scheduling deep reinforcement learning model. The structure of the Actor is to directly input the output of the graph network in the order of each Node into the fully connected neural network, which is composed of two layers of 64 neurons and one fully connected layer with one neuron per layer.

[0073] like Figure 3-Figure 5 , compare the convergence of the three loss curves of entropy loss, strategy loss and value loss of the bus dynamic scheduling model. It can be found that all three loss curves converge. When the entropy loss converges, it can be said that the policy network has converged to a relatively certain strategy and no longer generates too much randomness. This means that the policy network has learned a better strategy and can output a more reliable action selection for a given state; when the policy loss converges, it means that the update of the policy network gradually becomes stable and the learned strategy is getting closer and closer to the optimal strategy. This means that the policy network can more accurately select high-return actions under a given state, thereby improving the performance and effect of the algorithm; when the value loss converges, it means that the predicted value of the value function approximator gradually approaches the real return, indicating that the value function approximator can more accurately predict the expected return under a given state. This is crucial to the performance of the reinforcement learning algorithm, because the accuracy of the value function directly affects the optimization of the strategy and the accuracy of the decision. For the performance indicator of the operating cost of the bus company, the comparison results with the baseline method were obtained during the verification process. When the deep reinforcement learning model of the present invention is used to schedule buses, the total operating time of the bus is significantly less than that of the baseline method, thereby significantly reducing the total operating cost of the bus company.

[0074] Combination Figure 6, it can be seen that the performance of the method proposed in this paper is significantly better than other baseline models. The vertical axis is represented by the total operating time of the bus company. The smaller the value, the better the performance indicator of the bus company's operating cost. From the experimental performance of specific methods, the performance of the Random method is the worst. Since this method does not consider the coordinated scheduling between vehicles and changes in the traffic environment, it only operates according to randomly generated decisions, resulting in unstable scheduling results and large differences in the decision-making process. The performance of the A2C method is better than the DDPG method, because A2C does not need to use experience replay for training like DDPG, but can directly interact with the environment to update parameters and train multiple agents on multiple environment instances, thereby improving training efficiency. However, since A2C is a policy gradient-based method, the training process may be accompanied by a high increment, resulting in instability of the model, and A2C itself does not have a dedicated and reasonable exploration strategy, which is the biggest drawback of this model. In view of the analysis of the above baseline model, the BDS method proposed in this paper can solve the problems existing in the above baseline model in a targeted manner. This method takes advantage of the PPO's own exploration strategy. While making full use of the learned good strategies, it also explores new strategies to cope with different traffic conditions, achieving a balance between exploration and utilization. In addition, PPO can limit the amplitude of strategy updates, thereby ensuring the stability of model training. In addition, the BDS method (i.e., the method of the present invention) can effectively extract feature information of stations, routes, and vehicles from the urban transportation network, which is conducive to better guiding decision-making. Therefore, the method of the present invention significantly reduces the total operating cost of the bus company and verifies the effectiveness of the model in dynamic bus scheduling.

[0075] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A bus resource dynamic scheduling method based on spatial graph convolution and proximal strategy optimization, characterized in that: Here are the steps: S1. Use weighted graph and AggreGatE to model the road network structure, and perform sampling and aggregation operations through the weighted GraphSAGE model to achieve the correlation and edge feature capture between the nodes of vehicles and stations; S2, through the reinforcement learning proximal policy optimization algorithm, using its multi-output policy network architecture and the objective function of the reward mechanism to interact with the environment, to achieve dynamic, multi-line departure interval adjustment and passenger travel route planning; S3. The Dijkstra algorithm is used to calculate the initial path of the bus from the current location to the passenger's starting station. Then, the node and edge feature representations learned by the weighted GraphSAGE model are combined with the reinforcement learning proximal strategy optimization algorithm to adjust the station selection in the path and the order of passengers getting on and off the bus. The path is optimized through continuous iteration until the predetermined stopping condition is reached to maximize passenger satisfaction and reduce the overall travel time of passengers.

2. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 1 is characterized in that: In step S1, the GraphSAGE model uses uniform sampling to sample fixed neighborhood nodes. The sampling formula is as follows: ； in, For nodes, Represents a collection of nodes.

3. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 1 is characterized in that: In step S1, an aggregation function is used to aggregate the information of neighbor nodes to obtain the embedding of the target node, and then the information aggregated on the node is used to predict the node label. The aggregation function formula is as follows: ； in, and They represent the features of the current node and neighboring nodes respectively, and are both weight matrices of the neural network. e refers to the weight assigned to the edge, and mean represents the average of the features of the neighboring nodes. Refers to the characteristic information of the target node after the aggregation operation. Represents the characteristic information of the current node. Represents the characteristic information of the neighboring nodes of the current node, Represents the neighborhood set of the current node i; After the aggregation operation, the characteristic information of the central node and the domain node are combined together, as shown below: ； Among them, σ is a nonlinear activation function, k is the depth of the neural network, is the parameter of the k-th layer linear transformation, CONCAT refers to the feature aggregation of the central node and the neighboring nodes, represents the feature vector of the central node of the kth layer, represents the feature vector of the central node of the k-1th layer, Represents the feature vector of the neighborhood nodes of the central node in the kth layer; Finally, the normalization operation is performed, and the formula is as follows: ； in, The feature vector representing the central node of the kth layer is modulo-operated, and then the vector is normalized to a unit vector.

4. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 1 is characterized in that: In step S2, the multi-output strategy network architecture is an Actor-Critic architecture.

5. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 4 is characterized in that: In step S2, the objective function of the Actor-Critic architecture is as follows: ； ； in, Represents the empirical expected value, which is used to determine the update direction of the Actor network. Represents the parameters of the current policy function, , and They correspond to the state of the agent at a certain moment, the action performed, and the reward obtained. ε is an adjustable parameter. Represents the parameters of the strategy, through optimization Improve the performance of your strategy, is the advantage function, which is used to evaluate the quality of the action and guide the update direction of the strategy. is a cutting function that ensures the stability of the strategy update. It is the ratio of the probability of taking an action under a given state between the new strategy and the old strategy, reflecting the magnitude of the strategy update. , It means limiting the amplitude of policy update to a given range, where is a tunable hyperparameter.

6. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 5 is characterized in that: The proximal strategy optimization algorithm modifies and updates the objective function. The specific update formula is as follows: ； in, A parameterized policy network representing the old policy.

7. The method for dynamic scheduling of public transportation resources based on spatial graph convolution and proximal strategy optimization according to claim 5 is characterized in that: In step S2, in the reinforcement learning proximal strategy optimization algorithm, the agent will The data obtained by interacting with the environment is stored in the data container buffer, where is a state set, is a collection of actions. is the reward function, P is the state transition probability, is the discount factor. When the buffer is full, GAE is used to calculate the advantage value of each time step and the optimization target of Critic. Then the data in the container is taken out in sequence according to the batch size, and the entropy loss, policy loss and value loss of each batch are calculated. Then, gradient descent is used to update the parameters of Actor and Critic in the output policy network architecture.

Images