A Deep Graph Reinforcement Learning Generation Method for Multi-task and Multi-criteria Resource Scheduling

Abstract

The present invention proposes a deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling, which belongs to the field of emergency resource scheduling. It transforms the modeling of multi-task resource scheduling problems into a graph generation problem under goal-oriented and multi-constraint conditions, including an offline model training phase and an online solution generation feedback training optimization phase. A pseudo-label training data set based on the analytic hierarchy process and the genetic elite algorithm is constructed; a feasible solution state feature extraction unit is designed to characterize the matching degree of disaster points and resource points, and node hidden features are extracted through a graph neural network based on the feasible solution state. Reinforcement learning is used to gradually generate a resource scheduling solution graph based on the node hidden features; a conditional generative adversarial network framework is used to make the generated results more targeted.

Description

Technical Field

[0001] The present invention relates to the field of emergency resource scheduling, and in particular to a deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling. Background Art

[0002] After a disaster occurs, multiple emergency resource points need to be coordinated for support. The rapid formulation of resource scheduling plans is an effective guarantee for the rapid handling of events. Traditional methods convert multi-task emergency resource scheduling into a multi-objective optimization problem, design different objective functions and constraints according to the characteristics of the scenario, and solve it through optimization algorithms such as particle swarm optimization and genetic algorithm. There are problems such as slow convergence, easy to fall into local optimality, and subjective bias in the objective function. In recent years, some scholars have proposed using reinforcement learning to solve the problem of emergency resource scheduling, which greatly reduces the time for generating plans. However, there are problems such as slow convergence of multi-task node resource scheduling training, lack of data sets, and unstable generation effects. Summary of the invention

[0003] The present invention proposes a deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling, which can quickly and reasonably generate resource scheduling plans, and as the scheduling plan feedback model is manually adjusted, the model is retrained, thereby improving the versatility of the resource scheduling model.

[0004] In order to achieve the above object, the present invention adopts the following technical solution:

[0005] A deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling includes the following steps:

[0006] Step 1, construct resource scheduling pseudo-label data through a genetic elite algorithm solver that follows the emergency scenario resource scheduling criterion;

[0007] Step 2: construct a multi-task and multi-criteria multi-resource scheduling graph reinforcement learning generation model, and use a conditional generative adversarial learning framework to alternately train the multi-resource scheduling graph reinforcement learning generation model and the discriminator until dynamic equilibrium is reached;

[0008] Step 3: The multi-resource scheduling graph intensity learning generation model generates a resource scheduling solution;

[0009] Step 4: Manually adjust the resource scheduling plan and generate a model for reward and punishment feedback on the intensity learning of the multi-resource scheduling graph.

[0010] Furthermore, the specific process of step 1 is as follows:

[0011] Under the criterion of highest global time satisfaction, highest global resource number satisfaction and lowest global resource scheduling transportation cost, the weight of each criterion is determined by using the hierarchical analysis method. , the objective function of multi-resource scheduling is:

[0012]

[0013] F 1 is the task global time satisfaction, F 2 is the global resource quantity satisfaction, F 3 Scheduling transportation costs for global resources;

[0014] Through simulation, multiple sets of disaster point / resource point longitude and latitude locations, the number of resources required for each disaster point, and the number of resources available at each resource point are generated. The genetic elite algorithm is used to solve the objective function in parallel to obtain the optimal solution as the resource scheduling solution under different scenarios, which is used to construct pseudo-label data for resource scheduling.

[0015] Furthermore, the specific process of step 2 is as follows:

[0016] The longitude and latitude coordinates of the disaster point, the resource demand of the disaster point, and the longitude and latitude coordinates of the resource point are obtained. The resource supply of the resource point is used as the initial condition of the model to build a multi-resource scheduling graph reinforcement learning generation model. The multi-resource scheduling graph reinforcement learning generation model includes a feasible solution state feature extraction unit, state structure, action structure, reward mechanism structure, and strategy network structure.

[0017] Among them, the feasible solution state feature extraction unit is used to characterize the matching degree between disaster points and resource points; based on the feasible solution state, the node hidden features are extracted through the graph neural network, and the resource scheduling plan diagram is gradually generated based on the node hidden features by using reinforcement learning;

[0018] The state description of the multi-resource scheduling graph reinforcement learning generation model is:

[0019] ,

[0020] in, is the adjacency matrix of the resource scheduling graph, which indicates the current resource scheduling status. is the node feature matrix, representing the current features of each node;

[0021] The action description of the multi-resource scheduling graph reinforcement learning generation model is:

[0022]

[0023] Used to add a resource point-disaster point matching edge, Dispatch to point , T is the termination mark; the specific calculation process includes the following:

[0024] Step 211, the probability of selecting the first node type as an emergency point and a resource point, and generating the node type according to the probability distribution of the resource point;

[0025] Step 212, based on the node type generated in step 211, if it is a disaster point, the probability of selecting each disaster point as the first endpoint of the newly added edge is calculated, and the disaster point to be matched is generated by probability distribution; if it is a resource point, the probability of selecting each resource point as the first endpoint of the newly added edge is calculated, and the resource point to be matched is generated by probability distribution;

[0026] Step 213, calculating the probability of each node matching the point selected in step 212, and generating matching points with the probability distribution;

[0027] Step 214, connecting the points matched in step 212 and step 213, and generating a new edge;

[0028] Step 215, determine whether to terminate, if the disaster point all needs are met, or there is no dispatchable resources, the termination flag T is True, otherwise T is False;

[0029] The reward mechanism during the training cycle of the multi-resource scheduling graph reinforcement learning generation model is:

[0030]

[0031] is the total reward obtained by taking actions in a training cycle, yes The reward function is , F 1 is the task global time satisfaction, F 2 is the global resource quantity satisfaction, F 3 Scheduling transportation costs for global resources; is the discount factor; is the long-term number of steps in a training cycle; For the cycle termination reward, the reward function is set to: , through the status The reward level is determined by the difference between the resource scheduling graph under the label and the pseudo-label resource scheduling graph.

[0032] The graph neural network is used as the policy network of the multi-resource scheduling graph reinforcement learning generation model. It generates the action probability distribution of the node according to the current node state, and takes the action with the highest probability as the execution action of the current node state. The policy network consists of four parts: input layer, two-layer graph neural layer and output layer. The activation functions of the two neural layers are ReLU and Softmax respectively. The structural function of the policy network is:

[0033]

[0034] s The current resource scheduling status; is the matching degree adjacency matrix of disaster points and resource points. The value of the adjacency matrix is ​​obtained by calculating the similarity of hidden features of nodes; , is the weight matrix of the graph neural layer; , Indicates execution of an action Probability, is the action performed in state s;

[0035] The conditional generative adversarial network framework is used, the pseudo-label data generated in step 1 is used as the positive sample, and the resource scheduling graph data generated by the multi-resource scheduling graph reinforcement learning generation model is used as the negative sample; the distribution of pseudo-label data is assumed to be , the distribution generated by the reinforcement learning generator is , cross entropy loss function The definition is as follows:

[0036]

[0037]

[0038]

[0039] in Represents any initial disaster point, resource point location distribution, demand, and resource number input condition data in the pseudo-label data. is the pseudo label of the resource scheduling solution graph obtained by the solver under the conditional input; represents the fake data generated by the generator given the input condition x, and represents the judgment result of the discriminator on the pseudo-label data, Represents fake data generated by the discriminator in reinforcement learning The result of the above judgment;

[0040] Fixed the multi-resource scheduling reinforcement learning generation model and trained the discriminator D to maximize the cross entropy loss; fixed the discriminator D and trained the multi-resource scheduling reinforcement learning generation model to minimize the cross entropy loss; alternating iterative training until dynamic equilibrium.

[0041] Furthermore, the step 4 is specifically as follows:

[0042] Through artificially adjusting the generated multi-resource scheduling scheme, the artificially adjusted multi-resource scheduling diagram is regarded as the real resource scheduling diagram, and the plane structure diagram of the disaster point and resource point and the real resource scheduling diagram are retained in the training data set;

[0043] When the training data set reaches a certain amount, load the local model parameters and use the newly added training data in the training data set to train the multi-resource scheduling graph reinforcement learning generation model;

[0044] When all training data is completed, the model is published and the model parameters are saved locally.

[0045] The beneficial effects of adopting the above scheme are:

[0046] (1) A genetic algorithm solver that follows the resource scheduling criteria for emergency scenarios is proposed. The solver can be used to generate resource scheduling data in batches to solve the data shortage problem in this scenario. At the same time, since the resource scheduling criteria are followed, the reliability of the constructed data can be guaranteed.

[0047] (2) The multi-task resource scheduling problem is innovatively transformed into a resource scheduling graph generation problem, guided by reinforcement learning, using graph neural networks as graph feature information capturers, and by improving graph neural networks to add feasible solution state feature extraction units to accelerate the generation of multi-resource scheduling solutions.

[0048] (3) The conditional generative adversarial network framework is adopted to make the scheduling plan generated by reinforcement learning more targeted. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a structural diagram of the multi-task multi-criteria resource scheduling graph reinforcement learning generation method of the present invention.

[0050] Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings.

[0052] Reference Figure 1 and Figure 2 , a deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling, comprising the following steps:

[0053] Step 1, construct resource scheduling pseudo-label data through a genetic elite algorithm solver that follows the emergency scenario resource scheduling criterion;

[0054] Step 2: construct a multi-task and multi-criteria multi-resource scheduling graph reinforcement learning generation model, and use a conditional generative adversarial learning framework to alternately train the multi-resource scheduling graph reinforcement learning generation model and the discriminator until dynamic equilibrium is reached;

[0055] Step 3: The multi-resource scheduling graph intensity learning generation model generates a resource scheduling solution;

[0056] Step 4: Manually adjust the resource scheduling plan and generate a model for reward and punishment feedback on the intensity learning of the multi-resource scheduling graph.

[0057] Among them, step 1 is specifically as follows:

[0058] Under the criterion of highest global time satisfaction, highest global resource number satisfaction and lowest global resource scheduling transportation cost, the weight of each criterion is determined by using the hierarchical analysis method. , the objective function of multi-resource scheduling is:

[0059]

[0060] is the highest criterion for the global time satisfaction of the task, is the highest criterion for global resource quantity satisfaction, The minimum transportation cost criterion for the resource scheduling of the bureau. After the objective function is determined, the genetic algorithm is used to solve the objective function to obtain the resource scheduling diagram. Through simulation, multiple groups of disaster point / resource point longitude and latitude locations, the number of resources required for each disaster point, the number of resources available at each resource point and other information are generated. Three genetic elite algorithms are used to solve the objective function in parallel to obtain the optimal solution as the resource scheduling solution under different scenarios.

[0061] Step 2 is as follows:

[0062] The longitude and latitude coordinates of the disaster point, the resource demand of the disaster point, and the longitude and latitude coordinates of the resource point are obtained. The resource supply of the resource point is used as the initial condition of the model to build a multi-resource scheduling graph reinforcement learning generation model. The multi-resource scheduling graph reinforcement learning generation model includes a feasible solution state feature extraction unit, state structure, action structure, reward mechanism structure, and strategy network structure.

[0063] Among them, the feasible solution state feature extraction unit is used to characterize the matching degree between disaster points and resource points; based on the feasible solution state, the node hidden features are extracted through the graph neural network, and the resource scheduling plan diagram is gradually generated based on the node hidden features by using reinforcement learning;

[0064] The state description of the multi-resource scheduling graph reinforcement learning generation model is:

[0065] ,

[0066] is the adjacency matrix of the resource scheduling graph, which indicates the current resource scheduling status. is the node feature matrix, representing the current features of each node.

[0067] The actions of the multi-resource scheduling graph reinforcement learning generation model can be described as: , used to add a resource point-disaster point matching edge, meaning that the material from the point Dispatch to point , T is the termination mark, including the calculation:

[0068] Step 211, the probability of selecting the first node type as an emergency point and a resource point, and generating the node type according to the probability distribution of the resource point;

[0069] Step 212, based on the node type generated in step 211, if it is a disaster point, the probability of selecting each disaster point as the first endpoint of the newly added edge is calculated, and the disaster point to be matched is generated by probability distribution; if it is a resource point, the probability of selecting each resource point as the first endpoint of the newly added edge is calculated, and the resource point to be matched is generated by probability distribution;

[0070] Step 213, calculating the probability of each node matching the point selected in step 212, and generating matching points with the probability distribution;

[0071] Step 214, connecting the points matched in step 212 and step 213, and generating a new edge;

[0072] Step 215, determine whether to terminate, if all the needs of the disaster point are met, or there are no dispatchable resources, the termination flag T is True, otherwise T is False;

[0073] The reward mechanism during the training cycle of the multi-resource scheduling graph reinforcement learning generation model is:

[0074]

[0075] is the total reward obtained by taking actions in a training cycle, yes The reward function is , F 1 is the global time satisfaction of the task, F 2 is the global resource quantity satisfaction, F 3 Scheduling transportation costs for global resources; is the discount factor; is the long-term number of steps in a training cycle; For the period termination reward, the reward function is set to: , through the status The reward level is determined by the difference between the resource scheduling graph under the label and the pseudo-label resource scheduling graph.

[0076] The graph neural network is used as the policy network of the multi-resource scheduling graph reinforcement learning generation model. It generates the action probability distribution of the node according to the current node state, and takes the action with the highest probability as the execution action of the current node state. The policy network consists of four parts: input layer, two-layer graph neural layer and output layer. The activation functions of the two neural layers are ReLU and Softmax respectively. The structural function of the policy network is:

[0077]

[0078] s The current resource scheduling status; is the matching degree adjacency matrix of disaster points and resource points. The value of the adjacency matrix is ​​obtained by calculating the similarity of hidden features of nodes; , is the weight matrix of the graph neural layer; , Indicates execution of an action Probability, is the action performed in state s;

[0079] The conditional generative adversarial network framework is used, the pseudo-label data generated in step 1 is used as the positive sample, and the resource scheduling graph data generated by the multi-resource scheduling graph reinforcement learning generation model is used as the negative sample; the distribution of pseudo-label data is assumed to be , the distribution generated by the reinforcement learning generator is , cross entropy loss function The definitions are as follows.

[0080]

[0081]

[0082]

[0083] in Represents any initial disaster point, resource point location distribution, demand, and resource number input condition data in the pseudo-label data. is the pseudo label of the resource scheduling solution graph obtained by the solver under the conditional input; represents the fake data generated by the generator given the input condition x, and represents the judgment result of the discriminator on the pseudo-label data, Represents fake data generated by the discriminator in reinforcement learning The result of the above judgment;

[0084] Fixed the multi-resource scheduling reinforcement learning generation model and trained the discriminator D to maximize the cross entropy loss; fixed the discriminator D and trained the multi-resource scheduling reinforcement learning generation model to minimize the cross entropy loss; alternating iterative training until dynamic equilibrium is achieved.

[0085] Step 4 is as follows:

[0086] After all training data sets are trained, the trained model is published and the model parameters are saved locally. The graph neural network is used to obtain the graph embedding representation and the multi-resource scheduling reinforcement learning generation model is generated. It can be used to generate a multi-resource scheduling diagram. Input the plane structure diagram of the disaster point and resource point, and the multi-resource scheduling reinforcement learning model D generates the corresponding resource scheduling plan. The multi-resource scheduling plan generated by manual adjustment is regarded as the real resource scheduling diagram, and the plane structure diagram of the disaster point and resource point and the real resource scheduling diagram are retained in the training data set. When the training data set reaches a certain amount, load the local model parameters and use the newly added training data in the training data set to retrain the entire model. When all the training data are trained, publish the model and save the model parameters locally. In this way, the purpose of model tuning and retraining can be achieved.

[0087] It should be noted that any modifications, equivalent substitutions, improvements, etc. made outside the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims

1. A deep graph reinforcement learning generation method for multi-task and multi-criteria resource scheduling, characterized in that: The following steps are involved: Step 1, construct resource scheduling pseudo-label data through a genetic elite algorithm solver that follows the emergency scenario resource scheduling criterion; Step 2: construct a multi-task and multi-criteria multi-resource scheduling graph reinforcement learning generation model, and use a conditional generative adversarial learning framework to alternately train the multi-resource scheduling graph reinforcement learning generation model and the discriminator until dynamic equilibrium is reached; Step 3: The multi-resource scheduling graph intensity learning generation model generates a resource scheduling solution; Step 4: Manually adjust the resource scheduling plan, and generate a model for reward and punishment feedback multi-resource scheduling graph intensity learning; The specific process of step 1 is as follows: Under the criterion of highest global time satisfaction, highest global resource number satisfaction and lowest global resource scheduling transportation cost, the weight of each criterion is determined by using the hierarchical analysis method. , the objective function of multi-resource scheduling is: ； F1 is the global time satisfaction of the task, F2 is the global resource quantity satisfaction, and F3 is the global resource scheduling transportation cost; Through simulation, multiple groups of disaster point / resource point longitude and latitude locations, the number of resources required by each disaster point, and the number of resources available at each resource point are generated. The genetic elite algorithm is used to solve the objective function in parallel to obtain the optimal solution as the resource scheduling solution under different scenarios, which is used to construct pseudo-label data for resource scheduling. The specific process of step 2 is as follows: The longitude and latitude coordinates of the disaster point, the resource demand of the disaster point, and the longitude and latitude coordinates of the resource point are obtained. The resource supply of the resource point is used as the initial condition of the model to build a multi-resource scheduling graph reinforcement learning generation model. The multi-resource scheduling graph reinforcement learning generation model includes a feasible solution state feature extraction unit, state structure, action structure, reward mechanism structure, and strategy network structure. Among them, the feasible solution state feature extraction unit is used to characterize the matching degree between disaster points and resource points; based on the feasible solution state, the node hidden features are extracted through the graph neural network, and the resource scheduling plan diagram is gradually generated based on the node hidden features by using reinforcement learning; The state description of the multi-resource scheduling graph reinforcement learning generation model is: ， in is the adjacency matrix of the resource scheduling graph, which indicates the current resource scheduling status. is the node feature matrix, representing the current features of each node; The action description of the multi-resource scheduling graph reinforcement learning generation model is: ； Used to add a resource point-disaster point matching edge, Dispatch to point , T is the termination mark; the specific calculation process includes the following: Step 211, the probability of selecting the first node type as an emergency point and a resource point, and generating the node type according to the probability distribution of the resource point; Step 212, based on the node type generated in step 211, if it is a disaster point, the probability of selecting each disaster point as the first endpoint of the newly added edge is calculated, and the disaster point to be matched is generated by probability distribution; if it is a resource point, the probability of selecting each resource point as the first endpoint of the newly added edge is calculated, and the resource point to be matched is generated by probability distribution; Step 213, calculating the probability of each node matching the point selected in step 212, and generating matching points with the probability distribution; Step 214, connecting the points matched in step 212 and step 213, and generating a new edge; Step 215, determine whether to terminate, if the disaster point all needs are met, or there is no dispatchable resources, the termination flag T is True, otherwise T is False; The reward mechanism during the training cycle of the multi-resource scheduling graph reinforcement learning generation model is: ； is the total reward obtained by taking actions in a training cycle, yes The reward function is , F1 is the global time satisfaction of the task, F2 is the global resource quantity satisfaction, and F3 is the global resource scheduling transportation cost; is the discount factor; is the long-term number of steps in a training cycle; For the cycle termination reward, the reward function is set to: , through the status The reward level is determined by the difference between the resource scheduling graph under the label and the pseudo-label resource scheduling graph. The graph neural network is used as the policy network of the multi-resource scheduling graph reinforcement learning generation model. It generates the action probability distribution of the node according to the current node state, and takes the action with the highest probability as the execution action of the current node state. The policy network consists of four parts: input layer, two-layer graph neural layer and output layer. The activation functions of the two neural layers are ReLU and Softmax respectively. The structural function of the policy network is: s The current resource scheduling status; is the matching degree adjacency matrix of disaster points and resource points. The value of the adjacency matrix is ​​obtained by calculating the similarity of hidden features of nodes; , is the weight matrix of the graph neural layer; , Indicates execution of an action Probability, is the action performed in state s; The conditional generative adversarial network framework is used, the pseudo-label data generated in step 1 is used as the positive sample, and the resource scheduling graph data generated by the multi-resource scheduling graph reinforcement learning generation model is used as the negative sample; the distribution of pseudo-label data is assumed to be , the distribution generated by the reinforcement learning generator is , cross entropy loss function The definition is as follows: ； in Represents any initial disaster point, resource point location distribution, demand, and resource number input condition data in the pseudo-label data. is the pseudo label of the resource scheduling solution graph obtained by the solver under the conditional input; represents the fake data generated by the generator given the input condition x, and represents the judgment result of the discriminator on the pseudo-label data, Represents fake data generated by the discriminator in reinforcement learning The result of the above judgment; Fixed the multi-resource scheduling reinforcement learning generation model and trained the discriminator D to maximize the cross entropy loss; fixed the discriminator D and trained the multi-resource scheduling reinforcement learning generation model to minimize the cross entropy loss; alternating iterative training until dynamic equilibrium.

2. According to the method for generating deep graph reinforcement learning for multi-task and multi-criteria resource scheduling according to claim 1, it is characterized in that: The step 4 is specifically as follows: Through artificially adjusting the generated multi-resource scheduling scheme, the artificially adjusted multi-resource scheduling diagram is regarded as the real resource scheduling diagram, and the plane structure diagram of the disaster point and resource point and the real resource scheduling diagram are retained in the training data set; When the training data set reaches a certain amount, load the local model parameters and use the newly added training data in the training data set to train the multi-resource scheduling graph reinforcement learning generation model; When all training data is completed, the model is published and the model parameters are saved locally.

Images