An emergency material scheduling method based on fusion edge weight attention depth reinforcement learning

Abstract

The application discloses an emergency material scheduling method based on fusion of edge weight attention and deep reinforcement learning, comprising the following steps: constructing a Markov decision process based on an emergency material scheduling problem; constructing a policy network model, randomly generating emergency material scheduling simulation data under a certain distribution, inputting the emergency material scheduling simulation data into the policy network model, and performing offline training on the policy network model based on a REINFORCE algorithm with a rollback benchmark; and improving a locally optimal solution of the trained policy network model based on a 2-opt local search strategy to obtain an emergency material scheduling scheme. The network model trained offline can be used to quickly solve the problem within milliseconds. The policy network model constructed in the application is improved in combination with the 2-opt local search strategy, and compared with the existing deep reinforcement learning method, the application has higher practical application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent optimization scheduling, and particularly relates to an emergency material scheduling method fusing edge weight attention deep reinforcement learning. BACKGROUND

[0002] With the rapid development of human society, various disasters occur constantly along with human activities and adverse ecological environment, such as natural disasters like earthquake, snow disaster, typhoon, flood, etc. In order to guarantee the safe and sustainable development of human society, effective post-disaster emergency material scheduling is extremely important. Emergency material scheduling is to plan an effective emergency material distribution vehicle route by the emergency management center according to the material demand of disaster points, so as to timely distribute emergency materials from emergency warehouses to each disaster point. The research on this problem has important application value and can bring safety protection to governments, enterprises and individuals. This problem usually has the following characteristics:

[0003] (1) Emergence: At present, although the government has relatively perfected the disaster warning, there are still many emergent conditions in the disaster occurrence process, such as some regional problems like mountain collapse and road waterlogging caused by flood disaster, which will usually cause the interruption of emergency material scheduling task, thus causing the rescue task to be untimely and unreasonable, and will bring unpredictable disaster influence.

[0004] (2) High timeliness: For disaster areas, timely rescue is the common purpose of all tasks and the most important matter. Timely post-disaster emergency material scheduling not only guarantees the life demand of disaster personnel, but also prevents the disaster from further expanding. Therefore, the real-time response emergency material scheduling scheme also brings important guarantee for emergency rescue.

[0005] (3) Uncertainty: After the disaster occurs, in order to obtain the accurate disaster situation, the disaster is usually classified, and different disaster degrees will produce corresponding emergency material demand and have certain influence on road conditions. Since the influence degree of road conditions caused by disaster generally needs to be explored in the field to accurately collect corresponding data, in order to quickly generate a rescue scheme, it has important research significance to consider the emergency material scheduling scheme considering different disaster degrees.

[0006] Most of the current emergency material scheduling solution methods adopt heuristic methods, and formulate corresponding heuristic rules and algorithms for solution. However, the current solution method has the following shortcomings: (1) the traditional method usually needs to consume considerable solution time, however, after the disaster occurs, it is very important to arrange the emergency material distribution route for each disaster point in time, and the real-time response ability of the emergency material scheduling system is extremely important. (2) the supervised learning needs a large number of optimal solutions as training labels when training the network model, however, it is very difficult to obtain the optimal solution label of the scheduling optimization problem. (3) the emergency material scheduling problem has the characteristics of the real road network, and the nodes cannot be calculated by the Euclidean distance, and the existing deep reinforcement learning method cannot effectively solve the emergency material scheduling problem. SUMMARY

[0007] The purpose of the present application is to provide an emergency material scheduling method based on edge weight attention deep reinforcement learning to solve the above-mentioned problems existing in the prior art.

[0008] In order to achieve the above-mentioned purpose, the present application provides an emergency material scheduling method based on edge weight attention deep reinforcement learning, comprising the following steps:

[0009] Constructing a Markov decision process based on the emergency material scheduling problem;

[0010] Constructing a strategy network model, randomly generating emergency material scheduling simulation data under a certain distribution based on the Markov decision process, inputting the emergency material scheduling simulation data into the strategy network model, and offline training the strategy network model based on the REINFORCE algorithm with rollback benchmark;

[0011] Based on the 2-opt local search strategy, the local optimal solution of the trained strategy network model is improved to obtain the emergency material scheduling scheme.

[0012] Optionally, constructing a Markov decision process based on the emergency material scheduling problem comprises the following steps:

[0013] Constructing a state s = {s g ,s agent}, wherein s g is a global state, s agent is an agent state, and the global state s g represents the overall graph feature information; the agent state s agent = {h1, r k}, including the node hidden feature h1 selected by the agent at the last step and the current distribution vehicle remaining capacity r k , wherein l e V, k e K ;

[0014] The state transition is performed on the agent action, the remaining material amount of the current vehicle is calculated, and the current position of the vehicle is modified as the node selected by the agent action;

[0015] The cumulative return of the emergency material dispatching is set as a negative number of the total distribution time.

[0016] Optionally, the agent action a t ∈A is a node selected by the agent in a state at a time step t; the state transition represents that the agent action a t The corresponding node is added to the selected node set.

[0017] Optionally, the policy network model is composed of a decoder-encoder, wherein:

[0018] The encoder is composed of an embedding layer and a plurality of attention modules, the embedding layer maps the input node features to higher-dimensional space features, each attention module obtains the hidden features of the node by calculating the attention degree between the nodes, and the attention module is composed of an edge weight attention layer and a feedforward layer;

[0019] The decoder includes an embedding layer, an edge weight attention layer, and a single-head edge weight attention layer, the decoder outputs an action selection probability vector based on the high-level features extracted by the encoder, and guides the selection of the next time action of the agent.

[0020] Optionally, the feature transmission calculation formula of the encoder is as follows:

[0021]

[0022]

[0023]

[0024]

[0025] Wherein, formula (4) is an embedding layer feature transmission calculation formula, W X is an embedding layer network parameter, x i represents the input features of node i, including the node material demand and the node disaster level, formula (5) is an edge weight attention layer feature transmission calculation formula, BatchNorm represents batch normalization processing, l∈[1,N] represents the lth attention module, formula (6) is a feedforward layer feature transmission calculation formula, formula (7) is a forward propagation calculation formula, ReLu is a neuron activation function, W1 FF , W2 FF are feedforward layer network parameters.

[0026] Optionally, the feature transfer calculation of the decoder is as follows:

[0027]

[0028]

[0029] p = SEA(q) c (10)

[0030] Formula (8) is the formula for calculating the feature transfer of the embedding layer, W C The parameters of the embedding layer are represented by , concat is the feature concatenation function, and formula (9) is the formula for calculating the edge weight attention layer. The attention query vector is used to calculate the attention of the nodes accessible to the current vehicle. The access nodes include emergency warehouse points and disaster-stricken points where the required material quantity is less than the current vehicle's remaining capacity. Formula (10) represents the single-head edge weight attention calculation formula.

[0031] Optionally, the REINFORCE algorithm with rollback baseline consists of a policy network θ and a baseline network θ. bl composition.

[0032] Optionally, the REINFORCE algorithm with rollback baseline estimates the policy gradient by calculating the agent's cumulative reward and trains the agent's policy. The specific process includes:

[0033] For a given instance s, the policy network θ outputs the agent's action probability vector p at each time step. θ (π t Based on the action probability vector, policy π is output using a sampling selection method. t =sample(p θ (π|s));

[0034] The reference network θ bl Based on the action probability vector output by the benchmark network Output strategy using greedy selection

[0035] Evaluation of Expected Cumulative Returns of Strategies Based on Monte Carlo Algorithm where R(π) is the strategy π={π1,π2,...,π T The cumulative reward of}; calculate the policy gradient using the REINFORCE algorithm with rollback baseline, and update the parameters of the policy network using gradient descent, as shown in formulas (1) and (2):

[0036]

[0037] θ = Adam(θ, ∇ θ L(θ | s)) (2)

[0038] After each round of training, the policy network θ and the baseline network θ bl are compared, and if the solution output by the policy network θ is significantly better than the solution output by the baseline network θ bl at a significance level of alpha = (0.05) in a t-test, the baseline network θ is updated in a rollback manner θ bl ← θ.

[0039] Optionally, the process of improving the locally optimal solution of the trained policy network model based on the 2-opt local search strategy comprises:

[0040] S1. Randomly select two nodes of the current sub-loop r, and flip the path between the node pair to form a new sub-loop r';

[0041] S2. If the sub-loop r' is better than r, update the current sub-loop r = r', reset the iteration number Iter to 0 and return to Step 1, otherwise, the iteration number Iter = Iter + 1 and return to S1;

[0042] S3. If the iteration number reaches the maximum iteration number Iter = MaxIter and r is still not improved, end the 2-opt local search, and return the final r as the optimal sub-loop.

[0043] The technical effects of the present application are:

[0044] (1) Fast solution speed. Compared with traditional heuristic solving methods, the present application uses an offline trained network model to quickly solve in milliseconds.

[0045] (2) Suitable for real road network environment. A strategy network model capable of effectively handling emergency material dispatching under non-Euclidean distance is designed, which combines edge weight attention mechanism to effectively integrate node distance matrix information. Compared with existing deep reinforcement learning methods, the present application has more practical application value.

[0046] (3) Good solution effect. The REINFORCE reinforcement learning algorithm with rollback baseline is used to train the policy network model, and the 2-opt local search strategy is combined to further optimize the solution quality based on the model solution. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and do not limit the present application. In the drawings:

[0048] Figure 1 is an overall flowchart in an embodiment of the present application;

[0049] Figure 2 is an edge weight attention structure diagram in an embodiment of the present application;

[0050] Figure 3 is a strategy network encoder structure diagram in an embodiment of the present application;

[0051] Figure 4 is a strategy network decoder structure diagram in an embodiment of the present application;

[0052] Figure 5 is a 2-opt local search strategy diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0055] Embodiment one

[0056] As shown in the figure, the present embodiment provides an emergency material scheduling method based on fusion of edge weight attention and deep reinforcement learning, comprising the following steps: Figures 1-5

[0057] S1, according to the emergency material scheduling problem, a Markov decision process is constructed:

[0058] (1) state s = {s g ,s agent} is divided into global state s g and agent state s agent . The global state s g represents the overall graph feature information; the agent state s agent = {h l ,r k} contains the node hidden feature h l selected by the agent at the last step and the current remaining capacity r k of the distribution vehicle, where l e V and k e K.

[0059] (2) the agent action a t e A is the node selected by the agent at time step t state. ​

[0060] (3) State transition represents that a t The corresponding node joins the selected node set, and the remaining amount of goods of the current vehicle is calculated, and the current location of the vehicle is modified to a t The selected node.

[0061] (4) For the emergency goods scheduling problem, the objective function is to minimize the total emergency goods delivery time, and the smaller the total delivery time, the higher the cumulative return of the agent, so the cumulative return of the problem is defined as the negative of the total delivery time.

[0062]

[0063] S2, randomly generate emergency goods scheduling simulation data under a certain distribution, input it into the policy network, and use the REINFORCE algorithm with rollback benchmark to train the policy network model offline.

[0064] S2.1, the policy network model is composed of a decoder-encoder, and the specific structure is as follows:

[0065] S2.1.1, the encoder is composed of an embedding layer and N attention modules, the embedding layer maps the input node features to a higher dimensional space feature, and each attention module obtains the hidden features of the nodes by calculating the attention degree between the nodes, which is composed of an edge weight attention layer and a feedforward layer. The edge weight attention mechanism and the structure of the encoder are shown in FIGS. 1 and 2. The encoder feature transmission calculation is shown in equations (4), (5), (6) and (7): Figure 2 、 3

[0066]

[0067]

[0068]

[0069]

[0070] wherein equation (4) is an embedding layer feature transmission calculation formula, W X is an embedding layer network parameter, x i represents the input feature of node i, including the node goods demand and the node disaster level, equation (5) is an edge weight attention layer feature transmission calculation formula, BatchNorm represents batch normalization processing, l∈[1, N] represents the lth attention module, equation (6) is a feedforward layer feature transmission calculation formula, equation (7) is a forward propagation calculation formula, ReLu is a neuron activation function, W1 FF , W2 FF are feedforward layer network parameters. ​

[0071] S2.1.2, with the agent each time to perform the action and interact with the environment, the decoder uses the high-level features extracted by the encoder to output the action selection probability vector, guiding the agent next time action selection. The structure of the decoder is shown in Figure 4 The decoder shown in FIG. 2 includes an embedding layer, an edge weight attention layer, and a single-head edge weight attention layer. The decoder feature propagation calculation is shown in equations (8), (9), and (10):

[0072]

[0073]

[0074] p = SEA(q c ) (10)

[0075] wherein equation (8) is the embedding layer feature propagation calculation formula, W C represents the embedding layer network parameter, and concat is the feature concatenation function, equation (9) is the edge weight attention layer calculation formula, Attention is calculated on the accessible nodes of the current vehicle as the attention query vector, including the emergency warehouse points and the disaster points with a required amount of goods less than the remaining capacity of the current vehicle, and equation (10) represents the single-head edge weight attention calculation formula.

[0076] S2.2, the REINFORCE algorithm with rollback benchmark is composed of a policy network θ and a benchmark network θ bl The algorithm estimates the policy gradient by calculating the cumulative return of the agent and trains the agent policy, and the specific training process is as follows:

[0077] S2.2.1, for a given instance s, the policy network θ outputs the action probability vector p θ (π t |s) of the agent at each time, and the policy π t is output in a sampling selection manner according to the probability vector. θ

[0078] S2.2.2, the benchmark network θ bl then the action probability vector output by the benchmark network is output in a greedy selection manner.

[0079] S2.2.3, evaluate the expected cumulative return of the policy according to the Monte Carlo algorithm wherein R(π) is the policy π = {π1, π2,..., π T ​accumulated returns. Strategy gradients are computed by REINFORCE algorithm with baseline, and the policy network parameters are updated in the way of gradient descent, as shown in equations (1)(2):

[0080]

[0081] θ = Adam(θ, ∇ θ L(θ | s)) (2)

[0082] S2.2.4, At the end of each round of training, compare the policy network θ and the baseline network θ bl , if the solution output by the policy network is significantly better than the solution output by the baseline network, then update the baseline network in a rollback manner θ bl ← θ.

[0083] S3, The trained policy network model can quickly solve the emergency material scheduling problem using the greedy action selection strategy, but there is still some room for improvement in the local optimal solution for some difficult instances. In order to improve the quality of the solution, a 2-opt local search strategy is used, as shown in the following figure: Figure 5

[0084] S3.1, Randomly select two nodes of the current sub-loop r, and flip the path between the node pair to form a new sub-loop r'.

[0085] S3.2, If the sub-loop r' is better than r, update the current sub-loop r = r', reset the iteration number Iter to 0 and return to Step 1, otherwise Iter = Iter + 1 and return to S3.1.

[0086] S3.3, If the iteration number reaches the maximum iteration number Iter = MaxIter and r is still not improved, end the 2-opt local search, and return the final r as the optimal sub-loop.

[0087] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. An emergency material scheduling method of fusing edge weight attention depth reinforcement learning, characterized in that, The method comprises the following steps: constructing a Markov decision process based on an emergency material dispatching problem; constructing a policy network model, randomly generating emergency material dispatching simulation data under a certain distribution based on the Markov decision process, inputting the emergency material dispatching simulation data into the policy network model, and performing offline training on the policy network model based on a REINFORCE algorithm with a rollback benchmark; improving a locally optimal solution of the trained policy network model based on a 2-opt local search strategy to obtain an emergency material dispatching scheme; the policy network model is composed of a decoder-encoder, wherein: the encoder is composed of an embedding layer and a plurality of attention modules, the embedding layer maps input node features to higher-dimensional space features, each attention module obtains the hidden features of a node by calculating the attention degree between nodes, and the attention module is composed of an edge weight attention layer and a feedforward layer; the decoder includes an embedding layer, an edge weight attention layer, and a single-head edge weight attention layer, and the decoder outputs an action selection probability vector based on the high-level features extracted by the encoder to guide the next-time action selection of the agent; the feature transmission calculation formula of the encoder is as follows: wherein, formula (4) is an embedding layer feature transmission calculation formula, W X is an embedding layer network parameter, x i represents the input feature of node i, including the node material demand and the node disaster level, formula (5) is an edge weight attention layer feature transmission calculation formula, BatchNorm represents batch normalization processing, l∈[1, N] represents the lth attention module, formula (6) is a feedforward layer feature transmission calculation formula, formula (7) is a forward propagation calculation formula, and ReLu is a neuron activation function, W1 FF , W2 FF are feedforward layer network parameters; the feature transmission calculation of the decoder is as follows: wherein formula (8) is an embedding layer feature transmission calculation formula, W C denotes the embedding layer network parameters, concat is a feature concatenation function, formula (9) is an edge weight attention layer calculation formula, which is an attention query vector for attention calculation on the accessible nodes of the current vehicle, and the accessible nodes include emergency warehouse points and disaster points with a required amount of materials less than the remaining capacity of the current vehicle, and formula (10) represents a single-head edge weight attention calculation formula.

2. The emergency material scheduling method of claim 1, wherein, constructing a Markov decision process based on an emergency material dispatching problem comprises the following steps: The construction state s = {s g ,s agent} is obtained, wherein s g is a global state, s agent is an agent state, the global state s g indicates overall graph feature information; the agent state s agent = {h1, r k} includes a node hidden feature h1 selected by the agent in the last step and a current remaining capacity r k of the delivery vehicle, wherein l e V and k e K. performing state transition on the agent action, calculating the remaining material quantity of the vehicle, and modifying the current location of the vehicle to the node selected by the agent action; setting the cumulative return of the emergency material dispatching as the negative number of the total distribution time.

3. The emergency material scheduling method of claim 2, wherein, the agent action a t ∈ A is the node selected by the agent at time step t state; the state transition represents the agent action a t the corresponding node joins the selected node set.

4. The emergency material scheduling method of claim 1, wherein, The REINFORCE algorithm with a rollback baseline consists of a policy network θ and a baseline network θ bl comprise.

5. The emergency material scheduling method of claim 4, wherein, The REINFORCE algorithm with a rollback benchmark estimates the policy gradient by calculating the cumulative return of the agent and trains the agent policy, and the specific process comprises: For a given instance s, the policy network θ outputs a vector of action probabilities p θ (π t |s) at each time step for the agent t = sample(p θ (π|s)) based on the vector of action probabilities. The reference network θ bl An action probability vector outputted based on the reference network , outputting the strategy in a greedy selection manner ; Evaluating the expected cumulative return of a policy based on a monte carlo algorithm where R(π) is the cumulative return of a policy π = {π1, π2,..., π T} ; the policy gradient is calculated by the REINFORCE algorithm with a rollback baseline, and the parameters of the policy network are updated in the form of gradient descent, as shown in equations (1) (2): At the end of each round of training, the policy network θ and the baseline network θ bl are compared, and if the solution output by the policy network θ is significantly better than the solution output by the baseline network θ bl at a significance level of a = (0.05) in a t-test, then the baseline network θ is rolled back to θ bl ← θ.

6. The method of claim 1, wherein, the process of improving the locally optimal solution of the trained policy network model based on the 2-opt local search strategy comprises: S1. randomly selecting two nodes of the current sub-loop r and flipping the path between the node pair to form a new sub-loop r'; S2. if the sub-loop r' is better than r, update the current sub-loop r = r', reset the iteration number Iter to 0 and return to Step 1, otherwise, iteration number Iter = Iter + 1 and return to S1; S3. if the iteration number reaches the maximum iteration number Iter = MaxIter and r is still not improved, end the 2-opt local search, and return the final r as the optimal sub-loop.

Images