Flexible job shop scheduling method based on act-critic multi-agent deep reinforcement learning

Abstract

The invention belongs to the technical field of production plans, and particularly relates to a flexible job shop scheduling method based on action-critic multi-agent deep reinforcement learning, which comprises the following steps of: (1) modeling a flexible job shop scheduling environment by adopting a heterogeneous disjunction graph method, converting a scheduling decision problem into a sequential decision problem, and establishing a flexible job shop scheduling model; establishing a distributed partially observable Markov decision process eight-tuple model, and solving the model by using multi-agent deep reinforcement learning; (2) extracting a current state from a heterogeneous disjunction graph environment scheduled by the flexible job shop; (3) respectively fitting the strategy function and the value function by adopting a heterogeneous graph neural network; (4) the workpiece intelligent agent and the machine intelligent agent output the probability of each workpiece to be machined and the probability of each selectable machine as intelligent agent actions of reinforcement learning; (5) designing a reward function to evaluate the whole scheduling decision, and updating the strategy functions of the workpiece agent and the machine agent and the weight parameters of the value function of the central controller by using an act-critic algorithm; (6) state transition is carried out; (7) updating strategy network and value network parameters; the scheduling problem of the flexible job shop can be processed in real time, and the method has the advantages of being high in real-time performance and high in flexibility.

Description

Technical Field

[0001] The present invention belongs to the technical field of production planning, and is used for production scheduling optimization of a flexible job shop, and specifically relates to a flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning. Background Art

[0002] As an important link in the manufacturing system, production scheduling directly affects the production cost and production efficiency of the enterprise. Classical production scheduling problems can be divided into job shop scheduling problems, flow shop scheduling problems, parallel machine scheduling problems, single machine scheduling problems and open shop scheduling problems. Under severe economic conditions or in a highly competitive business environment, it is increasingly important to formulate optimal scheduling. The flexible job-shop scheduling problem (FJSP) is an extension of the classic job-shop scheduling problem. In addition to the process sequencing, the FJSP problem also adds a decision step about workpiece routing, namely machine selection. As a combinatorial optimization problem with wider applications and more complex solutions, the FJSP problem has aroused great interest among researchers.

[0003] Precise algorithms such as dynamic programming and branch-and-bound can obtain optimal solutions under certain conditions, but the calculations are complex. To solve this problem, scholars have proposed meta-heuristic algorithms to balance the solution results and the calculation time. However, the calculation time grows exponentially with the scale of the scheduling problem. In the era of Industry 4.0, scholars have used machine learning techniques to solve scheduling problems, such as reinforcement learning (RL), which uses linear functions or tables (Q-tables) that can be gradually trained to represent scheduling agents, trial-and-error interactions between agents and the environment, and takes optimal actions based on rewards. In research, reinforcement learning algorithms have better solution results than other algorithms. Deep reinforcement learning (DRL) uses deep neural networks to parameterize policies or approximate value functions, and features are represented by matrices or vectors. In recent years, many studies have explored the use of DRL algorithms to solve FJSP problems. However, in practical applications, the centralized architecture of single-agent deep reinforcement learning algorithms lacks flexibility and autonomy, and the algorithms of agents in multi-agent reinforcement learning (MARL) are difficult to stabilize and have slow learning speeds.

[0004] At present, in the FJSP problem, the problem of both the state space and the action space being too large has not been well solved. This invention adopts a multi-agent deep reinforcement learning (MADRL) algorithm with a centralized training with decentralized execution (CTDE) architecture, and uses two agents, namely, a workpiece agent and a machine agent, according to the two decision steps of the FJSP problem. Thus, this invention solves the problem of both the state space and the action space being too large. Summary of the invention

[0005] The purpose of the present invention is to propose a flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning.

[0006] The purpose of the present invention is to achieve heterogeneous disjunctive graph modeling of flexible job shop scheduling problems, scheduling decisions based on heterogeneous graph neural networks, and scheduling decision optimization based on actor-critic deep reinforcement learning algorithms, thereby optimizing the performance indicator of minimizing the maximum completion time in flexible job shop scheduling.

[0007] In order to achieve the above object, the technical solution of the present invention is to provide a flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning, comprising the following steps:

[0008] (1) The flexible job shop scheduling environment is modeled by using the heterogeneous disjunctive graph method. The scheduling decision problem is converted into a sequential decision problem. A decentralized partially observable Markov decision process (Dec-POMDPs) octet ε=(S, A, P, R, Z, O, K, γ) model is established. The minimization of the maximum completion time is used as the performance evaluation indicator of the model, and deep reinforcement learning is used to solve the model.

[0009] (2) Extract the current state s∈S from the heterogeneous disjunctive graph environment of flexible job shop scheduling;

[0010] The state s is composed of the node information and connection arcs of the heterogeneous disjunctive graph, where the heterogeneous disjunctive graph H = (O, M, C, E, F), where O is the set of process nodes, C is the set of directed connection arcs connected by the precedence constraint relationship between processes in the workpiece, and E is the set of disjunctive arcs between processes that can be processed on the same machine. The heterogeneous disjunctive graph adds the set of machine nodes M and the set of optional arcs F between processes and machines. The process node information describes the relevant features of the processing process, and the machine node information describes the machine scheduled information and processing information.

[0011] (3) Constructing the strategy function π of the workpiece agent and the machine agent o (s;θ) and π m (s; θ); construct the value function V(s; φ) of the central controller; use heterogeneous graph neural network to fit the strategy function and value function respectively. Strategy function π o (s;θ) and π m (s; θ) is fitted through the policy network, the value function V(s; φ) is fitted through the value network, and the policy function and value function are initialized π o (s;θ0),π m (s; θ0), V(s; φ0).

[0012] Among them, θ is the set of internal weights ω and biases b of the neural network in the policy network, and φ is the set of internal weights ω and biases b of the neural network in the value network.

[0013] (4) The workpiece agent and the machine agent output the probability of each workpiece to be processed and the optional machine as the agent action of reinforcement learning;

[0014] According to the workpiece to be processed and the optional machine set for each process, a task set is established for each executable machine, and the global state is input into the heterogeneous graph neural network of the workpiece agent and the machine agent. The SoftMax activation function is used to calculate the probability distribution of selecting the process task and the probability distribution of the optional machines. Then, when selecting the workpiece and machine according to the probability P, random selection or greedy method is adopted.

[0015] Among them, random selection allows the agent to have more possibilities when exploring the environment, and can better learn state characteristics and optimal strategies; the greedy strategy gives priority to the machine combination of processes and tasks that currently have the highest probability of obtaining each decision. It aims to use the optimal strategy currently learned by the agent to make the best choice, thereby reducing the uncertainty brought by exploration and quickly moving towards the optimal solution.

[0016] (5) Design a reward function R(t) to evaluate the entire scheduling decision and use the actor-critic algorithm to update the weight parameters θ of the workpiece agent and machine agent policy functions o and θ m and the weight parameter φ of the central controller’s value function to update the decision-making schedule;

[0017] In the design of reward function, the main consideration is the minimization of maximum completion time, which is a single performance indicator. Due to different scheduling problems, the minimization of the maximum completion time obtained by scheduling is different. In order to uniformly evaluate the scheduling behavior, the negative of the maximum completion time is used as a reward. The definition of the minimization of the maximum completion time is as follows:

[0018]

[0019] (6) Perform state transfer P(s′s,a) and store the state s, action a, reward r, and next state s′ into the memory pool;

[0020] Action a is executed from state s, and the state is transferred. The workshop environment is updated, state s′ is extracted, and state s, action a, reward r, and next state s′ are stored in the memory pool in sequence.

[0021] (7) Update the policy network and value network parameters;

[0022] When the workpiece agent and machine agent have collected enough historical data, they start to use this data to update the parameters of the policy network and value network. Each time the update is performed, the policy network is optimized using the truncated importance sampling method based on the latest collected data. Then, the value network V(s) is used to calculate the advantage function of each state-action To evaluate the quality of the action relative to the baseline. When the policy network is updated, the parameters are updated by optimizing the objective function. The optimization objectives of the workpiece agent and machine agent policy network are:

[0023]

[0024] Among them, σ is the hyperparameter of the entropy coefficient, B represents the batch size, is the number of agents, ∈ is the pruning hyperparameter, is the entropy of the strategy, and CLIP() represents the clipping function. Advantage function The GAE method is used for calculation. Control the update range of new and old strategies.

[0025] At the same time, the optimization objectives of the central controller are:

[0026]

[0027] At each update, the agent randomly extracts batch_size pieces of historical data from the collected historical data and performs multiple gradient descent updates until convergence.

[0028] Furthermore, in the step (1), the heterogeneous disjunctive graph model is represented by a set of five-tuples H = (O, M, C, E, F);

[0029] in O is a collection of process nodes.

[0030] C is a set of directed connection arcs connected by priority constraint relationships between processes in a workpiece.

[0031] E is the set of disjunctive arcs between processes that can be processed on the same machine.

[0032] The heterogeneous disjunctive graph adds a set of machine nodes M and a set of optional arcs F between processes and machines.

[0033] Furthermore, in step (2), the process node information is used to describe the workpiece-related processing information in the scheduling problem, and the mean is normalized in order to adapt to problems of different sizes.

[0034] Furthermore, in step (2), the machine node information is used to describe the scheduled information and available information of the machine in the scheduling problem. In order to adapt to problems of different sizes, the mean is normalized.

[0035] Furthermore, in step (3), for a graph G = (V, E), the graph neural network is used to calculate the embedding of each node v (v∈V), which is defined as follows:

[0036]

[0037] After the Lth update iteration, the HGNN layer can obtain the p-dimensional embedding vectors of the process nodes and machine nodes, perform graph pooling through sparse matrix multiplication, calculate the average value of the node embedding to obtain two fixed-dimensional embedding vectors, and then combine them into a heterogeneous disjunctive graph H t The graph embedding representation vector h H , the definition is as follows:

[0038]

[0039] Beneficial effects of the present invention:

[0040] 1. The present invention takes into account the performance indicator of minimizing the maximum completion time in flexible job shop scheduling, transforms the scheduling decision problem into a sequential decision problem using heterogeneous disjunctive graphs, establishes a flexible job shop scheduling model of Dec-POMDPs, and solves the model using the actor-critic algorithm. Through training, a better scheduling decision arrangement can be obtained. At the same time, the trained intelligent agent can quickly solve new flexible job shop scheduling problems of the same scale.

[0041] 2. The present invention is based on the actor-critic deep reinforcement learning algorithm to solve the relevant flexible job shop scheduling problems. The algorithm selects the process-machine according to the input state characteristics, which is more in line with the scheduling decision-making process of the actual order-responsive production and manufacturing system.

[0042] 3. The present invention uses a heterogeneous graph neural network model to fit the strategy function and the value function, inputs the manufacturing system processing state feature data into the model, combines the reinforcement learning reward and punishment mechanism, and selects the optimal process-machine as the strategy for each scheduling decision.

[0043] 4. The actor-critic deep reinforcement learning algorithm proposed in the present invention has the advantages of strong real-time performance and high flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a framework diagram of the flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning of the present invention.

[0045] Figure 2 It is a heterogeneous disjunctive graph model of FJSP instances.

[0046] Figure 3 The process of scheduling state transition for the present invention.

[0047] Figure 4 The present invention calculates the scheduling action process.

[0048] Figure 5 This is the training data diagram for the MK02 case.

[0049] Figure 6 This is the Gantt chart for the MK02 case. DETAILED DESCRIPTION

[0050] The present invention is further described in conjunction with specific implementation modes.

[0051] The present invention is a flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning, and its framework is as follows Figure 1 , including the following parts:

[0052] (1) FJSP problem definition: The flexible job shop scheduling problem is described as follows: the job set J = {J i :i=1,…,n} is composed of machine set M={M k :k=1,…,m} for processing, each workpiece J i Contains NO i process, among which O ih For workpiece J i The hth process. Each process O ihAll can be found in the optional machine set M ih Any M m Processing is carried out on ih On machine M m The processing time on is P ihm As shown in Table 1, the FJSP problem. The production scheduling goal is the maximum completion time C max minimize.

[0053] Table 1 FJSP examples for three workpieces and three machines

[0054]

[0055] (2) Disjunctive graph representation of FJSP problem: The learning environment is modeled using a disjunctive graph. The disjunctive graph model divides discrete time steps from the perspective of process sequencing, which can more conveniently add interaction constraints, making the scheduling process closer to the real production environment. In the FJSP problem, there are multiple connection relationships between process nodes and machine nodes. Therefore, the disjunctive graph model of the FJSP problem can be more accurately represented as a heterogeneous disjunctive graph model, which can more comprehensively reflect the relationship between nodes. This paper modifies the disjunctive graph and defines a heterogeneous disjunctive graph H = (O, M, C, E, F), where O is the set of process nodes, C is the set of directed connection arcs connected by the priority constraint relationship between processes in the workpiece, and E is the set of disjunctive arcs between processes that can be processed on the same machine. The heterogeneous disjunctive graph adds a set of machine nodes M and a set of optional arcs F between processes and machines. As Figure 1 Shown is a heterogeneous disjunction graph of the FJSP instance in Table 1 above.

[0056] (3) FJSP is modeled as a decentralized partially observable Markov decision process: In the MADRL algorithm, the key to solving the FJSP problem is to transform the scheduling problem into a Markov decision process. In a multi-agent system, two agents perform specific tasks in the same environment. By defining states, state transitions, actions, and reward functions, the flexible job shop scheduling process is mapped into a decentralized partially observable Markov decision process. Dec-POMDPs consists of a tuple (S, A, P, R, Z, O, K, γ), where k∈K is the set of workpiece agents and machine agents; S = [O 1 ,O 2 ,…,O k ] is the state space; O k is the local observation of the agent; A=[a1,a2,…,a k ] is the joint action space, which is jointly performed by agents with the same reward set R; Z is the observation function; P is the state transition function; γ is the discount factor.

[0057] State: The state s at the decision time step t Represented as a heterogeneous disjunctive graph in Indicates the processing order of the scheduled operations, is the disjunctive arc of the unscheduled process, and the connecting arc F of the processing machine. The state characteristics of the scheduling problem are described from the perspectives of process nodes and machine nodes, as shown in Table 2. The workpiece information uses the mean to describe the overall picture of the scheduling problem. In order to adapt to problems of different sizes, the mean is normalized. The process information describes the relevant features of the next process of the workpiece, and the machine information describes the scheduled information and available information of the machine.

[0058] Table 2 Scheduling characteristics

[0059]

[0060] State transition: After the two agents have completed their actions, their states transition to the state of the next decision time step and generate a new heterogeneous disjunction graph. Figure 2 As shown in the figure, in the scheduling environment of heterogeneous disjunctive graph, the undirected disjunctive arcs between process nodes in the initial state are ignored to reduce the computational complexity. The scheduling environment gives the state at the next moment through the state transition function, which is a conditional probability density function.

[0061] Action: At the decision time step, the scheduling environment adds the next process of the workpiece to the ready process set, and then the workpiece agent selects a process task from the ready process set. Each selection corresponds to a behavior. Since there are order constraints between the processes of each workpiece, effective actions can only select processes in the current ready process set. The mask mechanism is used to set the probability of selecting processes outside the current ready process set to -∞. The machine agent selects a machine from the available machine set of this process task for processing. The processing time of the process task on the machine that cannot be processed is set to the average processing time of the process task on all available machines.

[0062] Reward function: In Dec-POMDP, each agent observes its own state in the environment, takes its own actions, and receives the same reward to collaboratively complete the scheduling task. This paper takes the negative of the maximum completion time in the FJSP problem as the reward, and the discounted return of the workpiece agent and the machine agent is the discounted cumulative reward. The maximum completion time minimization is defined as follows:

[0063]

[0064] (4) Extracting state features from heterogeneous disjunctive graphs using graph neural networks: In order to learn scheduling policies using the MADRL algorithm, the neural network must be able to process the states represented by heterogeneous disjunctive graphs. tThere are two different types of nodes in the graph. Therefore, this paper uses heterogeneous graph neural networks to extract heterogeneous disjunctive graphs and node features. The algorithm uses a two-stage graph embedding to transform the graph data into a learnable and computable representation. The MADRL algorithm uses a graph convolutional neural network (GCN) to extract state features from heterogeneous disjunctive graphs. For a graph G = (V, E), GCN is used to calculate the embedding of each node v (v∈V), which is defined as follows:

[0065]

[0066] in, is the embedding of node v at layer l; is the original feature of node v; θ l is a learnable parameter; ε (l) is any number of weight matrices, N v is the set of adjacent nodes of node v.

[0067] The two-stage heterogeneous disjunctive graph embedding process is regarded as an HGNN layer. After the Lth update iteration, the HGNN layer can obtain the p-dimensional embedding vectors of the process nodes and machine nodes, perform graph pooling through sparse matrix multiplication, calculate the average value of the node embedding to obtain the embedding vectors with two fixed dimensions, and then combine them into the heterogeneous disjunctive graph H t The graph embedding representation vector h H , the definition is as follows:

[0068]

[0069] Among them, [,] is the vector concatenation operator, O and M are the embedding sets of process nodes and machine nodes respectively.

[0070] In order to improve the robustness of the model, a batch normalization layer is added after the HGNN layer to accelerate the training. The definition is as follows:

[0071]

[0072] At the decision time step, the scheduling environment is represented as a heterogeneous disjunctive graph state Use the above formula to calculate the embedding vectors of process nodes, machine nodes and heterogeneous disjunction graphs and h H .

[0073] (5) Parameterized strategy: The scheduling action is calculated by the MADRL algorithm. Figure 4 As shown. and h H Afterwards, the workpiece agent can obtain the input vector h O, the machine agent can obtain the input vector h M , as shown below:

[0074]

[0075] Among them, the vector h O The shape is |O×2p, and the vector h M The shape is M×2p. O and h M They are respectively input into the process selection network and the machine selection network to output scheduling actions.

[0076] The workpiece agent and machine agent use fully connected neural networks to calculate the score of each process in the ready process set and each machine in the available machine set of the process task, respectively. The definition is as follows:

[0077]

[0078] in, and are the policy network parameters of the artifact agent and the machine agent, respectively.

[0079] Then, the SoftMax activation function is used to calculate the probability distribution of the selected process task and the probability distribution of the available machines, which are defined as follows:

[0080]

[0081] Finally, during the training process, P is sampled or greedily O and P M Select behavior a O and behavior a M The greedy way selects O with the highest probability ih and M m As an act O and behavior a M , and the sampling method can provide more possibilities. When using the SoftMax activation function to calculate P O and P M Previously, the process and machine masks were used to mask the processes that have been scheduled, the processes that violate the priority constraints, and the machines that have been scheduled. O and scr M Set to -∞, so that P O and P M Equal to 0, achieving the purpose of shielding.

[0082] (6) Implementation of FJSP problem based on MADRL algorithm: This paper proposes a MADRL algorithm using CTDE architecture. During the training of MADRL algorithm, both the workpiece agent and the machine agent use the advantage function estimation value of the value network in the central controller. To evaluate the joint actions of the two agents. During the training process, each agent transmits observations to the central controller, which obtains rewards from the scheduling environment and then calculates the advantage function estimate Return two agents, The GAE method is used for calculation, and the definition is as follows:

[0083]

[0084] Among them, V ω (s t ) is the state value function. After the training process is completed, there is no need for communication between the workpiece agent and the machine agent, and only the arranged policy network is used to make decisions in real time. The input of each agent is the observation of all agents: S = [O 1 ,O 2 ,…,O k ], the output is the probability distribution of process tasks and available machines in the action space. The input of the central controller is also the observed state S of the two agents, and the output is the score of the state by the value network. The workpiece agent and the machine agent are trained by updating the "critic" with parameter ω on the central controller. The two agents learn through the entropy increase objective and the minimization of the pruning objective. The MADRL algorithm controls the update amplitude of the new and old strategies through the importance sampling weights, making the learning process of the agent more stable and avoiding the update of extreme strategies.

[0085] The goal of the workpiece agent and the machine agent is to find the optimal process sequencing strategy and machine selection strategy so that the expected cumulative reward in all possible states is maximized, which is defined as follows:

[0086]

[0087] The above content is a further detailed description of the present invention in combination with specific implementation methods, and the specific implementation is limited to these descriptions. For ordinary technicians in the technical field of the present invention, some simple reasoning or substitution can be made without departing from and conceiving the present invention, and these equivalent forms also belong to the scope of this application.

Claims

1. A flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning, characterized in that: The following steps are involved: (1) The flexible job shop scheduling environment is modeled by using the heterogeneous disjunctive graph method, the scheduling decision problem is converted into a sequential decision problem, and the eight-tuple of decentralized partially observable Markov decision processes (Dec-POMDPs) ε = (S, A, P, R, Z, O, K, γ ) The model uses the minimization of the maximum completion time as the performance evaluation indicator of the model and uses deep reinforcement learning to solve the model; (2) Extract the current state s∈S from the heterogeneous disjunctive graph environment of flexible job shop scheduling; state s It is composed of node information and connection arcs of a heterogeneous disjunctive graph, wherein the heterogeneous disjunctive graph H = (O, M, C, E, F), where O is the set of process nodes, C is the set of directed connection arcs connected by the priority constraint relationship between processes in the workpiece, and E is the set of disjunctive arcs between processes that can be processed on the same machine; the heterogeneous disjunctive graph adds the set of machine nodes M and the set of optional arcs F between processes and machines; the process node information describes the relevant features of the processing process, and the machine node information describes the machine scheduling information and processing information; (3) Constructing the strategy function π of the workpiece agent and the machine agent o (s;θ) and π m (s; θ); construct the value function V(s; φ) of the central controller; Heterogeneous graph neural networks are used to fit the strategy function and value function respectively; the strategy function π o (s;θ) and π m (s; θ) is fitted through the policy network, the value function V(s; φ) is fitted through the value network, and the policy function and value function are initialized π o (s;θ0),π m (s; θ0), V(s; φ0); Among them, θ is the set of internal weights ω and biases b of the neural network in the policy network, and φ is the set of internal weights ω and biases b of the neural network in the value network; (4) The workpiece agent and the machine agent output the probability of each workpiece to be processed and the optional machine as the agent action of reinforcement learning; According to the workpiece to be processed and the optional machine set for each process, a task set is established for each executable machine, and the global state is input into the heterogeneous graph neural network of the workpiece agent and the machine agent. The SoftMax activation function is used to calculate the probability distribution of the selected process task and the probability distribution of the optional machine. Then, when selecting the workpiece and machine according to the probability P, random selection or greedy method is adopted; Among them, random selection allows the agent to have more possibilities when exploring the environment, and can better learn state characteristics and optimal strategies; the greedy strategy is to give priority to the machine combination of the process and task with the highest probability at each decision. It aims to use the optimal strategy currently learned by the agent to make the best choice, so as to reduce the uncertainty brought by exploration and quickly move towards the optimal solution; (5) Design a reward function R(t) to evaluate the entire scheduling decision and use the MADRL algorithm to update the weight parameters θ of the workpiece agent and machine agent strategy functions o and θ m The weight parameter φ of the central controller’s value function is used to update the decision-making schedule; In the design of reward function, the main consideration is the minimization of maximum completion time, which is a single performance indicator. Due to different scheduling problems, the minimization of the maximum completion time obtained by scheduling is different. In order to uniformly evaluate the scheduling behavior, the negative of the maximum completion time is used as a reward; the definition of the minimization of the maximum completion time is as follows: (6) Perform state transfer P(s′s,a) and store the state s, action a, reward r, and next state s′ into the memory pool; Execute action a from state s, and then transfer the state, update the workshop environment, extract state s′, and store state s, action a, reward r, and next state s′ into the memory pool in sequence; (7) Update the policy network and value network parameters; When the workpiece agent and machine agent have collected enough historical data, they start to use this data to update the parameters of the policy network and value network. During each update, the policy network is optimized based on the latest collected data using the clipping importance sampling method. Then, the value network V(s) is used to calculate the advantage function of each state-action. To evaluate the quality of the action relative to the baseline; when the policy network is updated, the parameters are updated by optimizing the objective function. The optimization objectives of the workpiece agent and machine agent policy network are: Among them, σ is the hyperparameter of the entropy coefficient, B represents the batch size, is the number of agents, ∈ is the pruning hyperparameter, is the entropy of the strategy, CLIP() represents the clipping function; advantage function Calculated using the GAE method; weighted by importance sampling Control the update range of new and old strategies; At the same time, the optimization objectives of the central controller are: At each update, the agent randomly extracts batch-size pieces of historical data from the collected historical data and performs multiple gradient descent updates until convergence.

2. A flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In the step (1), the state S is obtained from the environment, the state is input into the workpiece agent and the machine agent, the action a currently executed by the workpiece agent and the machine agent is output, the global state S is changed, and the reward R is obtained. The process is repeated until all machines complete the processing tasks, and an environment for flexible job shop scheduling is established based on the heterogeneous disjunctive graph.

3. A flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In the step (1), the heterogeneous disjunctive graph model is represented by a set of five-tuples H = (O, M, C, E, F); Where O is the set of process nodes, C is a set of directed connection arcs connected by priority constraint relationships between processes in a workpiece. E is the set of disjunctive arcs between processes that can be processed on the same machine; The heterogeneous disjunctive graph adds a set of machine nodes M and a set of optional arcs F between processes and machines.

4. The flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In the step (2), the process node information is used to describe the workpiece-related processing information in the scheduling problem. In order to adapt to problems of different sizes, the mean is normalized.

5. The flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In the step (2), the machine node information is used to describe the scheduled information and available information of the machine in the scheduling problem. In order to adapt to problems of different sizes, the mean is normalized.

6. A flexible job shop scheduling method based on actor-critic multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In step (3), for a graph G = (V, E), the graph neural network is used to calculate the embedding of each node v (v∈V), which is defined as follows: After the Lth update iteration, the HGNN layer can obtain the p-dimensional embedding vectors of the process nodes and machine nodes, perform graph pooling through sparse matrix multiplication, calculate the average value of the node embedding to obtain two fixed-dimensional embedding vectors, and then combine them into a heterogeneous disjunctive graph H t The graph embedding representation vector h H , the definition is as follows:

Images