Double-resource flexible job shop dynamic scheduling method based on graph neural network

Abstract

The embodiment of the invention discloses a double-resource flexible job shop dynamic scheduling method based on a graph neural network, and relates to the technical field of double-resource constraint shop production scheduling. The method specifically comprises the following steps: establishing a flexible job shop dynamic scheduling problem integer programming model considering man-machine double resource constraints of fatigue degree and proficiency degree of workers by taking minimization of maximum completion time and total cost as optimization objectives; constructing a heterogeneous disjunction graph at the current scheduling moment according to the characteristics of double resource constraints; using a novel heterogeneous graph neural network to obtain a global state embedding vector fusing original features; based on the global state embedding vector, constructing a reinforcement learning environment model; a multi-PPO algorithm is used to train the environment model, and an optimization scheduling model is generated; and outputting a scheduling scheme by using the optimal scheduling model. According to the scheduling method provided by the invention, the end-to-end rapid solution of the dual-resource constraint scheduling problem is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of dual-resource constraint workshop production scheduling, and in particular to a dual-resource flexible job shop dynamic scheduling method based on graph neural network. Background Art

[0002] The Flexible Job-shop Scheduling Problem (FJSP) plays a vital role in modern manufacturing. In the classic FJSP, operations are usually ranked with the goal of minimizing certain indicators, and a processing machine is selected for each operation from a set of candidate equipment. In recent years, with the development of high-end manufacturing industries such as equipment manufacturing, automobile manufacturing, and aerospace, human factors and labor scheduling in manufacturing systems have received extensive attention. The Dual-Resource Constrained Flexible Job-shop Scheduling Problem (DRC-FJSP) considers the flexibility and constraints of both machines and workers, narrowing the gap between scheduling models and actual production. However, in actual production, uncertain events such as machine failures, job insertions, and random arrival of workpieces often occur, resulting in deviations between planning and execution, and the original plan is no longer valid. Therefore, based on DRC-FJSP, a Dual-Resource Constrained Dynamic Flexible Job-shop Scheduling Problem (DRC-DFJSP) is proposed.

[0003] In order to meet production needs, companies usually cross-train workers to make them multi-skilled workers with different skill combinations and proficiency. When assigning tasks, taking into full account the differences in workers' skill combinations and proficiency and allocating personnel reasonably can effectively reduce production costs, improve production efficiency and product quality. In this mode, worker fatigue is difficult to avoid, and long-term fatigue accumulation will damage workers' health and reduce their work efficiency, worsening the performance of the production system. Therefore, how to incorporate the impact of worker proficiency and fatigue on process processing time into DRC-DFJSP, and complete the equipment allocation and worker assignment of each process when dynamic scheduling problems occur, has become an urgent problem to be solved in actual production.

[0004] At present, there are two main methods for solving dynamic scheduling: predictive reactive scheduling (rescheduling) and fully reactive scheduling (real-time scheduling). Predictive reactive scheduling decomposes dynamic problems into a series of static sub-problems, and then uses meta-heuristic algorithms to generate scheduling solutions. Although predictive reactive scheduling can generate high-quality scheduling solutions, it is difficult to meet real-time requirements. The fully reactive scheduling method does not generate a complete scheduling solution, but makes decisions immediately based on real-time information when dynamic events occur. Priority scheduling rules (PDRs) are usually used, but efficient PDRs rely on manual experience and require a lot of professional knowledge.

[0005] In short, the way of completing dynamic scheduling by means of human management is completely unable to meet the current production needs. It is necessary to develop corresponding dynamic scheduling solutions based on virtual workshop related technologies to improve the end-to-end rapid processing of dual-resource constraint scheduling problems. Summary of the invention

[0006] An embodiment of the present invention provides a dual-resource flexible job shop dynamic scheduling method based on graph neural network, which can improve the end-to-end processing speed in the dual-resource constraint scheduling problem.

[0007] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a method comprising:

[0009] A dual-resource flexible job shop dynamic scheduling method based on graph neural network is applied to the virtual workshop in the dynamic scheduling system, and the dynamic scheduling system also includes a physical workshop and a virtual-real interaction module. The dynamic scheduling system uses the virtual-real interaction module to control the real, two-way interaction, and closed-loop iteration. The virtual workshop completes resource scheduling based on the real-time production information of the physical workshop, and the physical workshop performs production activities according to the scheduling instructions issued by the virtual workshop. The method includes:

[0010] Step 1: According to the worker fatigue and proficiency, a dual-resource flexible job shop dynamic scheduling model is established and trained. In some scenarios, this model can also be called an integer programming model for the dynamic scheduling problem of a dual-resource constrained flexible job shop.

[0011] Step 2, using the real-time data of production factors, such as workpieces, machines, and workers, to construct a dynamic flexible job shop under dual resource constraints (DRC-DFJSP) heterogeneous disjunctive graph to characterize the actual production status of the workshop;

[0012] Step 3, construct a novel heterogeneous graph neural network (NHGNN) and extract features from the heterogeneous disjunctive graph to obtain a global state embedding vector;

[0013] Step 4, inputting the global state embedding vector into the dual-resource flexible job shop dynamic scheduling model, and recording the output results, and the multi-action strategy network outputs the scheduling actions of process selection, machine and worker allocation until a complete scheduling plan is generated;

[0014] When a dynamic event occurs, repeat steps 2 to 4 until the production task is completed. For example, the dynamic event includes: new workpieces arrive randomly, where the arrival time of the new workpieces approximately follows the Poisson distribution; the new workpieces that arrive are mixed with the unprocessed processes for dynamic scheduling, while the processing process cannot be terminated, and the processed process is no longer scheduled.

[0015] Specifically, in the dual-resource flexible job shop dynamic scheduling model, the impact of worker fatigue and proficiency on process processing time is expressed as: Among them, p ijkr For workers r On machine M k Upper processing step O ij The actual processing time, p ijk O ij In M k The preset processing time on is the parameter of the influence of fatigue on the processing time, FL r (t) is W at time t r Fatigue parameter, PL ijr W r Execute O ij The proficiency parameter of the i-th workpiece J i The index of j is workpiece J i The jth process O ij The index of M, k is the kth machine M k The index of r is the rth worker W r The index of .

[0016] The types of the real-time data of production factors include: real-time data of workpieces, real-time data of machines, real-time data of workers, real-time data associated with workpieces and machines, and real-time data associated with workpieces and workers; wherein the real-time data of workpieces include: the scheduling status of each process, the estimated processing time and the actual processing time of each process, the number of candidate machines for each process, the number of candidate workers for each process, the estimated start processing time and the actual start processing time of each process, the number of unscheduled processes for workpieces, the estimated completion time and the actual completion time of workpieces; the real-time data of machines include: the processing cost per unit time of the machine, the number of processes that can be processed by the machine, the available time of the machine, and the utilization rate of the machine; the real-time data of workers include: the degree of fatigue of the workers, the number of processes that can be processed by the workers, the available time of the workers, and the utilization rate of the workers; the real-time data associated with workpieces and machines include: the preset processing time of each process on the machine; the real-time data associated with workpieces and workers include: the proficiency of the workers in processing each process, and the processing cost per unit time of the workers in processing each process.

[0017] The dual-resource constrained dynamic flexible job-shop scheduling problem (also called dual-resource constrained dynamic flexible job-shop scheduling problem) is constructed to characterize the actual production status of the workshop. The method comprises the following steps: constructing a process node, wherein the original vector of the process node is the workpiece real-time data; constructing a machine node, wherein the original vector of the machine node is the machine real-time data; constructing a worker node, wherein the original vector of the worker node is the worker real-time data; adding conjunction arcs between process nodes to constrain the processing sequence between processes of the same workpiece; adding OM arcs between process nodes and their compatible machine nodes, wherein the original vector of the OM arc is the workpiece-machine associated real-time data; adding OW arcs between process nodes and their compatible worker nodes, wherein the original vector of the OW arc is the workpiece-worker associated real-time data; and then continuously updating the heterogeneous disjunctive graph, comprising: when a process is scheduled, only retaining the arcs between this process and the machines and workers to which it is assigned, and then removing the other OM arcs and OW arcs of this process; repeating the process of updating the heterogeneous disjunctive graph until all processes are scheduled, and generating a scheduling scheme graph.

[0018] In this embodiment, the step 3 includes: constructing an embedding layer, in which a machine node, a worker node and an operation node are sequentially embedded;

[0019] The NHGNN model is constructed by stacking N embedding layers. The final embedding vectors of the three types of nodes are obtained through the NHGNN model, where N is a positive integer. The average values ​​of the three types of embedding vectors are then calculated and concatenated to obtain the global state embedding vector.

[0020] In the process of machine node embedding, the worker nodes of the second-order neighborhood are embedded into the operation nodes of the first-order neighborhood, and the operation nodes of the first-order neighborhood are embedded into the machine nodes as the target nodes; in the process of worker node embedding, the machine nodes of the second-order neighborhood are embedded into the operation nodes of the first-order neighborhood, and the operation nodes of the first-order neighborhood are embedded into the artificial nodes as the target nodes; in the process of operation node embedding, the vectors of each node are processed and cascaded through six MLPs, wherein the vectors of each node include: the original vector of an operation node, the original vectors of the previous and next operation nodes of this operation node, the embedded vectors of the machine nodes adjacent to this operation node, and the embedded vectors of the worker nodes adjacent to this operation node. For example:

[0021] Machine node embedding:

[0022] (1) Embed the worker nodes (second-order neighborhood) into the operation nodes (first-order neighborhood):

[0023] Cascade the original vectors of worker nodes and OW arcs and map them to d dimensions; map the original vectors of operation nodes to d dimensions; calculate the attention coefficients of worker nodes to operation nodes and normalize them; aggregate the features of worker nodes to obtain the embedding vectors of operation nodes;

[0024] (2) Embed the operation node (first-order neighborhood) into the machine node (target node):

[0025] Concatenate the operation node embedding vector and the OM arc original vector and map them to d dimensions; map the machine node original vector to d dimensions; calculate the attention coefficient of the operation node to the machine node and normalize it; aggregate the operation node features to complete the machine node embedding;

[0026] Worker node embedding:

[0027] (1) Embed the machine node (second-order neighborhood) into the operation node (first-order neighborhood):

[0028] Cascade the machine node and OM arc original vectors and map them to d dimensions; map the operation node original vector to d dimensions; calculate the attention coefficient of the machine node to the operation node and normalize it; aggregate the machine node features to obtain the operation node embedding vector;

[0029] (2) Embed the operation node (first-order neighborhood) into the worker node (target node):

[0030] Concatenate the operation node embedding vector and the OW arc original vector and map them to d dimensions; map the worker node original vector to d dimensions; calculate the attention coefficient of the operation node to the worker node and normalize it; aggregate the operation node features and complete the worker node embedding;

[0031] Operation node embedding: Six MLPs are used to process and concatenate the original vectors of the previous and next operation nodes and the embedding vectors of the adjacent machine and worker nodes to obtain their embedding vectors.

[0032] The NHGNN model is constructed by stacking N embedding layers to obtain the final embedding vectors of the three types of nodes;

[0033] The average of the three types of embedding vectors is calculated and concatenated to obtain the global state embedding vector.

[0034] Furthermore, it also includes: training the dual-resource flexible job shop dynamic scheduling model through the MPPO (multi-action proximal policy optimization) algorithm; wherein, based on the features of the embedding vectors corresponding to feasible process actions and the global state embedding vectors, constructing a process Actor network; based on the features of the embedding vectors corresponding to feasible machine actions and the global state embedding vectors, constructing a machine Actor network; based on the features of the embedding vectors corresponding to feasible worker actions and the global state embedding vectors, constructing a worker Actor network; based on the features of the global state embedding vectors, constructing a Critic network.

[0035] In the MPPO algorithm, it is necessary to first construct an environment model based on the Markov Decision Process (MDP). The specific steps are as follows:

[0036] State: At time step t, state s t It contains all static and dynamic features of DRC-DFJSP and uses the proposed heterogeneous disjunctive graph to represent the state of DRC-DFJSP;

[0037] Action: At time step t, action a t By process action a o ∈A o 、Machine Action m ∈A m and worker action a w ∈A w Composition, A o is a set of optional processes, A m It is an action o A set of compatible machines, A w It is an action o The set of compatible workers;

[0038] Transfer: At time step t, the agent is based on state s t Take action a t , interact with the environment to move to a new state s t+1, and based on action a t Update heterogeneous disjunction graph;

[0039] Reward: Combine the two optimization objectives into the maximum completion time C max and the total cost L c The immediate rewards are defined as r1 and r2 respectively, and the weighted reward is r t =ω1r1+ω1r2, satisfying ω1+ω2=1, ω1, ω2 are weighting coefficients and ω1, ω2≥0.1;

[0040] Strategy: Multi-action strategy π θ (a o , a m , a w |s t ) By process strategy Machine Strategy and worker strategies composition.

[0041] The policy network is pre-trained offline using the multi-action proximal policy optimization (MPPO) algorithm. The specific steps are as follows:

[0042] (1) Training the Actor Network

[0043] Each Actor network is trained to maximize its own loss function Among them, c p =1 and c e =0.01 are the truncation hyperparameter and entropy coefficient hyperparameter respectively, o is the relevant parameter subscript of the process strategy; m is the relevant parameter subscript of the machine strategy; w is the relevant parameter subscript of the worker strategy. is the pruning proxy target, are entropy targets, which are as follows:

[0044]

[0045] in, It is a target strategy and behavioral strategies for sampling The probability ratio of , ∈∈(0, 1) is the relevant parameter subscripts (o, m, w) of the tailoring parameter h taking the process strategy, machine strategy, and worker strategy, indicating the actions corresponding to the three strategies respectively.

[0046] (2) Update the Critic Network

[0047] In this paper, the Critic network and the Actor network share the same NHGNN model. φ (st ) Use a d φ dimensional hidden layer and tanh activation MLP φ To accept the global state embedding vector g extracted by the NHGNN model t And output a scalar to estimate the cumulative return at:

[0048] v φ (s t )=MLP φ (g t )

[0049] The Critic network minimizes the mean squared error (MSE) target L MSE (φ) is updated:

[0050] L MSE (φ)=E t [MSE(r t , v φ (s t ))]

[0051] Initialize the MPPO algorithm parameters, the behavior Actor interacts with the environment to schedule each instance, and collects the experience quadruple (s t , a t , r t ,s t+1 ) The quadruple is stored in the experience replay pool until all tasks are scheduled, and Et is the expected future cumulative reward. Φ is the Critic network. Then, calculate the probability ratio Target Actor Error and Critic error L MSE (φ), and update the network by gradient ascent. Finally, the target Actor parameters Copy to Behavior Actor Parameters After completing one iteration, repeat the iteration until the convergence criteria are met, the training is completed, and the optimized scheduling model is obtained;

[0052] The optimization scheduling model based on the MPPO algorithm is used for workshop production optimization scheduling. The optimization scheduling model adopts an offline training mode. In actual production, the global state embedding vector is input into the optimization scheduling model based on the MPPO algorithm. The multi-action strategy network outputs the scheduling actions of process selection, machine and worker allocation until a complete scheduling plan is generated.

[0053] In a second aspect, the method provided by an embodiment of the present invention can be executed by a dual-resource flexible job shop dynamic scheduling system based on a graph neural network, the system comprising:

[0054] Virtual-reality interaction module, used for two-way interaction between physical workshop and virtual workshop;

[0055] The physical workshop is used to execute the scheduling plan and collect real-time data to support dynamic scheduling of the virtual workshop. The physical workshop includes: a data collection module for collecting real-time data of workshop production factors; a manufacturing execution module for executing the scheduling plan to organize actual production;

[0056] The virtual workshop is used to respond to dynamic events and formulate scheduling plans to guide the actual production of the physical workshop. The virtual workshop includes: a state representation module, which is used to construct a heterogeneous disjunctive graph of DRC-DFJSP; a feature extraction module, which is used to construct NHGNN to extract global state embedding vectors from the heterogeneous disjunctive graph; an environment model module, which is used to construct a reinforcement learning environment model of DRC-DFJSP; a scheduling model module, which is used to use the MPPO algorithm to train the scheduling model and generate an optimized scheduling model; an optimized scheduling module, which is used to generate a scheduling plan using the optimized scheduling model.

[0057] The embodiment of the present invention provides a dual-resource flexible job shop dynamic scheduling method based on graph neural network, constructs a DRC-DFJSP integer programming model that takes into account the proficiency and fatigue of workers, and proposes a heterogeneous disjunctive graph to represent the scheduling state in view of the characteristics of the dual-resource constraint problem model. A new type of heterogeneous graph neural network is designed to efficiently extract state features, and a multi-action proximal policy optimization (MPPO) is proposed, so as to solve DRC-DFJSP of different scales end-to-end, overcoming the shortcomings of the existing problem model of the dual-resource manufacturing system being different from the actual production and the lack of practicality of the scheduling scheme. Improve the end-to-end processing speed in the dual-resource constraint scheduling problem BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of a dual-resource flexible job shop scheduling problem provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a digital twin-enhanced dual-resource flexible job shop dynamic scheduling system provided in an embodiment of the present invention;

[0061] Figure 3A schematic diagram of a heterogeneous disjunction graph of DRC-DFJSP provided in an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of a global state embedding process based on NHGNN provided in an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of a multi-Actor-Critic network structure provided in an embodiment of the present invention;

[0064] FIG6 is a sample Gantt chart of new workpiece arrival scheduling with the machine as the vertical axis provided by an embodiment of the present invention; wherein: Figure 6a This is the result graph obtained based on the unimproved PPO. Figure 6b This is the result diagram obtained based on MPPO.

[0065] Figure 7 is a sample Gantt chart of new workpiece arrival scheduling with workers as the vertical axis. Figure 7a This is the result graph obtained based on the unimproved PPO. Figure 7b This is the result diagram obtained based on MPPO. DETAILED DESCRIPTION

[0066] In recent years, reinforcement learning has provided a new idea for solving scheduling problems with dynamic events. The reinforcement learning model learns by interacting with the environment, does not require pre-collected data, and can estimate the value of the future state, so it can effectively deal with various dynamic events in the actual production workshop. Although more and more scholars have begun to use reinforcement learning methods to solve workshop scheduling problems, current research mainly focuses on FJSP, and its application in the field of DRC-DFJSP is still rare. Reinforcement learning provides a new method for workshop scheduling problems in dynamic environments. Reinforcement learning optimizes scheduling strategies by interacting with the environment and learning in real time, thereby effectively dealing with various dynamic events in actual production. However, traditional scheduling methods based on reinforcement learning cannot effectively handle huge state spaces, and the trained strategies are often only applicable to instances of a single scale. Graph neural networks can process graphs of different sizes and can effectively extract deep feature information, thereby overcoming the limitations of matrix representation. Therefore, the present invention uses a new heterogeneous graph neural network to extract features from a dual-resource manufacturing system, and uses a reinforcement learning method to train a scheduling model, which can cope with scheduling problems of different scales and has good generalization.

[0067] Embodiment 1

[0068] The present invention provides a dual-resource flexible job shop dynamic scheduling method based on a graph neural network, the method comprising:

[0069] Step 1: An integer programming model for the dynamic scheduling problem of a flexible job shop with dual resource constraints considering worker fatigue and proficiency.

[0070] In step 1, the principle diagram of the dual resource constraint flexible job shop dynamic scheduling problem is as follows: Figure 1 There are n workpieces J, which are processed by l workers W on m machines M. Workpiece J i ∈J includes n i operations that follow the precedence relation, n i represents the ith operation of the nth workpiece, forming J i The operation set O i . Process O ij The processing is combined with its compatible machines A machine M in k and a worker W from the set of compatible workers r To coordinate the execution, the problem model is decomposed into three subtasks: (1) arranging the processing order of each process; (2) assigning machines to the process; (3) assigning workers to the process. The problem constraints are as follows:

[0071] (1) At time 0, all machines, workers, and work tasks are available, and the worker's fatigue value is 0;

[0072] (2) Each machine and worker can only process one process at a time;

[0073] (3) The processing time of each process on different machines and by different workers is known;

[0074] (4) There are priority constraints between the various processes of the same task, and only one process can be processed at a time;

[0075] (5) The process cannot be interrupted during the processing;

[0076] (6) Ignore machine failures and non-machine processing operations;

[0077] (7) Auxiliary time (such as workpiece transportation time) should be included in the processing time;

[0078] (8) All tasks have the same priority.

[0079] In step 1, the consideration of worker fatigue and proficiency is specifically as follows:

[0080] According to the process, the workers' proficiency is divided into elementary I, intermediate II and special III. Workers have heterogeneity in skill combination and proficiency. For the same process, the proficiency of the same worker is the same, but the proficiency of different workers may be different; for different processes, the proficiency of the same worker may be different. r Processing process Oij The proficiency of the ijr ={p I , p II , p III}, the corresponding unit time processing cost is given by parameter WC ijr ={[a I , b I ],[a II , b II ],[a III , b III ]} description. In addition, fatigue of multi-skilled workers is difficult to avoid. Therefore, considering the influence of worker fatigue and proficiency, the actual processing time of the process is as follows:

[0081]

[0082] Among them, p ijkr For workers r On machine M k Upper processing step O ij The actual processing time, p ijk O ij In M k The preset processing time on is the parameter of the influence of fatigue on the processing time, FL r (t) is W at time t r Fatigue level, PL ijr W r Processing ij proficiency.

[0083] In step 1, the model optimization scheduling objective is:

[0084] Traditional dynamic scheduling algorithms often optimize a single optimization objective. The present invention optimizes both the maximum completion time and the total cost at the same time to achieve multi-objective optimization. The objective function is defined as follows:

[0085] (1) Minimizing the maximum completion time: To improve the production efficiency of the workshop, it is necessary to shorten the production cycle of the order. The maximum completion time of the workpiece is minimized as the first objective function f1, as shown below:

[0086]

[0087] Among them, C i represents the completion time of workpiece i;

[0088] (2) Minimization of total cost: In order to improve the market competitiveness of enterprises, it is necessary to control the total production cost, that is, the total cost of machines and workers, and minimize it as the second objective function f2, as shown below:

[0089]

[0090] Among them, MC k Indicates machine M k The unit time processing cost, WC ijr Represents worker W r Processing process O ij The unit time cost is taken as the proficiency PL ijr The corresponding interval, x ijkr is a binary variable, when process O ij On machine M k By worker W r When processing, it is 1, otherwise it is 0, p ijkr For process O ij The actual processing time.

[0091] Step 2: Based on the real-time data of production factors such as workpieces, machines, and workers, a DRC-DFJSP heterogeneous disjunctive graph is constructed to represent the actual production status of the workshop.

[0092] In step 2, the real-time data of production factors are specifically:

[0093] (1) Real-time data of workpieces: scheduling status of each process, (estimated / actual) processing time of each process, number of candidate machines for each process, number of candidate workers for each process, (estimated / actual) start processing time of each process, number of unscheduled processes for workpieces, and (estimated / actual) completion time of workpieces;

[0094] (2) Real-time machine data: machine processing cost per unit time, number of processing steps that can be processed by the machine, machine available time, and machine utilization rate;

[0095] (3) Real-time data of workers: worker fatigue level, number of processing steps available for workers, worker available time, and worker utilization rate;

[0096] (4) Workpiece-machine association real-time data: the preset processing time of each process on the machine;

[0097] (5) Real-time data on workpiece-worker association: workers’ proficiency in processing each process and workers’ processing cost per unit time for processing each process.

[0098] In step 2, the DRC-DFJSP heterogeneous disjunction graph is specifically:

[0099] DRC-DFJSP not only needs to select the machine for the processing process, but also needs to select the worker who operates the machine. In order to more intuitively represent the status of real-time production, the present invention proposes a heterogeneous disjunctive graph H of DRC-DFJSP. t =(O t, M, W, C, E t ), add machine nodes and worker nodes and expand edge types. In the dynamic scheduling process, each state s t can be expressed as H t , where O t With E t Dynamic changes, such as Figure 3 The figure shows a heterogeneous disjunction graph of a 3×3×2 DRC-DFJSP instance. The specific construction method is:

[0100] (1) Constructing the initial heterogeneous disjunction graph:

[0101] The initial heterogeneous disjunctive graph H0 at time 0 is as follows Figure 3 (a) shows: constructing process nodes, the original vector is the real-time data of the workpiece, is the process node set, n is the total number of workpieces J; i is the i-th workpiece J i Index of; n i Workpiece J i The total number of processes O involved; j is workpiece J i The jth process O ij ; construct machine nodes, the original vector is the real-time data of the machine, is a set of machine nodes, m is the total number of machines M; k is the rth machine M k ; build worker nodes, the original vector is the worker real-time data, is the set of worker nodes, l is the total number of workers W; r is the rth worker W r Add conjunction arcs between processes to constrain the processing order between processes of the same workpiece, C = { <O ij , O ij+1 >|1≤i≤n, 1≤j≤n i -1) is the conjunctive arc set; add OM arcs between the process and its compatible machines, the original vector is the workpiece-machine association real-time data, E O-M ∈E is the undirected arc set between the process and the available machines; OW arcs are added between the process and its compatible workers, and the original vector is the real-time data of the workpiece-worker association, E O-W ∈E is the undirected arc set between the process and the available workers. The original vectors of various nodes and edges are shown in Table 1:

[0102] Table 1 Original vector

[0103]

[0104] (2) Update the heterogeneous disjunction graph:

[0105] If process O at time t 11 With O 21Scheduled, updated heterogeneous disjunction graph H t As shown in 0(b), specifically: when a process is scheduled, only the process node O is retained. ij and the machine node M to which it is assigned k The arc between Worker Node W r The arc between Remove other OM arcs and OW arcs in this process;

[0106] If a new job arrives at time t+1, H of the updated heterogeneous disjunction graph t+1 As shown in 0(c), specifically: when one or more new jobs arrive, the new jobs All the steps included in O ij Add to H t And each process node O ij With its machine candidate set M ij All machine nodes M k Connecting arcs With its worker candidate set W ij All worker nodes W in r Connecting arcs

[0107] Repeat this process until all processes are scheduled, generating the scheduling diagram H shown in 0(d) n .

[0108] Step 3: construct a novel heterogeneous graph neural network (NHGNN) to extract features from the heterogeneous disjunctive graph constructed in step 2 to obtain a global state embedding vector.

[0109] The end-to-end reinforcement learning method needs to extract effective information from the underlying original features to achieve efficient decision-making of the agent. t Extract the original features and obtain the global state embedding vector, which specifically includes the following steps:

[0110] Construct the embedding layer: embed the machine node, worker node and operation node in three stages. Take a 2×3×2 instance as an example. The embedding process is as follows: Figure 4 As shown:

[0111] (1) Stage 1: Machine node embedding, such as Figure 4 (a) as shown:

[0112] Step 1: Worker Node W r Through the edge With process node O ijconnected, through the cascade operation ·||·W r The original vector zr is expanded to And through the linear transformation Map it to d dimensions and transform O ij The original vector x ij Through linear transformation Mapped to d dimensions. r , for O ij The attention coefficient can be calculated as:

[0113]

[0114] Where a∈R 2d , N t (O ij _W) is O ij The adjacent W r A collection of ij The self-attention coefficient can be calculated as:

[0115] e ij =LeakyReLU(a T [W O x ij ||W O x ij ])

[0116] Use the softmax function to normalize the attention coefficient (α ijr , α ij ), and use the sigmoid function σ to process the node W r and O ij The aggregation process is as follows:

[0117]

[0118] Get the process node O that integrates the worker characteristics ijw The embedding vector x ijw ∈R d ;

[0119] Step 2: Process node O integrating worker characteristics ijw Through the edge With machine node M k Connected, O ijw Point M k The attention coefficient can be calculated as:

[0120]

[0121] Among them, the linear matrix N t (M k _O) is Mk The adjacent operation node O ijw Collection, u ijk Indicates the OM arc The original vector of M. k The self-attention coefficient can be calculated as:

[0122] e k =LeakyReLU(a T [W M y k ||W M y k ])

[0123] Normalized to get the attention coefficient (a ijk , α k ), aggregation node O ijw and node M k The process is as follows:

[0124]

[0125] Get machine node M k The embedding vector y′ k ∈R d ;

[0126] (2) Stage 2 Worker Node Embedding,Similar to machine nodes, worker nodes also integrate their second-order and first-order neighborhood feature information through two steps, such as Figure 4 (b) as shown:

[0127] Step 1: Calculate the number of machine nodes M k For process node O ij The attention coefficient f ijk and O ij The self-attention coefficient f ij , normalized to obtain (β ijk , β ij ), aggregation node M k and node O ij The features of the process node O are obtained by integrating the machine node features. ijm The embedding vector x ijm ∈R d , the process is as follows:

[0128]

[0129] Among them, the linear matrix N t (O ij _M) is O ij The adjacent M k A collection of ijThe self-attention coefficient can be calculated as:

[0130] f ij =LeakyReLU(a T [W O x ij ||W O x ij ])

[0131] Normalized processing yields (β ijk , β ij ). Aggregation node M k and node O ij Features:

[0132]

[0133] Among them, R d represents the dimension of the embedded vector after embedding, β ijk It is the fusion machine node M k For process node O ij The attention coefficient f ijk Normalized result; β ij is process node O ij The self-attention coefficient f ij The normalized result.

[0134] Step 2: Calculate the process node O that integrates machine features ijm For worker node W r The attention coefficient f ijr and W r The self-attention coefficient f r , normalized to obtain (β ijr , β r ), aggregation node O ijm and node W r , get the worker node W r The embedding vector z′ r ∈R d , the process is as follows:

[0135]

[0136] Among them, the linear matrix N t (W r _O) is W r Adjacent process node O ijm Collection. r The self-attention coefficient can be calculated as:

[0137] f r =LeakyReLU(a T [WW z r ||W W z r ])

[0138] Normalized processing yields (β ijr , β r ). Aggregation Node O ijm and node W r Features:

[0139]

[0140] Among them, β ijr It is the process node O that integrates machine features ijm For worker node W r The attention coefficient f ijr Normalized result; β r is the worker node W r The self-attention coefficient f r The normalized result. ijr Indicates OW arc The original vector.

[0141] (3) The third stage is process node embedding. Based on the embedding of machine nodes and worker nodes, process node embedding is performed, such as Figure 4 (c) as shown:

[0142] Define 6 MLPs For the eigenvector x ij-1 , x ij , x ij+1 , y′ k and z′ r Each MLP has a d-dimensional output, two d ω The hidden layer of the dimensional network is activated by the exponential linear unit (ELU). Then, the processing results are concatenated and ELU activation and MLP processing are performed. Finally, the node O is obtained. ij The embedding vector x′ ij ∈R d The embedding process is as follows:

[0143]

[0144] Among them, x ij-1 , x ij , x ij+1 Node O ij-1 , O ij and O ij+1 The original vector. It is node O ij All adjacent machine nodes M′ kThe embedding vector y′ k The sum average. It is node O ij All neighboring worker nodes W′ r The embedding vector z′ r The average of the sum.

[0145] Constructing the NHGNN model: Constructing the NHGNN model by stacking N embedding layers with the same structure but independently trainable parameters to obtain the final embedding vector and Pool the three types of embedding vectors and concatenate them to obtain the heterogeneous disjunction graph H t The global state embedding vector h t ∈R 3d , the process is as follows:

[0146]

[0147] Among them, |O|, |M| and |W| are H t The number of process nodes, machine nodes, and worker nodes in the process.

[0148] In step 4, the global state embedding vector obtained in step 3 is input into the scheduling model trained by the multi-action proximal policy optimization (MPPO) algorithm. The multi-action policy network outputs scheduling actions for process selection, machine and worker allocation until a complete scheduling plan is generated.

[0149] In step 4, the environment model of the multi-action proximal strategy optimization algorithm is specifically:

[0150] Construct an MDP-based environment model, including:

[0151] State: At time step t, state s t All static and dynamic features of DRC-DFJSP are included, and the proposed heterogeneous disjunctive graph is used to represent the state of DRC-DFJSP.

[0152] Action: At time step t, action a t (a o , a m , a w ) By process action a o ∈A o 、Machine Action m ∈A m and worker action a w ∈A w Composition, A o is a set of optional processes, A m It is action ao A set of compatible machines, A w It is an action o The set of compatible workers;

[0153] Transfer: At time step t, the agent is based on state s t Take action a t , interact with the environment to move to a new state s t+1 , and based on action a t Update heterogeneous disjunctive graph H t+1 ;

[0154] Reward: r1 is the instant reward for the maximum completion time, and the average utilization rate of the machine is calculated n is the total number of workpieces J; m is the total number of machines M; n i Is workpiece J i The total number of processes O involved; 1 is the total number of workers W; x ijkr :Process O ij Whether it is by worker W r On machine M k If it is processed, it is 1, otherwise it is 0; p ijkr :Process O ij It was by worker W r On machine M k The processing time for processing on the surface. Among them, CT(t) is the maximum actual completion time of time step t, and r1 takes the following values:

[0155]

[0156] r2 is the instant reward of the total cost, which is calculated by the total processing time of workers and machines With total cost L c The ratio between (t) The values ​​of r2 are as follows:

[0157]

[0158] The weighted reward is r t =ω1r1+ω1r2, satisfying ω1+ω2=1 and ω1, ω2≥0.1. i After training reaches a certain number of times, save the ω i Then, select the ω with the smallest Euclidean distance i+1 Continue training and repeat until all agent parameters corresponding to all weight vectors are saved;

[0159] Strategy: Multi-action strategy π θ (a o , a m , a w |st ) By process strategy Machine Strategy and worker strategies composition;

[0160] In step 4, the scheduling model is specifically:

[0161] The scheduling model consists of three Actor networks and one Critic network. The overall network structure is shown in Figure 6:

[0162] In state st, each feasible process action a o ∈A o The corresponding embedding vector and the global state embedding feature h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a o The probability distribution of choice P(a o ,s t ), normalized to obtain the process

[0163] In status t And select process action a o Under the condition that each feasible machine action a m ∈A m The corresponding embedding vector and h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a m The probability distribution of choice P(a m ,s t _a o ), normalized to obtain the machine strategy

[0164] In status t And select process action a o and machine action a m Under the condition that each feasible worker action a w ∈A w The corresponding embedding vector and h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a w The probability distribution of choice P(a w ,s t _a o _a m), normalized to obtain the worker strategy

[0165] Use a function with two d φ dimensional hidden layer and tanh activation MLP φ Processing t , get the Critic network v φ .

[0166] In step 4, the multi-action proximal policy optimization (MPPO) algorithm is specifically:

[0167] The proximal policy optimization (PPO) algorithm is an advanced policy gradient method of the Actor-Critic architecture and is widely used to process discrete and continuous control tasks. However, DRC-DFJSP is a multi-action task. In view of this feature, this paper proposes a multi-action proximal policy optimization (MPPO) algorithm, which uses the MPPO algorithm to pre-train the scheduling model offline. The specific steps are as follows:

[0168] Calculate the advantage function Where γ is the discount factor, t is the current time step, and v φ (s t ) is the Critic network according to the state s t The value function value obtained, T<<|O|;

[0169] In each iteration of the training process, three Actors interact with the environment to collect experience quads (s t , a t , r t ,s t+1 ), and train the Actor network and update the Critic network through the collected samples:

[0170] (1) Training the Actor Network

[0171] Each Actor network is trained to maximize its own loss function Among them, c p =1 and c e =0.01 are the cutoff hyperparameter and entropy coefficient hyperparameter respectively. is the pruning proxy target, are entropy targets, which are as follows:

[0172]

[0173] in, It is a target strategy and behavioral strategies for sampling The probability ratio of , ∈∈(0, 1) is the clipping parameter.

[0174] (2) Update the Critic Network

[0175] The Critic network minimizes the mean squared error (MSE) target L MSE (φ) is updated:

[0176] L MSE (φ)=E t [MSE(r t , v φ (s t ))]

[0177] In step 4, the scheduling model trained by the multi-action proximal policy optimization (MPPO) algorithm is specifically:

[0178] Initialize the MPPO algorithm parameters and randomly generate B instances. The behavior actor interacts with the environment to schedule each instance and collects the experience quadruple (s t , a t , r t ,s t+1 ) The quadruple is stored in the experience replay pool until all tasks are scheduled. Then, the probability ratio is calculated Target Actor Error and Critic error L MSE (φ), and update the network E by the gradient ascent method u Finally, set the target Actor parameter Copy to Behavior Actor Parameters Complete one iteration. Training total iteration E t times, each iteration e val Validate on an independent validation set, each iteration e rep Replace a batch of B instances each time until the convergence criteria are met, the training is completed, and the optimized scheduling model is obtained;

[0179] In step 4, the multi-action strategy network outputs scheduling actions for process selection, machine and worker allocation until a complete scheduling solution is generated, specifically:

[0180] The optimization scheduling model based on the MPPO algorithm is used for workshop production optimization scheduling. In actual production, the global state embedding vector is input into the optimization scheduling model trained offline by the MPPO algorithm. The multi-action strategy network outputs the scheduling actions of process selection, machine and worker allocation until a complete scheduling plan is generated.

[0181] Step 5: When a dynamic event occurs, repeat steps 2 to 4 until the production task is completed.

[0182] In step 5, the dynamic events are specifically:

[0183] The dynamic event is the random arrival of new workpieces, and the arrival time approximately follows the Poisson distribution. The new workpieces are mixed with unprocessed processes for dynamic scheduling. The processing process cannot be terminated, and the processed process will no longer be scheduled.

[0184] Embodiment 2

[0185] A dual-resource flexible job shop dynamic scheduling system based on graph neural network, the system comprising:

[0186] The virtual-real interaction module is used for two-way interaction between the physical workshop and the virtual workshop. In this embodiment, the physical workshop transmits the real-time data of production factors to the virtual workshop through the virtual interaction module, and the virtual workshop sends the production optimization scheduling plan to the physical workshop through the virtual interaction module;

[0187] The physical workshop is used to execute the scheduling plan and collect real-time data to support dynamic scheduling of the virtual workshop, including:

[0188] The data collection module is used to collect real-time data of workshop production factors.

[0189] The real-time data of production factors collected are as follows:

[0190] Real-time data of workpieces: scheduling status of each process, (estimated / actual) processing time of each process, number of candidate machines for each process, number of candidate workers for each process, (estimated / actual) start processing time of each process, number of unscheduled processes for workpieces, (estimated / actual) completion time of workpieces;

[0191] Real-time machine data: machine processing cost per unit time, number of processing steps that can be processed by the machine, machine available time, and machine utilization rate;

[0192] Real-time data of workers: worker fatigue level, number of processing steps available for workers, worker available time, and worker utilization rate;

[0193] Workpiece-machine association real-time data: preset processing time of each process on the machine;

[0194] Real-time data on workpiece-worker association: workers’ proficiency in processing each process, and workers’ processing cost per unit time for processing each process.

[0195] The manufacturing execution module is used to execute the scheduling plan to organize production. In this embodiment, the manufacturing execution module performs production according to the scheduling plan issued by the virtual workshop.

[0196] The virtual workshop is used to respond to dynamic events and formulate scheduling plans to guide the actual production of the physical workshop, including:

[0197] The state representation module is used to construct the heterogeneous disjunctive graph of DRC-DFJSP.

[0198] DRC-DFJSP not only needs to select the machine for the processing process, but also needs to select the worker who operates the machine. In order to more intuitively represent the status of real-time production, the present invention proposes a heterogeneous disjunctive graph H of DRC-DFJSP. t =(O t , M, W, C, E t ), add machine nodes and worker nodes and expand edge types. In the dynamic scheduling process, each state s t can be expressed as H t , where O t With E t Dynamic changes such as Figure 3 The following is a 3×3×2 DRC-DFJSP example.

[0199] (1) Constructing the initial heterogeneous disjunction graph:

[0200] The initial heterogeneous disjunctive graph H0 at time 0 is as follows Figure 3 (a) shows: constructing process nodes, the original vector is the real-time data of the workpiece, is a set of process nodes; constructs machine nodes, and the original vector is the real-time data of the machine. is a set of machine nodes; constructs worker nodes, and the original vector is the real-time data of the workers. is the set of worker nodes; add conjunction arcs between processes to constrain the processing order between processes of the same workpiece, C = { <O ij , O ij+1 >||1≤i≤n,1≤j≤n i -1} is a conjunctive arc set; an OM arc is added between the process and its compatible machine, and the original vector is the workpiece-machine association real-time data, E O-M ∈E is the undirected arc set between the process and the available machines; OW arcs are added between the process and its compatible workers, and the original vector is the real-time data of the workpiece-worker association, E O-W ∈E is the undirected arc set between the process and the available workers. The original vectors of various nodes and edges are shown in Table 1:

[0201] Table 1 Original vector

[0202]

[0203] (2) Update the heterogeneous disjunction graph:

[0204] If process O at time t 11 With O 21 Scheduled, updated heterogeneous disjunction graph H t As shown in 0(b), specifically: when a process is scheduled, only the process node O is retained. ij and the machine node M to which it is assigned k The arc between Worker Node W r The arc between Remove other OM arcs and OW arcs in this process;

[0205] If a new job arrives at time t+1, H of the updated heterogeneous disjunction graph t+1 As shown in 0(c), specifically: when one or more new jobs arrive, the new jobs All the steps included in O ij Add to H t And each process node O ij With its machine candidate set M ij All machine nodes M k Connecting arcs With its worker candidate set W ij All worker nodes W in r Connecting arcs

[0206] The feature extraction module is used to construct NHGNN to extract the global state embedding vector from the heterogeneous disjunctive graph.

[0207] Construct the embedding layer: embed the machine node, worker node and operation node in three stages. Take a 2×3×2 instance as an example. The embedding process is as follows: Figure 4 As shown:

[0208] (1) Stage 1: Machine node embedding, such as Figure 4 (a) as shown:

[0209] Step 1: Worker Node W r Through the edge With process node O ij connected, through the cascade operation ·||·W r Original vector z r Expand to And through the linear transformation Map it to d dimensions and transform O ijThe original vector x ij Through linear transformation Mapped to d dimensions. r For O ij The attention coefficient can be calculated as:

[0210]

[0211] Where a∈R 2d , N t (O ij _W) is O ij W r Collection. ij The self-attention coefficient can be calculated as:

[0212] e ij =LeakyReLU(a T [W O x ij ||W O x ij ])

[0213] Use the softmax function to normalize the attention coefficient (α ijr , α ij ), and use the sigmoid function σ to process the node W r and O ij The aggregation process is as follows:

[0214]

[0215] Get the process node O that integrates the worker characteristics ijw The embedding vector x ijw ∈R d ;

[0216] Step 2: Process node O integrating worker characteristics ijw Through the edge With machine node M k Connected, O ijw Point M k The attention coefficient can be calculated as:

[0217]

[0218] Among them, the linear matrix N t (M k _O) is M k The adjacent operation node O ijw Collection. k The self-attention coefficient can be calculated as:

[0219] e k=LeakyReLU(a T [W M y k ||W M y k ])

[0220] Normalized to get the attention coefficient (α ijk , α k ), aggregation node O ijw and node M k The process is as follows:

[0221]

[0222] Get machine node M k The embedding vector y′ k ∈R d ;

[0223] (2) Stage 2 Worker Node Embedding,Similar to machine nodes, worker nodes also integrate their second-order and first-order neighborhood feature information through two steps, such as Figure 4 (b) as shown:

[0224] Step 1: Calculate the number of machine nodes M k For process node O ij The attention coefficient f ijk and O ij The self-attention coefficient f ij , normalized to obtain (β ijk , β ij ), aggregation node M k Node and Point O ij The features of the process node O are obtained by integrating the machine node features. ijm The embedding vector x ijm ∈R d , the process is as follows:

[0225]

[0226] Among them, the linear matrix N t (O ij _M) is O ij The adjacent M k A collection of;

[0227] Step 2: Calculate the process node O that integrates machine features ijm For worker node W r The attention coefficient f ijr and W r The self-attention coefficient f r , normalized to obtain (β ijr , βr ), aggregation node O ijm and node W r , get the worker node W r The embedding vector z′ r ∈R d , the process is as follows:

[0228]

[0229] The linear matrix N t (W r _O) is W r O ijm A collection of .

[0230] (3) The third stage is process node embedding. Based on the embedding of machine nodes and worker nodes, process node embedding is performed, such as Figure 4 (c) as shown:

[0231] Define 6 MLPs For the eigenvector x ij-1 , x ij , x ij+1 , y′ k and z′ r Each MLP has a d-dimensional output, two d ω The hidden layer of the dimensional network is activated by the exponential linear unit (ELU). Then, the processing results are concatenated and ELU activation and MLP processing are performed. Finally, the node O is obtained. ij The embedding vector x′ ij ∈R d The embedding process is as follows:

[0232]

[0233] Among them, x ij-1 , x ij , x ij+1 Node O ij-1 , O ij and O ij+1 The original vector. It is node O ij All adjacent machine nodes M′ r The embedding vector y′ k The sum average. It is node O ij All neighboring worker nodes W′ r The embedding vector z′ r The average of the sum.

[0234] Constructing the NHGNN model: Constructing the NHGNN model by stacking N embedding layers with the same structure but independently trainable parameters to obtain the final embedding vector and Pool the three types of embedding vectors and concatenate them to obtain the heterogeneous disjunction graph H t The global state embedding vector h t ∈R 3d , the process is as follows:

[0235]

[0236] Among them, |O|, |M| and |W| are H t The number of process nodes, machine nodes, and worker nodes in the process.

[0237] The environment model module is used to build the reinforcement learning environment model of DRC-DFJSP.

[0238] Construct an MDP-based reinforcement learning environment model, including:

[0239] State: At time step t, state s t All static and dynamic features of DRC-DFJSP are included, and the proposed heterogeneous disjunctive graph is used to represent the state of DRC-DFJSP.

[0240] Action: At time step t, action a t (a o , a m , a w ) By process action a o ∈A o 、Machine Action m ∈A m and worker action a w ∈A w Composition, A o is a set of optional processes, A m It is an action o A set of compatible machines, A w It is an action o The set of compatible workers;

[0241] Transfer: At time step t, the agent is based on state s t Take action a t , interact with the environment to move to a new state s t+1 , and based on action a t Update heterogeneous disjunctive graph H t+1 ;

[0242] Reward: r1 is the instant reward for the maximum completion time, and the average utilization rate of the machine is calculated Among them, CT(t) is the maximum actual completion at time step t, and r1 takes the following values:

[0243]

[0244] r2 is the instant reward of the total cost, which is calculated by the total processing time of workers and machines With total cost L c The ratio between (t) The values ​​of r2 are as follows:

[0245]

[0246] The weighted reward is r t =ω1r1+ω1r2, satisfying ω1+ω2=1 and ω1, ω2≥0.1. i After training reaches a certain number of times, save the ω i Then, select the ω with the smallest Euclidean distance i+1 Continue training and repeat until all agent parameters corresponding to all weight vectors are saved;

[0247] Strategy: Multi-action strategy π θ (a o , a m , a w |s t ) By process strategy Machine Strategy and worker strategies composition;

[0248] The scheduling model module is used to train the scheduling model using the MPPO algorithm and generate an optimized scheduling model.

[0249] Scheduling model:

[0250] The scheduling model consists of three Actor networks and one Critic network. The overall network structure is shown in Figure 6:

[0251] In status t Each feasible process action a o ∈A o The corresponding embedding vector and the global state embedding feature h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a o The probability distribution of choice P(a o ,s t ), normalized to obtain the process

[0252] In statust And select process action a o Under the condition that each feasible machine action a m ∈A m The corresponding embedding vector and h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a m The probability distribution of choice P(a m ,s t _a o ), normalized to obtain the machine strategy

[0253] In status t And select process action a o and machine action a m Under the condition that each feasible worker action a w ∈A w The corresponding embedding vector and h t cascade, and use two d θ dimensional hidden layer and tanh activation Processing, get a w The probability distribution of choice P(a w ,s t _a o _a m ), normalized to obtain the worker strategy

[0254] Use a function with two d φ dimensional hidden layer and tanh activation MLP φ Processing t , get the Critic network v φ .

[0255] MPPO algorithm:

[0256] The proximal policy optimization (PPO) algorithm is an advanced policy gradient method of the Actor-Critic architecture and is widely used to process discrete and continuous control tasks. However, DRC-DFJSP is a multi-action task and the PPO algorithm cannot be directly applied. This paper proposes a multi-action proximal policy optimization (MPPO) algorithm, which uses the MPPO algorithm to pre-train the scheduling model offline. The specific steps are as follows:

[0257] Calculate the advantage function Where γ is the discount factor, t is the current time step, and v φ (s t ) is the Critic network according to the state s t The value function value obtained, T<<|O|;

[0258] In each iteration of the training process, three Actors interact with the environment to collect experience quads (s t , a t , r t ,s t+1 ), and train the Actor network and update the Critic network through the collected samples:

[0259] (1) Training the Actor Network

[0260] Each Actor network is trained to maximize its own loss function Among them, c p =1 and c e =0.01 are the cutoff hyperparameter and entropy coefficient hyperparameter respectively. is the pruning proxy target, are entropy targets, which are as follows:

[0261]

[0262] in, It is a target strategy and behavioral strategies for sampling The probability ratio of , ∈∈(0, 1) is the clipping parameter.

[0263] (2) Update the Critic Network

[0264] The Critic network minimizes the mean squared error (MSE) target L MSE (φ) is updated:

[0265] L MSE (φ)=E t [MSE(r t , v φ (s t ))]

[0266] MPPO algorithm offline training scheduling model:

[0267] Initialize the MPPO algorithm parameters and randomly generate B instances. The behavior actor interacts with the environment to schedule each instance and collects the experience quadruple (s t , a t , r t ,st+1 ) The quadruple is stored in the experience replay pool until all tasks are scheduled. Then, the probability ratio is calculated Target Actor Error and Critic error L MSE (φ), and update the network E by the gradient ascent method u Finally, set the target Actor parameter Copy to Behavior Actor Parameters Complete one iteration. Training total iteration E t times, each iteration e val Validate on an independent validation set, each iteration e rep Replace a batch of B instances each time until the convergence criteria are met, the training is completed, and the optimized scheduling model is obtained.

[0268] The optimization scheduling module is used to generate scheduling solutions using the optimization scheduling model.

[0269] The optimization scheduling model based on the MPPO algorithm is used for workshop production optimization scheduling. In actual production, the global state embedding vector is input into the optimization scheduling model trained offline by the MPPO algorithm. The multi-action strategy network outputs the scheduling actions of process selection, machine and worker allocation until a complete scheduling plan is generated.

[0270] When dynamic events occur in actual production, the scheduling plan is generated using the optimization scheduling model and sent to the manufacturing execution module for production until the production is completed. In the actual production data of an aviation equipment manufacturing company, the company has 6 equipment and 5 workers responsible for operating these equipment. The initial number of processed parts is 10, and the number of subsequent new workpieces is also 10. Under the same feature extraction process, the unimproved PPO method and MPPO are used for scheduling solution, and the results shown in Figures 6 and 7 are obtained. The white frame line in the figure is the original workpiece in the workshop, and the black frame line is the newly arrived workpiece. In comparison, the completion time of the improved MPPO algorithm is 214 hours and the cost is 2375 yuan, while the completion time of the unimproved PPO algorithm is 246 hours and the cost is 2485 yuan. Through feature extraction and MPPO optimization scheduling, production efficiency has been significantly improved, and both completion time and cost have been reduced. The completion time has been reduced by 13.20% and the cost has been reduced by 4.42%.

[0271] Compared with the prior art, the advantages of the present invention are as follows: the present invention solves the problems existing in the existing dual-resource manufacturing system scheduling method, such as low problem model accuracy, poor practicality of scheduling schemes, and difficulty in dealing with dynamic events. Since there are still few studies that consider the characteristics of workers, there are differences between the problem model and actual production, resulting in the lack of practicality of the scheduling scheme. Although the traditional scheduling method can obtain an accurate solution, its computational complexity is high, the solution time increases exponentially, and it is impossible to respond to dynamic events in a limited time to find a satisfactory solution for large-scale dynamic instances.

[0272] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-resource flexible job shop dynamic scheduling method based on graph neural network, characterized in that: include: Step 1: Establish and train a dynamic scheduling model for a dual-resource flexible job shop based on worker fatigue and proficiency; Step 2, using the real-time data of production factors, construct a dynamic flexible job shop heterogeneous disjunction diagram under dual resource constraints to characterize the actual production status of the workshop; Step 3, using a novel heterogeneous graph neural network to extract features from the heterogeneous disjunctive graph to obtain a global state embedding vector; Step 4, inputting the global state embedding vector into the dual-resource flexible job shop dynamic scheduling model, and recording the output result; When a dynamic event occurs, steps 2 to 4 are repeated until the production task is completed.

2. The method according to claim 1, characterized in that In the dual-resource flexible job shop dynamic scheduling model, the impact of worker fatigue and proficiency on process processing time is expressed as: Among them, p ijkr For workers r On machine M k Upper processing step O ij The actual processing time, p ijk O ij In M k The preset processing time on is the parameter of the influence of fatigue on the processing time, FL r (t) is W at time t r Fatigue parameter, PL ijr W r Execute O ij The proficiency parameter of the i-th workpiece J i The index of j is workpiece J i The jth process O ij The index of k is the kth machine M k The index of r is the rth worker W r The index of .

3. The method according to claim 1, characterized in that The class of real-time data of production factors The types include: workpiece real-time data, machine real-time data, worker real-time data, workpiece-machine associated real-time data and workpiece-worker associated real-time data; Among them, the real-time data of workers include: the worker's fatigue level, the number of processes that the worker can process, the worker's available time, and the worker's utilization rate; the real-time data associated with the workpiece and the machine include: the preset processing time of each process on the machine; the real-time data associated with the workpiece and the worker include: the worker's proficiency in processing each process and the unit time processing cost of the worker in processing each process.

4. The method according to claim 3, characterized in that The construction of a dynamic flexible job shop heterogeneous disjunctive graph under dual resource constraints for characterizing the actual production status of the workshop includes: Constructing a process node, wherein the original vector of the process node is the real-time data of the workpiece; Constructing a machine node, wherein the original vector of the machine node is the real-time data of the machine; Construct a worker node, wherein the original vector of the worker node is the real-time data of the worker; Add conjunction arcs between process nodes to constrain the processing sequence between processes of the same workpiece; An OM arc is added between a process node and its compatible machine node, wherein the original vector of the OM arc is workpiece-machine association real-time data; An OW arc is added between a process node and its compatible worker node, wherein the original vector of the OW arc is the workpiece-worker association real-time data; After that, the heterogeneous disjunctive graph is continuously updated, including: when a process is scheduled, only the arcs between this process and the machines and workers assigned to it are retained, and then the other OM arcs and OW arcs of this process are removed; the process of updating the heterogeneous disjunctive graph is repeated until all processes are scheduled and a scheduling plan graph is generated.

5. The method according to claim 4, characterized in that The step 3 comprises: constructing an embedding layer, in which machine nodes, worker nodes, and operation nodes are sequentially embedded; The NHGNN model is constructed by stacking N embedding layers. The final embedding vectors of the three types of nodes are obtained through the NHGNN model, where N is a positive integer. The average values ​​of the three types of embedding vectors are then calculated and concatenated to obtain the global state embedding vector.

6. The method according to claim 5, characterized in that In the process of embedding the machine nodes, the worker nodes of the second-order neighborhood are embedded into the operation nodes of the first-order neighborhood, and the operation nodes of the first-order neighborhood are embedded into the machine node as the target node; In the process of worker node embedding, the machine nodes of the second-order neighborhood are embedded into the operation nodes of the first-order neighborhood, and the operation nodes of the first-order neighborhood are embedded into the artificial nodes as the target nodes; In the process of embedding operation nodes, the vectors of each node are processed and cascaded through six MLPs, wherein the vectors of each node include: the original vector of an operation node, the original vectors of the previous and next operation nodes of this operation node, the embedded vectors of the machine nodes adjacent to this operation node, and the embedded vectors of the worker nodes adjacent to this operation node.

7. The method according to claim 1, characterized in that Also includes: The dual-resource flexible job shop dynamic scheduling model is trained by MPPO (multi-action proximal policy optimization) algorithm; Among them, based on the embedding vectors corresponding to feasible process actions and the features of the global state embedding vector, the process Actor network is constructed; based on the embedding vectors corresponding to feasible machine actions and the features of the global state embedding vector, the machine Actor network is constructed; based on the embedding vectors corresponding to feasible worker actions and the features of the global state embedding vector, the worker Actor network is constructed; Based on the features of the global state embedding vector, a Critic network is constructed.

8. The method according to claim 1, characterized in that The dynamic events include: new workpieces arrive randomly, where the arrival time of the new workpieces approximately follows Poisson distribution; the newly arrived workpieces are mixed with unprocessed processes for dynamic scheduling, while the processing processes cannot be terminated and the processed processes are no longer scheduled.

9. The method according to any one of claims 1 to 8, characterized in that: The method is performed by a dual-resource flexible job shop dynamic scheduling system based on a graph neural network, and the system includes: Virtual-reality interaction module, used for two-way interaction between physical workshop and virtual workshop; The physical workshop is used to execute the scheduling plan and collect real-time data to support dynamic scheduling of the virtual workshop. The physical workshop includes: a data collection module for collecting real-time data of workshop production factors; a manufacturing execution module for executing the scheduling plan to organize actual production; The virtual workshop is used to respond to dynamic events and formulate scheduling plans to guide the actual production of the physical workshop. The virtual workshop includes: a state representation module, which is used to construct a heterogeneous disjunctive graph of DRC-DFJSP; a feature extraction module, which is used to construct NHGNN to extract global state embedding vectors from the heterogeneous disjunctive graph; an environment model module, which is used to construct a reinforcement learning environment model of DRC-DFJSP; a scheduling model module, which is used to use the MPPO algorithm to train the scheduling model and generate an optimized scheduling model; an optimized scheduling module, which is used to generate a scheduling plan using the optimized scheduling model.

Images