Dynamic flexible workshop scheduling method and related equipment

Abstract

The embodiment of the invention provides a dynamic flexible workshop scheduling method and related equipment, and belongs to the technical field of industrial intelligent manufacturing and production scheduling. The method comprises the steps of obtaining current state information of a workshop in response to a scheduling event; inputting the state information into a pre-trained scheduling decision model for processing; the model outputs state feature embedding through a two-stage feature extraction network: in the first stage, feature extraction is performed on a heterogeneous disjunction graph by using a graph attention network, and in the second stage, expert output is dynamically fused through a hybrid expert model; and finally, outputting and executing a process-machine pairing decision by the actor network. Wherein the model is trained by adopting a near-end strategy optimization algorithm based on multiple commentators; and a meta-learning framework is integrated during training, so that the model obtains strong generalization ability. According to the method, the defects of an existing scheduling method in the aspects of state characterization, multi-target tradeoff and environmental adaptability are effectively overcome, and the efficiency, quality and robustness of dynamic flexible workshop scheduling are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial intelligent manufacturing and production scheduling, and in particular to a dynamic flexible job shop scheduling method and related equipment. BACKGROUND

[0002] Flexible job shop scheduling problem (FJSP) is one of the core problems of modern intelligent manufacturing system. Compared with the classic job shop scheduling problem (JSSP), each process in FJSP can be processed on multiple candidate machines, and the processing time may be different, so the solution space is larger and the complexity is higher. The dynamic flexible job shop scheduling problem (DFJSP) further considers the dynamic disturbance events commonly existing in the actual production environment, such as emergency order insertion and machine random failure, and requires the scheduling system to have the ability of real-time response and dynamic adjustment.

[0003] At present, the methods for solving DFJSP are mainly divided into four categories: exact solution algorithm, heuristic algorithm, meta-heuristic algorithm and reinforcement learning (RL) algorithm. Although the exact solution algorithm (such as mixed integer linear programming MILP) can obtain the optimal solution, it has high computational complexity and is difficult to meet the dynamic real-time scheduling requirements. The heuristic algorithm has fast response speed, but the solution quality is not high and it is easy to fall into local optimum. The meta-heuristic algorithm (such as genetic algorithm GA, particle swarm optimization PSO) can obtain a relatively optimal solution, but it usually needs a long iteration calculation time and has poor adaptability to dynamic events.

[0004] In recent years, deep reinforcement learning has been applied to DFJSP due to its strong sequence decision-making ability. It models the scheduling process as a Markov decision process (MDP) and learns the optimal scheduling strategy by interacting with the environment. However, the existing DRL methods still face many challenges when dealing with DFJSP: 1) the state space is huge and dynamically changing, which is difficult to effectively represent; 2) there is a conflict between multiple optimization objectives (such as completion time, machine load balancing, delivery period), and a single reward function is difficult to effectively balance; 3) when the training environment and the test environment (such as machine failure distribution, production scale) differ greatly, the generalization performance of the trained model decreases sharply, and it is difficult to adapt to the variability of real production scenarios. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a dynamic flexible job shop scheduling method based on deep reinforcement learning (Deep Reinforcement Learning, DRL) and meta-learning (Meta-Learning) and related equipment.

[0006] To achieve the above purpose, one aspect of the embodiments of the present application proposes a dynamic flexible job shop scheduling method, which comprises the following steps:

[0007] S1, in response to a scheduling event trigger, obtaining current state information of a flexible job shop, the current state information including a feature vector of at least one to-be-scheduled process, a feature vector of at least one available machine, and a feature vector of a process-machine pair;

[0008] S2, inputting the current state information into a pre-trained scheduling decision model, processing by a feature extraction network in the scheduling decision model, and outputting a state feature embedding;

[0009] The feature extraction network adopts a two-stage feature extraction architecture: in the first stage, a graph attention network is used to extract features from a heterogeneous abstraction graph composed of process nodes and machine nodes, to obtain first feature embeddings of the process nodes and first feature embeddings of the machine nodes; in the second stage, the first feature embeddings are input into a hybrid expert module for processing, the hybrid expert module including a first expert group for processing process features and a second expert group for processing machine features, and the state feature embedding is obtained by dynamically selecting experts and weighting and fusing expert outputs through a gating network.

[0010] S3, outputting a scheduling action based on the state feature embedding by an actor network in the scheduling decision model, the scheduling action being a process-machine pairing selected from a current available action space;

[0011] S4, executing the scheduling action to assign the selected process to the selected machine for processing;

[0012] The scheduling decision model is trained by a multi-critic PPO algorithm based on multiple critics, and the multi-critic PPO algorithm sets up independent critic networks for multiple optimization objectives, and each critic network is used to evaluate the state value of the corresponding optimization objective.

[0013] In some embodiments, during training of the scheduling decision model, an integrated meta-learning framework is used, specifically including:

[0014] A plurality of different scheduling task environments are sampled from a task distribution for meta-training, each scheduling task environment having a different dynamic event distribution;

[0015] For each sampled scheduling task environment, in an inner loop, the current model parameters are interacted with the environment, a task-specific loss is calculated, and the model parameters are updated for the first time;

[0016] In an outer loop, the meta-loss is calculated based on the performance of the model parameters updated for the first time in each task environment, and the original model parameters are updated for the second time;

[0017] The meta-training enables the scheduling decision model to obtain initial parameters capable of quickly adapting to a new scheduling task environment.

[0018] In some embodiments, the meta-learning framework is a model-agnostic meta-learning (MAML) framework, and the meta-loss gradient calculation in the outer loop adopts a first-order approximation (FOMAML) algorithm.

[0019] In some embodiments, the heterogeneous disjunctive graph is a heterogeneous disjunctive graph with machine competition relations, which includes:

[0020] a set of process nodes, a set of machine nodes, a set of conjunctive arcs representing the processing sequence between processes, a set of selective arcs representing the connection relationship between machines and processable processes, and a set of competition association edges representing the competition relationship between machine nodes capable of processing the same process.

[0021] In some embodiments, in the first stage, for feature extraction of a machine node, the neighbor nodes thereof include: a process node that can be currently processed by the machine, and other machine nodes having a competition relationship connected through the competition association edge; and the attention coefficient between machine nodes is calculated based on the features of the two machine nodes and a competition feature vector representing the competition relationship therebetween.

[0022] In some embodiments, the plurality of optimization objectives include minimizing total completion time, minimizing machine load standard deviation, and minimizing average job delay time; and the number of critic networks is three, each corresponding to one of the three optimization objectives.

[0023] In some embodiments, the reward signal of each step is a triple r t = <r t (1) , r t (2) , r t (3) >, respectively corresponding to the reward values of the three optimization objectives; and the total loss function is:

[0024]

[0025] wherein L t (i)CLIP (θ) and are respectively a policy loss and a value function loss calculated based on the ithcritic network, L t E (θ) is an entropy regularization term, and c1, c2, and c3 are weight coefficients.

[0026] To achieve the above object, another aspect of the embodiments of the present application proposes a dynamic flexible workshop scheduling device, which comprises:

[0027] a state acquisition module configured to acquire current state information of the flexible job shop in response to a dispatch event trigger;

[0028] a dispatch decision module in which a pre-trained dispatch decision model is embedded, configured to receive the current state information and output a dispatch action; the dispatch decision model comprises:

[0029] a feature extraction network adopting a two-stage feature extraction architecture, configured to: in a first stage, utilize a graph attention network to perform feature extraction on a heterogeneous abstraction graph composed of process nodes and machine nodes, to obtain first feature embeddings of the process nodes and first feature embeddings of the machine nodes; in a second stage, input the first feature embeddings into a hybrid expert module for processing, dynamically select experts through a gating network and weight the expert outputs to obtain a state feature embedding;

[0030] an actor network configured to output a dispatch action based on the state feature embedding;

[0031] an action execution module configured to issue process-machine pairing instructions output by the dispatch decision module to the shop for execution;

[0032] wherein the dispatch decision model is trained by a multi-critic proximal policy optimization (PPO) algorithm, and the multi-critic PPO algorithm sets independent critic networks for multiple optimization objectives.

[0033] In some embodiments, the apparatus further comprises a model training module configured to integrate a meta-learning framework when training the dispatch decision model, and perform inner loop and outer loop updates by sampling different task environments, so that the model obtains initial parameters capable of quickly adapting to new tasks.

[0034] In some embodiments, the heterogeneous abstraction graph is a heterogeneous abstraction graph with machine competition relationships, and contains competition association edges connecting machine nodes with competition relationships.

[0035] In some embodiments, the hybrid expert module comprises a first expert group for processing process features and a second expert group for processing machine features; and the gating network is configured separately for each expert group.

[0036] In some embodiments, the model training module configures a load balancing loss for the gating network of each expert group when training, to encourage all experts to be activated evenly.

[0037] In some embodiments, the multiple optimization objectives include minimizing total completion time, minimizing machine load standard deviation, and minimizing average job delay time.

[0038] In some embodiments, the device is communicatively connected with a manufacturing execution system (MES) and / or a workshop Internet of Things platform to obtain state information in real time and issue scheduling instructions.

[0039] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0040] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0041] To achieve the above object, another aspect of the embodiments of the present application provides a computer program product, which comprises a computer program, and the computer program implements the above method when executed by a processor

[0042] The embodiments of the present application at least have the following beneficial effects: the present application provides a dynamic flexible workshop scheduling method, device, electronic device, storage medium and program product, which effectively improves the decision quality, multi-objective trade-off ability and generalization robustness of the scheduling strategy in a dynamic and variable environment through the innovative two-stage feature extraction network, multi-critic optimization architecture and meta-learning training framework. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the overall flowchart of the dynamic flexible workshop scheduling method based on deep reinforcement learning and meta-learning provided by the embodiments of the present application;

[0044] Figure 2 is the heterogeneous disjunctive graph with machine competition relationship in the embodiments of the present application;

[0045] Figure 3 is the structure diagram of the stage feature extraction method in the embodiments of the present application;

[0046] Figure 4 is the meta-learning training flowchart in the embodiments of the present application;

[0047] Figure 5 is the solution distribution diagram of different methods on real scene instances in the embodiments of the present application;

[0048] Figure 6 is the three-dimensional scatter diagram of the meta-learning ablation experiment under different instances in the embodiments of the present application;

[0049] Figure 7 is the three-dimensional scatter diagram of the multi-critic experiment under different instances in the embodiments of the present application;

[0050] Figure 8 is a flow chart of a dynamic flexible job shop scheduling method provided by an embodiment of the present application; DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description relates to the accompanying drawings, the same numerals in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all the implementations consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0053] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0054] 1) Deep Reinforcement Learning (DRL) is an artificial intelligence method combining the perception ability of deep learning and the decision-making ability of reinforcement learning, which processes high-dimensional state input through neural network and realizes autonomous decision optimization in complex environment.

[0055] 2) Meta-Learning is a methodology in the field of machine learning, and the core goal is to enable the model to learn a general strategy from multiple tasks, so as to have the ability to quickly adapt to new tasks, which is called "learning to learn".

[0056] Modern smart factories begin to realize the interconnection of devices, machines and systems through the Internet of Things, and extract valuable information from the production process using big data and analysis tools to achieve automation and intelligence of the production process. Flexible job shop scheduling problem (FJSP) is a highly abstracted production environment in modern smart factories. Compared with the classic job shop scheduling problem (JSSP), which assumes that each operation of each job can only be processed by one machine, the operation in FJSP can be processed by one or more candidate machines, and the processing time of each operation may be different on the same or different machines with different configurations. In addition to considering the operation selection in JSSP, FJSP also needs to consider the machine assignment problem, so it has a more complex topology and a larger solution space than JSSP, further increasing the complexity of the problem. However, the static FJSP is too idealistic, and in the actual production environment, there are various uncertainties caused by dynamic events such as order insertion, cancellation or modification, and machine failure. Therefore, in recent years, how to solve the dynamic flexible job shop scheduling problem (DFJSP) has attracted widespread attention, and how to effectively respond to various dynamic events in the production of the workshop and maintain the smooth and efficient progress of the scheduling process has become a major challenge in the field of dynamic scheduling.

[0057] DFJSP is an attractive research area in academia and industry, and has been intensively studied in the past few decades. Various methods have been proposed, and the four most commonly used methods are exact solution algorithms, heuristic algorithms, meta-heuristic algorithms, and reinforcement learning algorithms. For exact solution algorithms, it is difficult to accurately model the constraints in the actual industrial scenario, and the calculation time of the exact solution algorithm is relatively long, so it is not suitable for complex and real-time changing scenarios. Heuristic algorithms can immediately respond to dynamic events, achieving the best time efficiency. However, heuristic algorithms have the following problems: 1) Heuristic algorithms are usually greedy algorithms that can only find local optimal solutions and have no effective way to escape local optimal solutions, so it is difficult to obtain better scheduling results. 2) In addition, specific experience is required to design corresponding heuristic algorithms for specific problems, and there is no universal framework, so it is not universal. Meta-heuristic algorithms divide the dynamic scheduling problem into a series of static sub-problems and solve them respectively using intelligent optimization algorithms such as genetic algorithms and particle swarm optimization algorithms. These algorithms usually design methods to escape local optimal solutions, so they can obtain better solution quality than heuristic algorithms, and meta-heuristic algorithms are designed for general problems and have a basic framework, so they are more universal. However, meta-heuristic algorithms usually require a certain number of iterations to obtain better solutions, so they can be very time-consuming and are not suitable for real-time scheduling in dynamic scenarios.

[0058] With the development of the field of reinforcement learning (RL), in order to realize large-scale dynamic scheduling, the method based on reinforcement learning is widely concerned because it can quickly and effectively approximate the optimal solution. Since the DFJSP can be regarded as a sequential decision problem, the scheduling process can be modeled as a Markov decision process (MDP), so more and more research considers applying reinforcement learning to DFJSP, and at each time step, the optimal action is selected according to the current state to make sequential scheduling. However, as the problem size increases, the number of possible states increases dramatically, which leads to an exponential increase in the demand for computing and storage resources, i.e., the state space explosion problem, so in recent years, more and more research considers using deep reinforcement learning (DRL) to solve this problem, so that the reinforcement learning algorithm can be applied to more complex and high-dimensional actual production scheduling.

[0059] For task-related dynamic scheduling scenarios such as random task arrival, sequential decision-making is required, which is very suitable for using deep reinforcement learning to solve related scheduling problems, so in recent years, deep reinforcement learning has focused more on solving task-related dynamic scenarios. However, in device-related dynamic scenarios such as machine breakdown, there is less research on deep reinforcement learning, and it is still mainly focused on using meta-heuristic algorithms to solve the problem. The reason is that device-related dynamic scenarios are often difficult to predict and occur infrequently, and although reinforcement learning can adapt to uncertainty, if the dynamic event is difficult to predict and the sample is small, it cannot learn an effective response strategy in time. And once the machine in the scene is interrupted, the resource availability of the current scene changes, which may cause the performance of the reinforcement learning model trained based on the static scene to drop sharply. At the same time, the performance of the deep reinforcement learning model depends largely on the accuracy of the current scene state representation, and in current research, the FJSP scenario is often modeled using a disjunctive graph, treating the process as a graph node, and using directed edges between nodes to indicate the processing order of operations. The information of each node in the disjunctive graph is extracted using graph attention networks (GAT) and self-attention mechanisms to represent the current scene state. However, previous research often ignores the differences between different jobs and different machines, and cannot effectively distinguish information that is irrelevant or important to the current decision, resulting in inaccurate extraction of current state feature information and affecting the model's expression ability. In addition, when modeling the DFJSP problem as a Markov decision process (MDP) model, unreasonable state representation, action space definition, and reward function design can greatly affect the performance of the model and make it difficult to learn an effective scheduling strategy. Finally, in actual production scenarios, multiple optimization objectives such as machine efficiency and energy consumption often need to be considered, and these optimization objectives often conflict with each other, so how to balance multiple optimization objectives is also an important consideration when applying deep reinforcement learning to DFJSP scenarios.

[0060] For the above problems, the present application proposes an end-to-end deep reinforcement learning model framework for solving the DFJSP problem with random job arrival and random machine interruption. The optimization goal of the present application is to minimize the total completion time, machine load standard deviation and average job delay delay time. First, a two-stage feature extraction module combining a graph attention extraction module and a mixture of experts (MoE) is proposed. In the first stage, the graph attention network (GAT) is used to extract feature information of process nodes and machine nodes from a heterogeneous abstraction graph. In the second stage, the MoE mechanism is introduced to construct a mixed expert layer to further extract features of the process feature embedding and machine feature embedding obtained in the first stage. Different experts focus on feature extraction in different scenarios, and the importance of different features is dynamically balanced during execution, thereby enhancing the expression ability of the model in dynamic scenarios. Second, the present application proposes a multi-critic PPO algorithm based on multiple critics (Multi-Critic PPO). Different critic networks are configured for different optimization goals to help the model effectively balance different goals during training and improve the decision-making performance of the model in multi-objective problem scenarios. Finally, in order to enhance the scheduling robustness of the model, the present application integrates a meta-learning framework to improve the model's ability to adapt to dynamic changing environments, thereby training a more general decision-making model.

[0061] As shown in Figure 8 The present embodiment provides a dynamic flexible job shop scheduling method, which comprises the following steps:

[0062] S1, in response to a scheduling event trigger, obtaining current state information of a flexible job shop, the current state information comprising a feature vector of at least one to-be-scheduled process, a feature vector of at least one available machine, and a feature vector of a process-machine pair;

[0063] S2, inputting the current state information into a pre-trained scheduling decision model, processing by a feature extraction network in the scheduling decision model, and outputting a state feature embedding;

[0064] The feature extraction network adopts a two-stage feature extraction architecture: in the first stage, a graph attention network is used to extract features from a heterogeneous abstraction graph composed of process nodes and machine nodes to obtain first feature embeddings of process nodes and first feature embeddings of machine nodes; in the second stage, the first feature embeddings are input into a mixed expert module for processing, the mixed expert module comprising a first expert group for processing process features and a second expert group for processing machine features, and the experts are dynamically selected and weighted fused by a gating network to obtain the state feature embedding;

[0065] S3, output a scheduling action based on the state feature embedding through an actor network in the scheduling decision model, the scheduling action being a process-machine pair selected from a current available action space;

[0066] S4, execute the scheduling action to assign the selected process to the selected machine for processing;

[0067] Wherein, the scheduling decision model is trained based on a multi-critic proximal policy optimization (PPO) algorithm, the multi-critic PPO algorithm setting an independent critic network for each optimization objective, each critic network being used to evaluate the state value of the corresponding optimization objective.

[0068] In some embodiments, a mixed integer linear programming (MILP) model is first established for a multi-objective dynamic flexible job shop scheduling problem (MODFJSP) to be solved, to accurately describe the problem constraints and objectives. The modeling defines decision variables (such as process-machine assignment variables X ijk , process order variables Y iji′j′k ), constraint conditions (such as process uniqueness constraints, timing constraints, machine interruption constraints, etc.), and optimization objectives (such as minimizing the maximum completion time C max , machine load standard deviation W std , average delay time T mean . This MILP model can be used to verify the quality of the solution results of the DRL model.

[0069] Then, the scheduling process is modeled as a Markov decision process (MDP):

[0070] 1) State (State): State S t is a set of three types of feature vectors: process feature vectors (containing static features such as average processing time, dynamic features such as waiting time), machine feature vectors (containing static features such as machine interruption duration, dynamic features such as current load), and process-machine pair feature vectors (such as the ratio of processing time t ijk to maximum processing time).

[0071] 2) Action (Action): Action a t is a feasible "process-machine" pair selected from the action space composed of the current ready process set and available machine set at decision time t.

[0072] 3) Reward (Reward): Reward r t is a three-dimensional tuple r t = <r t (1) , r t (2) , r t(3) , which correspond to the instantaneous rewards of the three optimization objectives respectively, are designed as the reduction of the estimated value of each objective after each action.

[0073] In some embodiments, the core of the scheduling decision model is a two-stage feature extraction network:

[0074] 1) Stage 1: GAT feature extraction. The input is the constructed heterogeneous disjunctive graph with machine competition relationship, which adds undirected edges (competition association edges) connecting machine nodes with competition relationship (i.e., can process the same process) on the basis of traditional disjunctive graph. Two-layer multi-head graph attention network (GAT) is used to extract features of process nodes and machine nodes respectively. For machine nodes, its neighbors include process nodes that can be processed and machine nodes connected through competition association edges. The attention coefficient between machine nodes is calculated by the formula , where g kq is the feature vector representing the competition relationship between machine M k and M q . The preliminary feature embedding of process and machine is output in this stage.

[0075] 2) Stage 2: MoE feature extraction. The feature embedding obtained in stage 1 is input into two independent mixed experts (MoE) layers respectively. The process MoE layer contains 5 expert networks, and the machine MoE layer contains 3 expert networks. Each expert is a feedforward neural network. The gating network dynamically calculates the weight of each expert according to the current input feature, and weights the fusion of expert output, as shown in the formula . This mechanism enables the model to adaptively select the most suitable feature extraction expert for the current dynamic scenario. Finally, the process embedding, machine embedding, their global pooling embedding, and the hand-crafted process-machine pair feature are spliced to form the final state feature embedding.

[0076] In some embodiments, the scheduling decision model is trained using a multi-critic PPO algorithm (Multi-Critic PPO).

[0077] 1) The actor network (Actor Network) π θ (a t |s t ) is responsible for outputting the action probability distribution based on the state feature embedding.

[0078] 2) Three independent critic networks (Critic Network) are innovatively introduced to estimate the state value of the three optimization objectives respectively.

[0079] 3) The advantage function A t . Also based on the respective target rewards rt and critics Calculated by GAE method.

[0080] The total loss function is the sum of each target policy loss, value loss and shared entropy regularization term: This way effectively avoids interference between multiple objectives.

[0081] In some embodiments, to improve the generalization of the model, a meta-learning framework MAML is integrated in training:

[0082] 1) Sample N different DFJSP task environments (such as different machine failure rates, different job arrival patterns) from the task distribution Λ.

[0083] 2) Inner loop: for each task T i , interact with the current policy parameter θ to collect data Calculate the loss and update to get the task-specific parameter

[0084] 3) Outer loop: use each task's θ' i Interact with the respective environment to collect new data Calculate the average meta-loss on all tasks and update the initial parameters: This embodiment uses FOMAML to simplify the calculation.

[0085] 4) Through meta-training, the model obtains a set of "sensitive" initial parameters, which can quickly adapt to new tasks.

[0086] After the model training is completed, it is deployed online. When a new job arrives or a machine completes the current task in the system, a scheduling event is triggered. The system obtains the current state s t , inputs it into the trained scheduling decision model, and the model outputs the optimal "process-machine" allocation decision through the above feature extraction network and actor network, and issues it to the manufacturing execution system (MES) or the workshop Internet of Things platform for execution.

[0087] Next, in combination with the accompanying drawings and specific application examples, the scheme of the embodiments of the present application will be described in detail.

[0088] The overall process of the end-to-end scheduling decision framework proposed in this embodiment is shown in Figure 1 , which is composed of the following key modules:

[0089] 1) Mixed Integer Linear Programming (MILP) modeling: To accurately describe the multi-objective dynamic flexible job shop scheduling problem (MODFJSP) scenario targeted by this study, MILP is used to describe the problem. By establishing corresponding constraint formulas for each constraint and optimization objective, the problem scenario is accurately described, and the problem can be accurately solved using a solver, thereby assisting in verifying the performance of the deep reinforcement learning model proposed in this study.

[0090] 2) Markov Decision Process (MDP) modeling: To accurately describe the complete sequence decision-making process, this study models MDP, designs a feature vector that accurately represents the environment state, clearly defines the action space centered on "process-machine" combinations, and constructs reward functions for the three types of optimization objectives. This modeling approach can accurately depict the scheduling process and provide a foundation for training the policy model.

[0091] 3) Two-Stage Feature Embedding: To fully exploit the node features and dynamic event information in heterogeneous abstraction graphs, this study proposes a two-stage feature extraction method. In the first stage, the multi-head attention graph network (GAT) is used to extract local and neighborhood features of process and machine nodes. In the second stage, a hybrid expert model (MoE) is introduced to extract high-level semantic features from the preliminary embedding by dynamically selecting experts to adapt to changes in the scene. The final state feature embedding serves as input to the decision-making network, effectively representing complex dynamic scenarios.

[0092] 4) Multi-Critic PPO: To effectively improve the training performance and convergence quality of the model in multi-objective optimization scenarios, this paper proposes a Multi-Critic PPO algorithm. By constructing critic networks for each optimization objective, it accurately estimates the value function and advantage function, enabling more reasonable target decomposition and strategy optimization, and improving overall learning effectiveness.

[0093] 5) Model-Agnostic Meta-Learning (MAML): To enhance the model's adaptability in dynamic scheduling environments, this paper introduces a meta-learning framework during training. This mechanism enables the agent to quickly adjust its strategy when faced with frequent changes in resource availability due to machine interruptions and other events, thereby improving its generalization and robustness in different dynamic events and different scale scenarios.

[0094] (1) MILP modeling of MODFJSP

[0095] The MODFJSP problem is described as follows: There are n consecutively arriving jobs J = {J1, J2, ..., J...} n} and m available machines M = {M1, M2, ..., M} m Each assignment J i By n i It consists of several processes, of which O ij Indicates assignment J i The j-th process. Each process O ij Only from feasible machine sets Select any machine M k Processing, in machine M k Previous process O ij The processing time is t ijk Homework J i The arrival and end times are denoted as A. i and D i Process O ij The actual start time and finish time are denoted as S. ij and C ij Machine M k The time of the failure and the duration of the repair are respectively denoted as B. k and R k The symbols used in this study are described in Table 1:

[0096] Table 1. Symbol Description

[0097]

[0098]

[0099] To address the problem described above, the Big M method is used to model the problem using MILP: let M be a very large constant value. The binary decision variable X... ijk ∈{0,1}, used to represent O ij Is it in machine M? k Processing is performed if process O ij In machine M k If X is processed above, then ijk =1, otherwise 0. Binary decision variable Y iji′j′k ∈{0,1} is used to represent the order of two processing steps on the same machine. If process O ij In machine M k Above before O i′j′ If Y is processed, then iji′j′k =1, otherwise 0. Continuous decision variable S ij and C ij These represent process O respectively. ijthe actual start time point and the completion time point. The specific MILP modeling is as follows:

[0100] 1) Constraint 1: each process can only be processed by one feasible machine once

[0101]

[0102] 2) Constraint 2: the start processing time of the process is not less than the arrival time of the job

[0103]

[0104] 3) Constraint 3: the difference between the completion time and the start time of the process on the machine is at least the required processing time

[0105]

[0106] 4) Constraint 4: all processes of each job are processed in a fixed order, and the next process can only be processed after the previous process is completed

[0107] S i,j+1 ≥C ij (4)

[0108] 5) Constraint 5: a machine cannot process two processes at the same time

[0109] a) On machine M k , if process O ij is processed before process O i′j′ :

[0110]

[0111] On machine M k , if process O ij is processed after process O i′j′ :

[0112]

[0113] Y iji′j′k and Y i‘j’ijk satisfy the following constraints, that is, when O ij and O i′j′ are located on machine M k , Y iji′j′k +Y i‘j’ijk =1, otherwise Y iji′j′k and Y i‘j’ijk are 0:

[0114]

[0115] 6) Constraint 6: If the process processing is suspended due to machine interruption, the process processing should be resumed immediately on the same machine after the repair is completed. Introduce binary decision variable Z ijk ∈ {0, 1}, which represents whether the process O ij is affected by machine interruption when processing on machine M k , if affected by machine interruption, then Z ijk = 1, otherwise 0. If the process processing is not affected by machine interruption, i.e. Z ijk = 0, then there are two cases, i.e. the process O ij starts after the end of interruption or the process O ij is completed before the start of interruption, which can be expressed in the following constraint form:

[0116]

[0117] where binary decision variables z 1ijk ∈ {0, 1}, z 2ijk ∈ {0, 1} respectively represent whether formula (9) and formula (10) are satisfied, when formula (9) is satisfied, z 1ijk = 1, z 2ijk = 0, at this time Z ijk = 0; when formula (10) is satisfied, z 1ijk = 0, z 2ijk = 1, at this time Z ijk = 0; if neither formula (9, 10) is satisfied, then z 1ijk = 0, z 2ijk = 0, at this time the process O ij is affected by machine interruption, i.e. Z ijk = 1. From the above, the constraint formula of Z ijk , z 1ijk , z 2ijk can be obtained:

[0118]

[0119] If the process O ij is affected by machine interruption, i.e. Z ijk = 1, if the process processing is suspended due to machine interruption, the process processing should be resumed immediately on the same machine after the repair is completed, the following constraints should be satisfied:

[0120]

[0121] Considering the above two cases of being affected by interruption and not being affected by interruption, the following constraint formula is obtained:

[0122]

[0123] To linearize equation (11), introduce auxiliary binary variable w ijk ∈{0,1} and the following constraints to realize w ijk = Z ijk X ijk :

[0124]

[0125] Equation (11) is then transformed to:

[0126]

[0127] The MILP formulation of the three optimization objectives is as follows:

[0128] a) Minimize total completion time: define variable C i , C max represent the completion time of job J i and total completion time

[0129]

[0130] b) Minimize machine workload standard deviation (including machine downtime): define variable W k , W std represent the machine M k load, average machine load and machine load standard deviation:

[0131]

[0132] Since equation (15) is nonlinear, to linearize the objective constraint, introduce continuous decision variable dev k represent the deviation value of actual machine M k load from average machine load:

[0133]

[0134] Minimizing machine workload standard deviation W std can be approximated to minimizing total deviation value dev total , then the constraint equation is as follows:

[0135]

[0136] c) Minimize average job delay time: define variable T mean represent average delay time

[0137] Optimization objectives:

[0138] In addition, the following pre-defined prerequisites need to be met for this embodiment:

[0139] 1) In this study, the transportation time of jobs and the preparation time of machines are ignored.

[0140] 2) Each job has a delivery deadline that must be met, otherwise it will be considered as overtime.

[0141] 3) Each process can be processed on different machines, and the processing time of the same process on different machines is different.

[0142] 4) Based on practical experience, the length of a single interruption of each machine is known. If the machine is selected for process processing when the machine is in interruption, the process will start after the machine recovers.

[0143] (2) MODFJSP Markov Decision Process Model

[0144] The sequence decision process of the MODFJSP of this embodiment can be modeled as a MDP model, mainly including the modeling of state, action and reward. The MDP modeling of this study is described as follows:

[0145] 2.1) State: used to describe the real-time information of the environment, and the current environmental information is represented as detailed as possible. The state of this study is represented as a set of feature vectors, including process feature vectors, machine feature vectors and process-machine combination feature vectors. The specific composition of the above feature vectors is shown in Table 2:

[0146] Table 2: Composition of feature vectors

[0147]

[0148]

[0149] 2.2) Action: At the decision time point t, the process set O and the machine set M contain the processes and machines that have been ready at the current time point, and the elements in O and M together constitute the current action space A(t). The action in A(t) is each feasible process-machine combination pair.

[0150] 2.3) Reward: For the three optimization objectives of minimizing total completion time, machine load standard deviation and average job delay rate, three reward functions are designed as follows:

[0151] r 1t =C max (S t )-C max (S t+1 ) (19)

[0152] r 2t =EWstd (S t )-EW std (S t+1 ) (20)

[0153] r 3t =ET mean (S t )-ET mean (S t+1 ) (21)

[0154] where C max (S t ), EW std (S t ), ET k (S t ) denote the lower bound of the total completion time, the estimated standard deviation of machine load and the estimated machine load of each machine at the end of all process steps, respectively, in state S t The total reward of each action can be calculated as: t = r 1t + r 2t + r 3t When the discount factor γ = 1, the cumulative total reward is: Since (C max (S0) + W std (S0) + ET mean (S0)) is a constant calculated at the initial state S0, and when all process steps are completed, the estimated standard deviation of machine load EW std (S t ) is equivalent to the actual standard deviation of machine load W std (S t ), and the estimated average delay time ET mean (S t ) is equivalent to the actual average delay time T mean , maximizing the cumulative reward is equivalent to minimizing the total completion time, the standard deviation of machine load and the average job delay time (C max , W std , T mean ).

[0155] (3) Two-stage feature extraction method

[0156] 3.1) Heterogeneous AND-Graph with Machine Competition

[0157] The traditional heterogeneous AND-Graph structure is represented as G = (O, M, C, E), where the set O contains all process nodes O ijand two virtual nodes (start node and end node); M represents the set of machine nodes; C is the set of conjunctive arcs, which are directed edges, used to represent the sequence of processes; E is the set of selection arcs, which are undirected edges, connecting machine nodes and their ready processes. The above heterogeneous disjunctive graph structure cannot establish direct connections between machine nodes, which may limit the associated node information obtained during graph information extraction. Therefore, the above heterogeneous disjunctive graph structure is further modified by introducing associated edges between machines with competitive relationships (machines that can process the same process have competitive relationships) to establish connections between machines. Then, the heterogeneous disjunctive graph with machine competition relationship is represented as G=(O, M, C, E, F), as shown in Figure 2 , F is the set of undirected edges connecting machine nodes with competitive relationships.

[0158] 3.2) Two-stage feature extraction method

[0159] In order to fully extract the node feature information and the uncertainty dynamic event information contained in the heterogeneous disjunctive graph, a two-stage feature extraction method is proposed in this embodiment. In the first stage, a graph attention network (GAT) with multi-head attention mechanism is used to extract features from the above heterogeneous disjunctive graph: process nodes and machine nodes are respectively extracted by process feature extraction module and machine feature extraction module. In this stage, the feature information of the node itself and the context information of its adjacent nodes can be fully captured, and the output process feature embedding and machine feature embedding will be used as the input of the second stage. In the second stage, in order to adapt to the dynamic changes of the scene caused by the change of machine availability, a mixed expert model (MoE) mechanism is introduced in the mixed expert layer, and corresponding feature expert groups are configured for process feature embedding and machine feature embedding. Each expert network in the expert group focuses on feature processing requirements in different scenarios, and the dynamic selection of experts is realized by training the gating network. This stage further extracts higher-level semantic feature information, and after pooling and splicing operations, the final state feature embedding is formed, which is used as the input of the subsequent scheduling decision-making agent. As shown in Figure 3 , it is the structure diagram of the two-stage feature extraction method.

[0160] 1) Stage one: GAT feature extraction

[0161] The input of the GAT feature extraction stage is the node and edge information of the heterogeneous abstraction graph, which is composed of two layers of GAT extraction layers. Each layer of GAT extraction layer is composed of a process feature attention module and a machine feature attention module. The attention weight between the node and the neighbor node is calculated in parallel through the multi-head graph attention mechanism (the number of heads is set to 2 in this embodiment), and the corresponding process feature embedding and machine feature embedding are output as the input of the second stage. The neighbor nodes of the process node are its previous process and next process, and the neighbor nodes of the machine node are the process nodes that can be processed by the machine in the current state and the machine nodes that have a competitive relationship.

[0162] Two modules receive the process feature vector and the machine feature vector respectively and output the corresponding feature embedding. The feature vectors of the process node O ij and the machine node M k are represented as and respectively. The specific process and machine feature information is shown in Table 2. In order to better capture the correlation information between competitive machines, this study designs a competitive feature vector as the input of the machine feature extraction module, which is used to participate in the calculation of the attention coefficient e kq between machines, so as to realize the representation of the local competitive situation. For the machine nodes M k and M q that have a competitive relationship, g kq is calculated as follows:

[0163]

[0164] where C ij represents the set of processes that the machines M k and M q compete for (the processes that are ready in the current state), and c ij is the size of the set, which represents the degree of competition. is the competitive process vector, which combines the original features of the process and the processing capacity difference of the two machines for the process, thereby describing the "importance" of a single process in each pair of competitive machines, as shown in formula (23):

[0165]

[0166] where represents the processing performance difference of the process on the two machines, represents the processing similarity of the process on the two machines, is the feature vector of the process that is competed in the Lth GAT feature extraction layer. The attention coefficient e kq of the machine node M q ∈N(M k ) is calculated as follows:

[0167]

[0168] 2) Stage two: MOE feature extraction

[0169] To adapt to the dynamic environment and improve the generalization of the model, the second stage of feature extraction is introduced in this embodiment. MoE structure is introduced to further extract the process feature embedding and machine feature embedding obtained in the first stage, so as to construct more suitable state feature embedding in real time for the dynamic environment.

[0170] The MOE feature extraction module is composed of 3 layers of mixed expert layers, each layer of mixed expert layer is composed of a feature expert group corresponding to the above two kinds of features, and the number of experts in the two groups of feature expert groups is respectively n operation = 5 and n machine = 3. Taking the process feature expert group as an example, the input of the expert group is the process feature embedding extracted in the first stage Each expert in the group is composed of a double-layer feedforward neural network, denoted as Where k represents the kth expert in the group, and the output of expert k is denoted as The process feature gating network assigns a corresponding weight to the output of each process expert, and then calculates the final expert group output by weighting As shown in formula (25):

[0171]

[0172] Where the weight Each expert is dynamically assigned a weight by the gating network, and the expert output is fused by weighting according to the weight, so that the more suitable expert output feature embedding is selected according to the current state, so that the model can maintain a certain flexibility in complex dynamic scenarios. The machine feature expert group is the same.

[0173] In order to prevent the experts in the expert group from being unevenly loaded, resulting in some experts dominating the training process while the remaining experts have no contribution to the training, the load balancing loss is designed for each feature gating network to encourage the gating network to activate each expert evenly, so as to urge each expert to explore different areas. The load balancing loss L LB is defined as follows:

[0174]

[0175] Where E represents the number of experts in the same expert group, p i is the weight assigned to each expert, and ε is a very small positive number to prevent numerical instability in logarithmic calculation. The smaller the load balancing loss L LB , the smaller the probability pi The more uniform the distribution of the load is, the smaller the load balancing loss L LB is added to the total loss L total to guide the training, where the coefficient λ controls the strength of the influence of the load balancing loss on the training process:

[0176] L total = L total + λL LB (27)

[0177] By constructing a 3-layer mixed shared layer, the expert sequentially processes and transforms the features at multiple depth levels, thereby gradually constructing more complex mapping relationships, and obtaining the process and machine feature embeddings of the second stage output

[0178] 2.3) Final output of the two-stage feature extraction method

[0179] The global feature embedding is constructed by performing pooling operations on the above-mentioned process feature embedding and machine feature embedding to obtain the global process feature embedding and global machine feature embedding, and then splicing them to obtain the global feature embedding h The calculation process is shown in equation (28):

[0180]

[0181] The process feature embedding h the machine feature embedding h and the global feature embedding h L G are spliced with the manually constructed process-machine pair feature vector h pair (see Table 2), and the resulting state feature embedding is used as the output of the two-stage feature extraction module for subsequent agent input. The agent makes specific action decisions based on the current state.

[0182] (4) PPO algorithm based on multiple critics

[0183] 4.1) Proximal Policy Optimization (PPO)

[0184] The deep reinforcement learning algorithm used in this embodiment is PPO. PPO is an algorithm based on the actor-critic structure, that is, in the standard PPO framework, the actor network π θ (a t |s t ) is used as the decision-making network, and outputs the action a tAction distribution under Critic network. Then it is used to estimate the state value function V(s) t ), and is used to calculate the advantage function during policy optimization:

[0185] A t =Q(s) t ,a t )-V(s t (29)

[0186] In the above formula, since the true action value Q(s) cannot be obtained... t ,a t ) and state value V(s) t Therefore, in PPO, GAE (Generalized Advantage Estimation) is usually used to estimate the advantage function A. t Gaussian Advantage (GAE), as a method for estimating the advantage function, combines the advantages of temporal difference (TD) and Monte Carlo (MC) methods. Compared to the single-step TD method, GAE introduces the parameter λ to exponentially weight and accumulate the residuals of multi-step TD, making full use of future reward information and thus effectively reducing estimation bias. Compared to the Monte Carlo method, GAE avoids the high variance problem caused by relying solely on the accumulated reward of the entire trajectory, making the advantage estimation more stable. Therefore, GAE effectively balances the bias and variance of advantage estimation. The advantage function estimation of GAE is defined as follows:

[0187]

[0188] Where γ is the discount factor, λ is the GAE parameter, used to control the weighting of multi-step TD residuals and adjust the balance between bias and variance, and TD residual δ t The calculation is as follows:

[0189] δ t =r t +γV(s t+1 )-V(s t (31)

[0190] PPO constructs the following surrogate objective function L t CLIP (θ), the model parameters are optimized using the gradient ascent method to maximize the objective function:

[0191] L t CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)At )]

[0192] where represents the probability ratio of new and old policies, ∈ is the clipping coefficient, and r t (θ) is the output of the clipping function clip(r t (θ),1-∈,1+∈), if r t (θ)<1-∈, then output 1-∈, and if r(θ)>1+∈, then output 1+∈. The clipping function clip limits the update amplitude of the policy, preventing the new policy from changing too much relative to the old policy, thus ensuring a smoother and more stable training process. In addition to the main objective function of policy optimization, the total loss of PPO usually also includes a mean square error term and an entropy regularization term. The mean square error term As shown in equation (33), it is used to train the critic network to minimize the difference between the value function prediction and the target value.

[0193]

[0194] where

[0195] The entropy regularization term L t E As shown in equation (34), it is used to enhance the exploration ability of the policy, encourage the policy to maintain randomness, and avoid falling into local optima too early.

[0196]

[0197] where represents the entropy of the policy distribution in state s t , which is used to measure the randomness of the policy output action.

[0198] The total loss of PPO is calculated as follows:

[0199]

[0200] where L t CLIP (θ), and respectively represent the policy optimization loss term, the mean square error term and the entropy regularization term, and c1, c2, c3 respectively represent the weights of each loss.

[0201] 4.2) Multi-Critic PPO Structure

[0202] The three optimization objectives in this embodiment are minimizing total completion time, minimizing machine load standard deviation, and minimizing average delay time. In existing research, when using deep reinforcement learning to optimize multiple objectives, the reward values corresponding to multiple objectives are usually weighted and summed in the environment, and returned in scalar form, thereby converting the multi-objective optimization problem into a single-objective optimization problem for solving, as shown in equation (36), where M is the number of objectives, w i is the objective weight:

[0203]

[0204] In the above manner, simply combining the rewards of multiple objectives can easily cause mutual interference between reward signals, resulting in distorted reward signals. Directly optimizing the total reward after fusion can make the model more likely to favor the reward-dominant objective, thereby reducing the trade-off ability of the strategy under multiple objectives. In addition, in a single critic structure, when training a single value function for the weighted and fused total reward, if the scale difference and change trend of the rewards between different objectives are inconsistent, it can cause their gradients to cancel each other out or interfere with each other during backpropagation, thereby affecting the accuracy of value estimation and the stability of training. Therefore, this research proposes a new Multi-Critic PPO structure. In the improved PPO framework of this research, independent critic networks are introduced for value estimation for the three optimization objectives.

[0205] In the Multi-Critic PPO of this embodiment, the three critic networks are represented as In a Markov decision process, the reward signal of the environment at each step is returned in the form of a three-dimensional tuple, i.e., r t t (1) , r t (2) , r t (3) >, respectively, representing the reward values corresponding to the three optimization objectives. For the i-th objective, the corresponding critic network is used to fit its state value, and the corresponding advantage function A t (i) is calculated according to the state value estimated by the critic network.

[0206]

[0207] Based on the independent advantage functions A t (i) , the policy optimization loss term L t (i)CLIP (θ) and the mean square error term ​Three target shared entropy regularization terms L t E (θ), and the final total loss is calculated by fusing each target loss with a weight:

[0208]

[0209] Through the above multi-critic structure, the value function of each target can be accurately modeled. Compared with the traditional reward weighted summation method, the mutual interference of multi-objectives in the value estimation stage can be avoided, and the learning and training process of the agent is smoothed. At the same time, the value output by each critic network can be clearly explained as the performance of the current strategy under a certain target, so it has better interpretability. In addition, the model is trained in a simulation environment, the model is decoupled from the real production system, and the trajectories collected are completely from simulated scheduling instances. Multiple critic networks are only used in the training stage to calculate the advantage function and value loss. After training, only the policy network is deployed for online scheduling. Therefore, the model parameter amount in actual deployment is similar to that of the standard single-critic PPO, which does not significantly increase the computational or storage overhead, thereby ensuring the usability of the algorithm in industrial scheduling systems.

[0210] (5) Integrated meta-reinforcement learning framework Model-Agnostic Meta-Learning (MAML)

[0211] When using reinforcement learning methods to solve flexible job shop scheduling problems, existing research usually targets a static and deterministic scheduling environment, and trains through continuous interaction between the model and the specific environment until the policy model converges. At this time, the strategy obtained by training has good decision-making performance in the current environment. However, in the actual dynamic flexible job shop scheduling problem, due to dynamic disturbance events such as machine interruption, uncertain arrival time of jobs, process route change, etc., the state distribution of the system environment often changes significantly, causing the policy model trained based on a static scenario to degrade or fail in the new environment. At the same time, the scale of the actual scenario is usually quite different from the scale of the training instances, and whether the model can remain effective in scenarios of different scales is also an important problem to be concerned about. To address the above problems, it is necessary to improve the model's adaptability to changing environments to reduce the performance deviation caused by environmental changes. This research proposes to combine deep reinforcement learning with the meta-learning framework Model-Agnostic Meta-Learning (MAML), thereby improving the model's generalization ability and robustness when facing a variable scheduling environment.

[0212] Unlike traditional reinforcement learning methods that train models on a single task environment, meta-reinforcement learning aims to train models on multiple task environments with different characteristics or dynamic changes, enabling the learned initial policy to quickly adapt to new tasks. Meta-learning algorithms focus on "learning how to learn quickly," simulating a multi-task training process to ensure the model maintains strong performance stability and transferability even when facing changes in environment distribution. MAML is a model-agnostic meta-learning framework applicable to supervised learning and reinforcement learning algorithms. In supervised learning scenarios, MAML optimizes the initial values ​​of model parameters, allowing the model to achieve good performance with only a small number of gradient updates when facing new task data. In reinforcement learning scenarios, task data is generated through interaction with the environment; that is, during the meta-training phase, MAML samples a set of different task environments (e.g., with different dynamic event distributions) and executes inner and outer loop processes on each task. Figure 4 As shown.

[0213] In this embodiment, the parameter settings for meta-reinforcement learning are shown in Table 3, and the training process for MAML-based meta-reinforcement learning is as follows:

[0214] 1) Task distribution definition and environment sampling: Sample N task environments T from the task distribution Λ. i Each task environment represents a DFJSP instance with different disturbance patterns (such as machine failure events with different distributions).

[0215] 2) Inner Loop Training: For each task environment T i By interacting with the environment using the currently shared initial policy parameters θ, a batch of trajectories are sampled. As a support set for this task. Use the support set. Calculate the gradient for the policy parameters Backpropagation and updating of the shared initial policy parameters θ yields the adapted parameters θ′ for this task environment. i The update process is shown in formula (39), where α is the learning rate of the inner loop. This step demonstrates the policy's ability to quickly adapt to new tasks with a small number of gradient updates when facing different task environments.

[0216]

[0217] 3) Outer Loop Training: After the inner loop updates for all tasks are completed, the adapted parameters θ′ for each task are used. i Again with the same task environment T i Interact with the data to obtain a new batch of trajectories. as Query Set. Then calculate all task environment T i Corresponding strategy Loss under its corresponding Query Set and average the loss of all tasks to get the total meta-loss, which is used to backpropagate and update the shared initial strategy parameters θ. The update process is shown in equation (40), where β is the learning rate of the outer loop. This step improves the model's ability to quickly adapt to new task environments:

[0218]

[0219] To improve training efficiency, an improved first-order model-independent meta-learning method FOMAML is used, which ignores the second-order derivative term in the calculation process, i.e. equation (40) is modified as follows:

[0220]

[0221] Through the above "inner loop + outer loop" complete process, the model completes a round of meta-reinforcement learning training. By repeating this process, a general strategy that can quickly adapt to different types of dynamic scheduling environments can be learned step by step, significantly improving the model's generalization performance. The strategy trained by MAML can quickly adapt to new environments without or with only a small amount of fine-tuning when encountering new environments similar to the training task distribution. In this embodiment, the model trained by meta-learning can be directly applied to new scenarios under the condition that the task distribution does not change significantly, and maintains good generalization performance, thereby reducing or avoiding additional fine-tuning overhead.

[0222] Table 3 Meta-learning parameters

[0223]

[0224] (6) Experimental results

[0225] To verify the model performance, this embodiment conducts experiments on commonly used public benchmark datasets Hurink and real datasets, and compares the trained strategy model with meta-heuristic methods (MOEA / D, NSGA2), deep reinforcement learning algorithm (Yuan2025) and two heuristic scheduling rules (SPT and MWKR) that perform well in experiments.

[0226] Since the data in the dataset Hurink is static instances, we introduce two types of dynamic events, job random arrival and machine breakdown, by the following probability distributions. We use a Poisson process to simulate the dynamic arrival process of subsequent jobs, which assumes that each job arrival is an independent event and the time interval between two consecutive arrivals follows an exponential distribution T ~ Exponential(λ). Weibull distribution is used to simulate the occurrence time of machine breakdown. Weibull distribution can adjust the shape parameter β to simulate the different failure trends of equipment in the early failure, stable period or aging stage, and the scale parameter η is used to control the time scale, representing the "characteristic life". The larger η is, the longer the average service life of the machine is, and the more durable the machine is. The sampling formula is as follows, where p ~ Uniform(0, 1):

[0227]

[0228] t represents the time from the start of normal operation to the occurrence of machine breakdown, β ~ U(1.2, 5), η ~ U(150, 250), and the repair time R ~ U(50, 200). The deadline D of each job i The calculation is shown in formula (43):

[0229]

[0230] where A i represents the arrival time of job J i , is the average processing time of each process, and f represents the urgency of the job. In order to comprehensively evaluate the performance of the algorithm in multi-objective optimization, three commonly used multi-objective optimization indicators are used in this embodiment:

[0231] 1) GD (Generational Distance): used to measure the closeness between the obtained solution set and the true Pareto frontier. The smaller the GD value is, the closer the solution set is to the true Pareto frontier PF * , and the better the convergence performance of the algorithm is.

[0232] 2) IGD (Inverted Generational Distance): considers the convergence and distribution of the solution set. IGD measures the average distance from each point on the true Pareto frontier to the obtained solution set, which can reflect whether the algorithm uniformly covers the Pareto frontier. The smaller the IGD value is, the more uniform the solution set is distributed and the better the entire frontier is approximated.

[0233] 3) Spread (Δ, diversity index): mainly measures the diversity of the solution set, that is, the distribution range and uniformity of the solution in the target space. A smaller Spread value indicates that the solution is more uniformly distributed and can cover a larger search space range.

[0234] 6.1) Experiments on public benchmark datasets

[0235] The experimental results on benchmark datasets are shown in Table 4, Table 5, Table 6. From the experimental data, we can find that our method (Our MAML) is significantly better than the two multi-objective meta-heuristic methods (MOEA / D, NSGA2) in three indicators, indicating that on public benchmarks, relative to the two meta-heuristic methods, our model can be closer to the reference Pareto front and achieve more uniform distribution on the front. Relative to the deep reinforcement learning method (Yuan), our model is better than Yuan on most benchmarks, especially on medium and large-scale instances. Relative to the PDRs method, SPT / MWKR occasionally outperforms our model in GD / IGD on small-scale benchmark instances, indicating that when the instance structure is simple and the target conflict is weak, the priority scheduling rule method has obvious advantages in determinacy and low variance. On medium / large scale, the GD / IGD of our model is usually the smallest or close to the optimal indicator, showing that the model has stronger solving ability for complex scenarios. In the Spread indicator, our model and PDR have each other's advantages, so overall, the model better balances the uniformity and coverage of the solution set while maintaining convergence.

[0236] In summary, on public benchmarks, our model has certain advantages in GD, IGD, and spread indicators compared to MOEA / D, NSGA2. Compared to deep reinforcement learning methods and PDRs, it is superior on most medium and large-scale instances, indicating that the model has stronger modeling and generalization ability for complex multi-objective scenario problems.

[0237] Table 4. GD indicator

[0238]

[0239]

[0240] Table 5. IGD indicator

[0241]

[0242] Table 6. Spread indicator

[0243]

[0244] 6.2) Experiments on real datasets

[0245] To verify the performance of the proposed scheduling model in real scenarios, the study selected actual production data from a carton production factory for case study. The production system of the enterprise can be abstracted as a dynamic flexible job shop scheduling problem (DFJSP) with random job arrival and frequent machine interruption, with typical industrial characteristics. The enterprise processes the corrugated paperboard produced into corresponding carton finished products according to the orders received daily, and the finished products are distributed to customers by the logistics system. The production process of the carton of the enterprise includes paper splitting, printing, slotting, die cutting, paper mounting, color marking, nailing, and box sticking, etc. The study collected the machine information and historical production information of the enterprise, constructed a set of dynamic scheduling environment instances under real production scenarios, selected two representative production shift data representing idle and busy actual production scenarios respectively, and compared the execution effects of different algorithms under real production data instances, taking total completion time, machine load standard deviation, and average job delay rate as optimization objectives, to verify the performance of the model, and the comparison results are shown in Table 7:

[0246] Table 7. Model comparison under real scenarios

[0247]

[0248] Overall, in idle (instance 1) and busy (instance 2) actual production scenarios, the meta-heuristic method (MOEA / D, NSGA2) can achieve better results compared to the deep reinforcement learning algorithm (Our, Yuan), especially in GD and IGD indicators, showing high convergence. The reason is that our deep reinforcement learning model is trained based on relatively uniform or regularized simulation data, while in real production data, there are significant differences in order quantity and machine capacity, making it difficult for the model to achieve optimal performance in actual production scenarios. The meta-heuristic method does not rely on specific data for training, so it can adapt to complex task size differences, bottleneck processes, and uneven capacity in production scenarios, and has good transferability.

[0249] However, the calculation time required by our model is much lower than that of the meta-heuristic method, so it is more suitable for real-time response to frequent dynamic events. In both instances, our model outperforms the Yuan method, indicating that the model still has good generalization ability and can adapt to actual production environments. In instance 2, our model achieves the optimal Spread indicator, indicating that the solution set obtained by the model is more evenly distributed on the Pareto frontier, and can well balance different optimization objectives, thus obtaining more diverse candidate solutions, providing more choices for decision-makers in actual production.

[0250] Figure 5The distribution of the solution set generated by several methods under the above examples is intuitively shown, and it can be seen that our model mainly differs from the meta-heuristic method in the machine load standard deviation target, and is close to or better than the meta-heuristic method in other optimization targets. Therefore, it is illustrated that our model can still maintain relatively optimal performance in actual production scenarios and has certain usability.

[0251] 6.3) Ablation experiment

[0252] In order to verify the effectiveness of the model method, two groups of ablation experiments are designed as follows, which are respectively used to verify the effectiveness of the meta-learning training method and the PPO algorithm based on multiple critics in the dynamic scene of multi-objective, and the simulation data of different scales are used as experimental data. In the meta-learning ablation experiment, the effect of our model with and without meta-learning training is compared; in the multi-critic ablation experiment, the effect of our model and the traditional PPO training algorithm using weighted summation to calculate the total reward is compared. As shown in the following table, the experimental results of the two groups of ablation experiments are as follows.

[0253] From the meta-learning ablation experiment data shown in Table 8, it can be seen that by adding meta-learning training to the model, the GD and IGD indicators can be significantly improved on instances of different scales, indicating that through meta-learning training, the adaptability of the model in different scale and dynamic event scenarios can be significantly improved, and the convergence and coverage are better, and the effect on the spread indicator is stable, sometimes slightly better, indicating that while improving convergence and coverage, the diversity of the model is not sacrificed.

[0254] From the multi-critic ablation experiment data shown in Table 9, it can be observed that the multi-critic PPO structure can significantly improve the indicators GD and IGD, and the effect on the spread indicator is stable. Therefore, it is illustrated that in the multi-objective optimization problem, by designing multiple critics to optimize each target respectively, compared with the traditional way of weighting and fusing the reward to obtain the total reward for training, the mutual interference between different targets can be effectively alleviated, the strategy improvement direction is more stable, and the convergence and coverage are improved at the same time, which is helpful for training a strategy model that can obtain a high-quality solution set.

[0255] Table 8. Meta-learning ablation experiment

[0256]

[0257] Table 9. Multi-critic ablation experiment

[0258]

[0259] Figure 6 、 Figure 7The three-dimensional scatter plots of the meta-learning ablation experiment and the multi-critic experiment under different instances are respectively shown, and from the figures, it can be obviously seen that the solution set distribution of the model trained by the method has obvious advantages, and the performance improvement of the model brought by the meta-learning and multi-critic structure is more directly shown.

[0260] (7)Beneficial effects

[0261] In summary, compared with the prior art, the method of the application has the following advantages and beneficial effects:

[0262] 1) For the multi-objective DFJSP problem (MODFJSP) in the present research, a mixed integer linear programming (MILP) is used to mathematically model the problem, and a new Markov decision process model (MDP) is designed, which improves the state representation, action space definition and reward design in the MDP modeling, thereby improving the learning efficiency and decision-making ability of the model.

[0263] 2) By combining the two-stage feature extraction method of graph attention network (GAT) and mixed expert model (MoE), the node information and dynamic event information in the heterogeneous abstraction graph can be fully mined, and the most relevant features can be adaptively selected, thereby generating more accurate and expressive state embeddings, providing a solid foundation for intelligent decision-making.

[0264] 3) By using the multi-critic PPO architecture, an independent critic network is designed for each optimization target to estimate the value, avoiding the mutual interference of multi-objective reward signals during fusion, so that the agent can more accurately and stably learn how to optimize multiple conflicting objectives at the same time.

[0265] 4) By integrating the meta-learning framework (MAML), the scheduling model is trained to "learn to learn" on a large number of tasks with different dynamic event distributions, thereby obtaining a set of initial parameters with rapid adaptation capability. When facing unprecedented new dynamic scenarios or production tasks of different scales, the model can maintain excellent scheduling performance with only a small amount of samples or without adjustment, greatly improving the feasibility of deployment in actual industrial environments.

[0266] 5) The experimental results show that the present application is superior to traditional meta-heuristic algorithms, heuristic rules and existing deep reinforcement learning scheduling methods in terms of total completion time, machine load balancing and average delay time, etc. on public benchmark datasets and real production datasets, proving its effectiveness and superiority.

[0267] The embodiment of the application also provides a dynamic flexible workshop scheduling device, which can implement the above-mentioned method, and the device comprises:

[0268] The state acquisition module is configured to acquire current state information of the flexible job shop in response to a scheduling event trigger.

[0269] The scheduling decision module, in which a pre-trained scheduling decision model is embedded, is configured to receive the current state information and output scheduling actions. The scheduling decision model comprises:

[0270] The feature extraction network adopts a two-stage feature extraction architecture and is configured to: in a first stage, utilize a graph attention network to perform feature extraction on a heterogeneous abstraction graph composed of process nodes and machine nodes to obtain first feature embeddings of the process nodes and first feature embeddings of the machine nodes; and in a second stage, input the first feature embeddings into a hybrid expert module for processing, dynamically select experts through a gating network, and weight and fuse expert outputs to obtain state feature embeddings.

[0271] The actor network is configured to output scheduling actions based on the state feature embeddings.

[0272] The action execution module is configured to issue process-machine pairing instructions output by the scheduling decision module to the shop floor for execution.

[0273] The scheduling decision model is trained based on a multi-critic proximal policy optimization (PPO) algorithm. The multi-critic PPO algorithm sets independent critic networks for multiple optimization objectives.

[0274] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The device embodiments specifically implement the same functions as the above method embodiments and achieve the same beneficial effects as the above method embodiments.

[0275] The preferred embodiments of the application are described above with reference to the accompanying drawings, but this does not limit the scope of the application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the application should be within the scope of the application.

Claims

1. A dynamic flexible job shop scheduling method, characterized by, The method comprises the following steps: obtaining current state information of the flexible job shop in response to a scheduling event trigger, the current state information comprising a feature vector of at least one to-be-scheduled process, a feature vector of at least one available machine, and a feature vector of a process-machine pair; inputting the current state information into a pre-trained scheduling decision model, processing by a feature extraction network in the scheduling decision model, and outputting a state feature embedding; the feature extraction network adopts a two-stage feature extraction architecture: in the first stage, a graph attention network is used to extract features from a heterogeneous abstraction graph composed of process nodes and machine nodes to obtain first feature embeddings of the process nodes and first feature embeddings of the machine nodes; in the second stage, the first feature embeddings are input into a hybrid expert module for processing, the hybrid expert module comprising a first expert group for processing process features and a second expert group for processing machine features, and a gating network is used to dynamically select experts and weight and fuse expert outputs to obtain the state feature embedding; outputting a scheduling action based on the state feature embedding by an actor network in the scheduling decision model, the scheduling action being a process-machine pairing selected from a current available action space; executing the scheduling action to assign the selected process to the selected machine for processing; wherein the scheduling decision model is trained by a multi-critic proximal policy optimization (PPO) algorithm, the multi-critic PPO algorithm setting independent critic networks for multiple optimization objectives, each critic network being used to evaluate the state value of the corresponding optimization objective.

2. The method of claim 1, wherein, In training the scheduling decision model, an meta-learning framework is integrated, specifically comprising: sampling multiple different scheduling task environments for meta-training from a task distribution, each scheduling task environment having a different dynamic event distribution; for each sampled scheduling task environment, in an inner loop, interacting with the environment based on the current model parameters, calculating a task-specific loss and performing a first update on the model parameters; in an outer loop, calculating a meta-loss based on the performance of the model parameters after the first update in each task environment, and performing a second update on the original model parameters; through the meta-training, the scheduling decision model obtains initial parameters that can quickly adapt to new scheduling task environments.

3. The method of claim 2, wherein, The meta-learning framework is a model-agnostic meta-learning (MAML) framework, and the meta-loss gradient calculation in the outer loop adopts a first-order approximation (FOMAML) algorithm.

4. The method of claim 1, wherein, The heterogeneous abstraction graph is a heterogeneous abstraction graph with machine competition relationships, which comprises: a set of process nodes, a set of machine nodes, a set of conjunction arcs representing the processing order between processes, a set of selection arcs representing the connection relationship between machines and processable processes, and a set of competition association edges representing the competition relationship between machine nodes that can process the same process.

5. The method of claim 4, wherein, In the first stage, for feature extraction of the machine node, its neighbor nodes include the process node that the machine can currently handle and other machine nodes having a competitive relationship connected through the competitive association edge; and the attention coefficient between machine nodes is calculated based on the features of the two machine nodes and the competitive feature vector representing the competitive relationship.

6. The method of claim 1, wherein, The plurality of optimization objectives include minimizing total completion time, minimizing machine load standard deviation, and minimizing average job delay time; and the number of critic networks is three, each corresponding to one of the three optimization objectives.

7. The method of claim 6, wherein, The reward signal of each step is a triple r t = <r t (1) ,r t (2) ,r t (3) > respectively corresponding to the reward values of the three optimization objectives; the total loss function is: where L t (i)CLIP (θ) and are the policy loss and value function loss based on the ith critic network, respectively, L t E (θ) is the entropy regularizer, and c1, c2, c3 are weight coefficients.

8. A dynamic flexible job shop scheduling apparatus, characterized by, The method comprises: a state acquisition module configured to acquire current state information of the flexible job shop in response to a scheduling event trigger; a scheduling decision module having a pre-trained scheduling decision model embedded therein, configured to receive the current state information and output a scheduling action; the scheduling decision model comprises: a feature extraction network adopting a two-stage feature extraction architecture, configured to perform feature extraction on a heterogeneous disjunction graph composed of process nodes and machine nodes using a graph attention network in a first stage to obtain first feature embeddings of the process nodes and first feature embeddings of the machine nodes; and input the first feature embeddings into a hybrid expert module for processing in a second stage, dynamically select experts through a gating network, and weight and fuse expert outputs to obtain state feature embeddings; an actor network configured to output a scheduling action based on the state feature embeddings; an action execution module configured to issue process-machine pairing instructions output by the scheduling decision module to the shop for execution. The scheduling decision model is trained by a multi-critic PPO algorithm based on multiple critics, and the multi-critic PPO algorithm sets independent critic networks for multiple optimization objectives.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images