Flexible job shop scheduling method based on deep reinforcement learning

Abstract

The invention relates to the field of production scheduling and resource optimization, in particular to a flexible job shop scheduling method based on deep reinforcement learning, which is characterized by comprising the following steps: 1) constructing a flexible shop scheduling environment; (2) a flexible workshop scheduling environment is abstracted to be Disjunction Graph, and the flexible workshop scheduling environment is abstracted to be Disjunction Graph; 3) extracting state features; 4) coding the state features; 5) executing a decision based on the encoded features to obtain scheduling probability distribution; and 6) generating a scheduling plan according to the probability distribution. According to the invention, through multi-level feature extraction and fusion, accurate perception of a production environment is realized, and the accuracy and robustness of a scheduling decision are effectively improved; by utilizing a deep reinforcement learning framework, the dynamic change of the production environment can be adaptively learned, the scheduling strategy is continuously optimized, and the generalization ability is relatively high; and feature representation is enhanced through an attention mechanism, so that the model can still quickly find a global optimal solution when facing a large-scale and high-complexity FJSP problem.

Description

Technical Field

[0001] The present invention relates to the field of production scheduling and resource optimization, and in particular to a flexible job shop scheduling method based on deep reinforcement learning. Background Art

[0002] The Flexible Job Shop Scheduling Problem (FJSP) is a classic production scheduling problem in the manufacturing and logistics fields. The core of the problem is how to reasonably allocate multiple operations to multiple machines with different processing capabilities to optimize key performance indicators, such as minimizing total production time (makespan), reducing total delays, and improving machine utilization. The uniqueness of FJSP lies in its scheduling flexibility. Each operation can be performed on multiple machines, which provides great flexibility for operation sequencing and machine selection, but also increases the complexity of the problem.

[0003] Traditional optimization methods show significant deficiencies when dealing with FJSP, especially when facing large-scale and complex production environments, and it is difficult to meet the requirements of modern manufacturing for high solution quality and real-time response.

[0004] At present, the common FJSP solution methods mainly include two categories: meta-heuristic algorithms and priority dispatch rules (PDRs). Meta-heuristic algorithms (such as genetic algorithms, simulated annealing, particle swarm optimization, ant colony algorithms, etc.) explore the solution space through different strategies to find the optimal solution. Although they have strong flexibility and can perform well in a variety of scheduling environments, their computational complexity is high, especially in large-scale problems. It requires a lot of time cost. On the other hand, PDRs quickly sort and allocate jobs based on simple scheduling rules, such as "first come, first served" (FIFO), "most operations remaining" (MOPNR), "shortest processing time" (SPT), etc. This type of method is widely used in dynamic production environments due to its simple calculation and fast response speed. However, since PDRs only rely on local information for decision-making and lack a global perspective, it is easy to lead to suboptimal solutions, thereby affecting the improvement of scheduling efficiency. Summary of the invention

[0005] This paper proposes a flexible job shop scheduling method based on deep reinforcement learning, which aims to solve the problem of complex interactions between multiple operations and multiple machines in the production scheduling process. This method can dynamically optimize scheduling decisions, improve production efficiency, reduce total delay time, and increase machine utilization, and is particularly suitable for real-time and high-efficiency scheduling requirements in modern manufacturing.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A flexible job shop scheduling method based on deep reinforcement learning includes the following steps:

[0008] 1) Build a flexible workshop scheduling environment;

[0009] 2) Abstract the flexible workshop scheduling environment into a Disjunctive Graph;

[0010] 3) Extract state features;

[0011] 4) Encode state features;

[0012] 5) Execute decisions based on the encoded features to obtain a scheduling probability distribution;

[0013] 6) Generate a scheduling plan based on the probability distribution.

[0014] The step 1) comprises the following steps:

[0015] 1.1) Define the tasks in the flexible workshop and the machines that complete the tasks:

[0016] The task set in the flexible workshop is represented as J = {J 1 ,J 2 ,…,J n}, where each task J i Contains a set of operations to be completed (O i1 ,O i2 ,…,O im ), the machine set is represented as M = {M 1 ,M 2 ,…,M k};

[0017] 1.2) Define each task operation and constraints:

[0018] Each task operation can be executed on multiple machines, but each operation can only be assigned to one machine to complete. For the same task, operation O ij Must be in the previous operation O i,j-1 Each machine can only process one task operation at a time, and cannot process other tasks until the operation is completed.

[0019] The step 2) comprises the following steps:

[0020] 2.1) Construct a graph G = (N, C, D), where the node set N represents the operation of the task and each node corresponds to a specific operation O ij , the directed arc set C represents the order relationship between operations, and the undirected arc set D represents the set of operations that can be processed on the same machine;

[0021] 2.2) Initialize the graph:

[0022] At the beginning of the scheduling process, the graph G contains the operation nodes of all tasks and two types of arcs: the node set N includes all the operation nodes to be executed; the directed arc set C represents the order of operations within each task; the undirected arc set D connects the operations that can be executed on the same machine;

[0023] 2.3) Transform the decision points and states of the graph:

[0024] At each scheduling decision point, one of the undirected arcs is selected and given a direction to determine the order of operations on the machine. The scheduling graph is updated to trigger a state transition, fixing the undirected arc to a directed arc, thereby locking the operation order of the machine. By continuously assigning directions to the undirected arcs, a directed acyclic graph representing the final scheduling plan is eventually generated.

[0025] The step 3) is specifically as follows:

[0026] At time t, the state s t Represented as a set of feature vectors, including the feature vector for each operation The feature vector for each machine and the feature vector for each operation-machine pair in, represents a real number, and the superscript value represents the number of features, so the state vector s t It is expressed as:

[0027]

[0028] Among them, machine characteristics Used to describe the current status of each machine and its related information in scheduling, operation characteristics Used to describe the status of each operation in the scheduling process and its importance in scheduling, operation-machine pair characteristics Used to represent relevant information when an operation is executed on a certain machine.

[0029] The step 4) comprises the following steps:

[0030] 4.1) The state vector s t Input into the CrossNet network to obtain the CrossNet features;

[0031] 4.2) The state vector s t Machine features in Input into the machine attention module to obtain machine features;

[0032] 4.3) The state vector s t The operating characteristics of Input into the machine attention module to obtain the operation features.

[0033] The step 5) comprises the following steps:

[0034] 5.1) Perform average pooling operations on machine features and operation features respectively;

[0035] 5.2) Score the operation-machine pairs based on the features and obtain the probability distribution of the operation-machine pairs:

[0036] 5.3) Weight the operation-machine pair features to obtain a decision enhancement coefficient, and use the decision enhancement coefficient to weight the operation-machine pair probability distribution.

[0037] The step 5.1) is specifically as follows:

[0038] The specific method of average pooling machine features is as follows:

[0039]

[0040] Among them, h M Represents the global characteristics of the machine, M u is the set of all machines relevant to the schedule;

[0041] The specific operation features are averaged pooled as follows:

[0042]

[0043] Among them, h O Represents the global characteristics of the operation, O u It is a collection of all operations related to scheduling.

[0044] The step 5.2) is specifically as follows:

[0045] 5.2.1) Using the machine features, machine global features, operation features, operation global features, and CrossNet features as inputs to the Actor-Critic network to obtain the first operation-machine pair probability distribution;

[0046] 5.2.2) Using the CrossNet features as the input of the Actor-Critic network to obtain the second operation-machine pair probability distribution;

[0047] 5.2.3) Fusion of probability distributions of two operator-machine pairs.

[0048] The step 5.3) comprises the following steps:

[0049] 5.3.1) Machine-operator interaction features After processing, the initial interaction features are mapped to a 32-dimensional hidden layer:

[0050]

[0051] Among them, z 1 represents the hidden layer, W 1 is the weight matrix, b 1 is the bias, SELU is the scaled exponential linear unit activation function;

[0052] 5.3.2) Map the output of the hidden layer to the 1D output layer, and obtain the final probability adjustment value through the Sigmoid function:

[0053] output=σ(W 2 z 1 +b 2 )

[0054] Among them, σ is the Sigmoid activation function, W 2 and b 2 They are weight matrix and bias respectively;

[0055] 5.3.3) Use the decision enhancement weight output to act on the selection probability of each operation-machine pair to obtain the adjusted scalar μ(a t |s t ):

[0056]

[0057] 5.3.4) Convert the scalar to a probability distribution through the Softmax function:

[0058]

[0059] The step 6) comprises the following steps:

[0060] 6.1) Calculate the matching probability between all currently pending operations and available machines, and generate the corresponding probability distribution P(O ij ,M k );

[0061] 6.2) Select the one with the highest current probability in the probability distribution (O ij ,M k ) and assign the operation to the corresponding machine M k ;

[0062] 6.3) Update the scheduling status, including the completion of the operation, the load of the machine, and the list of pending operations;

[0063] 6.4) Repeat steps 6.1) to 6.3) until all operations are assigned to the corresponding machines and the scheduling is completed.

[0064] The present invention has the following beneficial effects and advantages:

[0065] 1. Improve scheduling accuracy: By introducing feature cross-network and decision enhancement modules, the model can accurately capture the complex relationship between operations and machines, thereby significantly improving the accuracy of scheduling decisions and ensuring better decisions in a dynamic production environment.

[0066] 2. Enhance system robustness: This method effectively avoids the excessive reliance of traditional scheduling algorithms on local features, can adaptively adjust feature weights, improve the robustness and stability of the system in different scenarios, and is more adaptable.

[0067] 3. Optimize resource utilization: By dynamically modeling the competitive relationship between machines, the scheduling decision module can optimize the efficiency of machine use and ensure that resources are fully utilized, thereby reducing idle time and resource waste in the production process.

[0068] 4. Adapt to the rapidly changing production environment: The scheduling framework has strong real-time performance and can be dynamically adjusted under rapidly changing production conditions, ensuring that the system can quickly respond to changes in production plans and meet the real-time scheduling requirements of intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is the overall flow chart of the present invention;

[0070] Figure 2 is a directed graph of scheduling status;

[0071] Figure 3 It is a characteristic cross structure diagram;

[0072] Figure 4 is a diagram of the machine & operation information fusion structure;

[0073] Figure 5 Encoding structure diagram for state features;

[0074] Figure 6 Create a structure diagram for decision making. DETAILED DESCRIPTION

[0075] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0076] The main innovation of this invention is to introduce an end-to-end deep reinforcement learning framework, which includes the following parts:

[0077] S10: Flexible workshop scheduling environment;

[0078] S20: state abstraction;

[0079] S30: state feature extraction;

[0080] S40: state feature coding;

[0081] S50: Decision making;

[0082] S60: generating a scheduling plan;

[0083] Furthermore, S10 includes the following parts:

[0084] S11: Definition of Tasks and Machines

[0085] S12: Operations and Constraints

[0086] S20 includes the following parts

[0087] S21: Definition of graph

[0088] S22: Graph initialization

[0089] S23: Decision Points and State Transitions

[0090] S30 includes the following parts

[0091] S31: Machine feature extraction

[0092] S32: Operational feature extraction

[0093] S33: Operation-Machine Feature Extraction

[0094] S40 includes the following parts

[0095] S41: CrossNet structure

[0096] S42: Machine Attention Module

[0097] S43: Operational Attention Module

[0098] S50 includes the following models

[0099] S51: Global Features

[0100] S52: Operation-Machine Scoring

[0101] S53: Decision-making enhancement part

[0102] S60 includes a scheduling plan generation part

[0103] The core part of the present invention is mainly composed of the following modules:

[0104] 1. Feature extraction module

[0105] The core of the present invention is to introduce the CrossNet network as a key module for feature extraction, which is used to capture the complex interaction features between operations and machines. CrossNet transforms and models the operation features, machine features and their interaction features layer by layer through multi-layer cross operations. Compared with traditional networks, CrossNet can not only retain the original feature information, but also generate richer and more complex feature representations by deeply exploring the high-order interaction relationship between different features. This efficient feature modeling method greatly enhances the accuracy and global optimization capabilities of scheduling decisions, and provides more intelligent scheduling support for complex production environments.

[0106] 2. Dual Attention Network (DAN)

[0107] This paper introduces a dual attention network (DAN), which consists of an operation attention module and a machine attention module, which are used to capture the complex relationship between the operation and the machine:

[0108] ■ Operation attention module: This module is used to extract the priority relationship between each operation, and dynamically adjust the information flow between operations by calculating the attention coefficient of the operation and its sequence relationship, thereby affecting the scheduling decision. This module can effectively handle the order constraints of the previous and next operations in the same job, ensuring that the model accurately captures local features while considering global features.

[0109] ■Machine Attention Module: This module is designed to model the competition relationship between machines and dynamically adjust the priority of machines. By calculating the competition intensity vector between machines, this module can determine the relative importance of each machine in the current scheduling environment, thereby optimizing the scheduling plan. By modeling the complex relationship between machines, the machine attention module significantly improves the efficiency of resource utilization.

[0110] 3. Decision Enhancement Module (DAKF)

[0111] The present invention designs a decision augmentation through key features (DAKF) module, which aims to dynamically enhance the contribution of key features in the scheduling decision process. The DAKF module combines the weight adaptive adjustment strategy of key features to identify and amplify the operation-machine features that have the greatest impact on the scheduling results.

[0112] This module not only adapts to the changes in feature importance in different scenarios, but also effectively avoids the over-reliance of traditional methods on local features, thereby improving the robustness and generalization ability of overall scheduling. In a dynamic production environment, the DAKF module performs well and can adjust scheduling strategies in real time to ensure full utilization of important features.

[0113] 4. Scheduling decision module

[0114] The scheduling decision module of the present invention is based on the Actor-Critic architecture in deep reinforcement learning. It uses a deep neural network to model the probability distribution of each operation-machine pair and learns the optimal scheduling strategy by continuously interacting with the environment. Specifically, the Actor network is responsible for generating the selection probability distribution of the operation-machine pair, while the Critic network evaluates the pros and cons of the current strategy and continuously optimizes the scheduling decision through a feedback mechanism.

[0115] This module transforms the Flexible Job Shop Scheduling Problem (FJSP) into a dynamic decision-making process. With the help of the Markov Decision Process (MDP) modeling method, it can update the status in real time during the scheduling process and select the optimal scheduling solution.

[0116] Example

[0117] Figure 1 The overall flow chart of the flexible job shop scheduling method based on deep reinforcement learning is shown, including the following contents:

[0118] Furthermore, S10 is as follows:

[0119] S11: Definition of Tasks and Machines

[0120] The task set in the flexible workshop can be expressed as J = {J 1 ,J 2 ,…,J n}, each task J i Contains a set of operations to be completed (O i1 ,O i2 ,…,O im ). Each operation corresponds to a specific process, and each task has its own specific order requirements, usually needing to be performed on different machines in sequence. The goal of the task is to minimize the production cycle (makespan).

[0121] The machine set is represented as M = {M 1 ,M 2 ,…,M k}. Each machine M j Have different capabilities or resource limits and may only be suitable for certain task operations. In addition, machines can be heterogeneous, meaning that different machines can process the same task at different speeds and efficiencies. Therefore, the scheduling system must take into account factors such as machine availability, processing time, and resource limits.

[0122] S12: Operations and Constraints

[0123] Each task operation can be executed on multiple machines, but each operation can only be assigned to one machine to complete. ij There is a set of executable machines The system needs to decide the best machine to perform the operation.

[0124] The start and end time of an operation are strictly limited. ij Must be in the previous operation O i,j-1 In addition, the execution of all tasks hopes to optimize the completion time as much as possible, and in actual production, the machine switching time, waiting time, etc. need to be considered.

[0125] Each machine can only process one task operation at a time, and cannot process other tasks until the operation is completed. Therefore, the scheduling system must ensure that resources are allocated reasonably to avoid excessive idleness of machines.

[0126] S20 is as follows:

[0127] S21: Definition of graph

[0128] In the flexible shop scheduling problem, the incompatibility graph is used to represent the scheduling dependencies between task operations and the order of operations on the same machine. The graph G = (N, C, D) consists of the following elements:

[0129] Node set N: The node set represents the operation of the task. Each node corresponds to a specific operation O ij , where i represents the task number and j represents the jth operation in the task. In addition, the virtual start node and end node are also included in the node set to uniformly represent the start and end states of the schedule.

[0130] Directed arc set C: Directed arc set is used to connect adjacent operations in the same task, indicating the order relationship between operations. i,j-1 and O ij The directed arcs between indicate that the previous operation must be completed before the next operation can begin. These arcs represent the prerequisite constraints of the task.

[0131] Undirected arc set D: Undirected arc sets represent operations that are connected and executed on the same machine. The directions between these undirected arcs need to be dynamically determined during the scheduling process, representing the choices in the scheduling decision. By giving directions to undirected arcs, the scheduling algorithm can determine the execution order of operations on the same machine.

[0132] S22: Graph initialization

[0133] like Figure 2aAs shown in the figure, at the beginning of the scheduling process, the graph G contains the operation nodes of all tasks and two types of arcs: the node set N includes all the operation nodes to be executed; the directed arc set C represents the order of operations within each task, and these orders are fixed; the undirected arc set D connects the operations that can be executed on the same machine, and the directions of these arcs are determined by the algorithm during the scheduling process.

[0134] S23: Decision Points and State Transitions

[0135] like Figure 2b As shown in the figure, at each scheduling decision point, the scheduling system determines the order of operations on the machine by selecting one of the undirected arcs and assigning it a direction. This operation will update the scheduling graph and trigger a state transition. After the operation is completed, the undirected arc will be fixed as a directed arc, thereby locking the operation order of the machine. By continuously assigning directions to undirected arcs, the scheduling problem is gradually transformed into a decision problem for the direction of the arcs in the graph, and finally a directed acyclic graph (DAG) is generated to represent the final scheduling plan.

[0136] S30 is as follows:

[0137] At time t, the state s t Represented as a set of feature vectors, including the feature vector for each operation The feature vector for each machine and the feature vector for each operation-machine pair Therefore, the state vector s t It is expressed as:

[0138]

[0139] S31: Machine feature extraction

[0140] Machine Features Used to describe the current status of each machine and its related information in scheduling, including:

[0141] ●Number of available workpieces on the current machine: indicates the number of available workpieces on the current machine, which is used to measure the load capacity of the machine.

[0142] ●Number of available operations on the current machine: Indicates the number of operations that can be executed on the current machine, reflecting the scheduling potential of the machine.

[0143] ●Minimum machine processing time: refers to the minimum processing time among all executable operations on the current machine, indicating the processing efficiency of the machine.

[0144] ●Average machine processing time: refers to the average processing time of all executable operations on the current machine, which represents the average processing capacity of the machine.

[0145] ●Machine waiting time: refers to the waiting time that the machine is idle since the last task was executed, which is used to reflect the utilization rate of the machine.

[0146] ●Machine remaining workload: indicates the total workload of tasks that have not been completed by the current machine, which is used to evaluate its current task load.

[0147] ●Machine idle time: indicates when the machine will become idle, which is used to determine its scheduling opportunity.

[0148] ●Machine working status flag: used to mark whether the current machine is in working status to optimize resource allocation.

[0149] ●Machine idle time: indicates when the machine will become idle, which is used to determine its scheduling opportunity.

[0150] ●Machine working status flag: used to mark whether the current machine is in working status to optimize resource allocation.

[0151] S32: Operational feature extraction

[0152] Operating characteristics It is used to describe the status of each operation in the scheduling process and its importance in the scheduling, including:

[0153] ●Operation scheduling flag: used to mark whether the operation has been scheduled and indicate whether the operation has entered the production plan.

[0154] ●Lower bound of the earliest completion time of the operation: refers to the earliest possible completion time of the operation under the current scheduling conditions, which is used to optimize the scheduling order.

[0155] ●Minimum processing time of operation: indicates the minimum processing time required for the operation on the selectable machine, reflecting its processing efficiency.

[0156] ●Processing time span: refers to the difference in processing time of the operation on different machines, indicating the time span of the operation on different resources.

[0157] ●Average processing time of operation: refers to the average processing time of the operation on all optional machines, which represents its average execution time.

[0158] ●Operation waiting time: refers to the waiting time from the generation of the operation to the current time, reflecting the urgency of its scheduling.

[0159] ●Operation remaining workload: Indicates the amount of work that has not been completed in the workpiece to which the operation belongs, and is used to measure its importance in the workpiece task.

[0160] ●Number of remaining operations for a workpiece: refers to the number of unfinished operations remaining in the workpiece, indicating the completion progress of the workpiece.

[0161] ●Remaining workload of the workpiece: indicates the total remaining workload of the workpiece, reflecting the completion status of the workpiece.

[0162] ●Number of optional machines for the operation: refers to the number of machines on which the operation can be performed, indicating the resource selection range of the operation.

[0163] S33: Operation-Machine Feature Extraction

[0164] Operation-Machine Pair Characteristics Used to indicate the relevant information when an operation is executed on a machine, including:

[0165] ●Processing time of candidate operation: indicates the processing time of the current candidate operation on the machine.

[0166] ●The ratio of candidate operation to maximum processing time: This ratio is the ratio of the processing time of the candidate operation to the maximum processing time of all selected operations, reflecting the relative complexity of the operation.

[0167] ●The ratio of the candidate operation to the maximum processing time on the machine: This ratio is the ratio of the processing time of the candidate operation on the machine to the maximum processing time of the machine, which is used to measure the impact of the operation on the machine.

[0168] ●The ratio of the processing time of the candidate operation to the remaining maximum operation: This ratio is the ratio of the processing time of the candidate operation to the maximum processing time of all remaining operations, indicating the importance of the operation.

[0169] ●The ratio of the candidate operation processing time to the maximum remaining operation of the machine: This ratio is the ratio of the processing time of the candidate operation on the machine to the maximum processing time of the remaining operations of the machine.

[0170] ●The ratio of the maximum processing time of the candidate operation to the operation-machine pair: This ratio is the maximum processing time ratio of the candidate operation among all operation-machine pairs, which represents the efficiency of the combination.

[0171] ●The ratio of candidate operation to the remaining workload of the workpiece: This ratio is the ratio of the processing time of the candidate operation to the total remaining workload of the workpiece, which is used to measure the priority of the operation within the workpiece.

[0172] ● Waiting time of an operation-machine pair: indicates the waiting time of the candidate operation on the machine, reflecting the scheduling urgency of the operation.

[0173] S40 is as follows:

[0174] The overall process is as follows Figure 5 As shown, in the state feature encoding module, operation features, machine features and operation-

[0175] The machine pair features are input into a multi-layer neural network for deep encoding. Specifically, the encoding process includes the following parts:

[0176] S41: CrossNet structure

[0177] Figure 3 The specific structure of the feature extraction and enhancement module is shown, which mainly includes the architecture of the CrossNet network and the feature interaction mechanism of Cross Expert. The main function of this module is to deeply extract high-order features of operations, machines and their combinations through the Cross Expert network to enhance the feature input quality of scheduling decisions.

[0178] The CrossNet network consists of multiple layers of Cross Experts, which are responsible for mining complex nonlinear relationships from input features. The Cross Expert module of each layer crosses the input features, generates new feature representations, and passes them to the next layer layer by layer, forming a mechanism for deep feature interaction. The input of CrossNet is the features that have been initially embedded, and the output is the enhanced features after multiple layers of feature interaction, called "Cross Feature".

[0179] Cross Expert is the core unit of CrossNet. Figure 3 As shown in the upper part of , its main function is to perform feature crossover and feature enhancement. For the input feature vector The output formula of the cross layer l is:

[0180] x l+1 =x 0 ⊙(W l x l +b l )+x l

[0181] Where: W l is the learnable weight matrix, b l is the bias vector and ⊙ represents element-wise multiplication.

[0182] This formula is stacked layer by layer to capture high-order interactions between different features. In this way, CrossNet can integrate machine, operation and their interaction features to generate rich feature interaction information.

[0183] Gate mechanism: CrossNet introduces a gate mechanism between layers to control the flow direction and strength of feature interactions. Through Gate control, the network can adaptively adjust the depth and complexity of feature interactions to avoid the negative impact of redundant features on scheduling decisions.

[0184] Assume there are K expert networks E 1 (x),E 2 (x),…,E K (x), each expert learns and extracts feature interactions on its subspace. The gating mechanism G(x) dynamically assigns weights G to each expert i (x), these weights are normalized by the Softmax function:

[0185]

[0186] Each weight G i The calculation formula for (x) is:

[0187]

[0188] in, is the weight matrix corresponding to the i-th expert, is the bias term.

[0189] The final output h cross is the weighted sum of all expert network outputs, and the formula is:

[0190]

[0191] This dynamic combination mechanism can adaptively learn in different feature subspaces and improve the accuracy and robustness of scheduling decisions.

[0192] S42: Machine Attention Module

[0193] Figure 4a The structure of the machine attention module is shown:

[0194] The machine attention module aims to model the dynamic competition relationship between machines. k , and other competing machines M q The competitive relationship between M and M is modeled by calculating the attention coefficient. First, define the machine M k and M q The competitive relationship between kq , the calculation formula of the attention coefficient is:

[0195]

[0196] Where: b is the learnable attention weight vector, Z 1 and Z 2 is the linear transformation matrix, and They are machine M k and M q The eigenvector ofkq is the competition intensity vector, representing the machine M k and M q The degree of competition between them is obtained by summing the features of the competition operation.

[0197] In order to generate the normalized attention weight, the competitive relationship coefficient u kq Normalized by Softmax function, we get the attention weight α kq :

[0198]

[0199] Here, α kq Indicates machine M k Its competitor machine M q The attention coefficient, N k Represents the same machine M k A collection of all competing machines.

[0200] Next, the normalized attention weight α kq Applied to competitive machines M q The characteristics of the generated machine M k The output feature vector Calculated by nonlinear activation function σ:

[0201]

[0202] The machine attention module affects the priority of each machine by dynamically adjusting the relationship between machines, especially based on their competition intensity. The machine output features generated by this module can optimize resource allocation in the scheduling decision process and improve the overall efficiency of scheduling.

[0203] S43: Operational Attention Module

[0204] Figure 4b The structure of the operation attention module is shown:

[0205] In order to capture the priority constraints between operations, the operation attention module calculates the current operation O ij With its predecessor O i,j-1 And subsequent operations i,j+1 The calculation formula of the attention coefficient is:

[0206]

[0207] Where: a is the learnable attention weight vector, W is the linear transformation matrix, and Operation O ij and Operation O ip The characteristic vector of

[0208] In order to adapt to the dynamic relationship between operations, dynamic masks are used for non-existent preceding or subsequent operations, thereby ignoring their influence. The calculated attention coefficient e i,j,p Normalized by Softmax function, we get the attention weight α i,j,p :

[0209]

[0210] This formula ensures that the operation O ij The dependency on its predecessor and successor operations is expressed by weight α i,j,p The attention weights are adjusted dynamically and normalized to ensure that the sum of the weights is 1. The normalized attention weights are used to weight the input features of adjacent operations to generate the operation O ij The output feature vector of the operation is calculated by the nonlinear activation function σ:

[0211]

[0212] By operating the attention module, the current operation O ij The features of can be propagated through layers, so that the information of each operation not only contains its own features, but also integrates the features of its predecessor and subsequent operations. This mechanism enables the dependencies between operations to be explicitly modeled during the scheduling process, thereby improving the accuracy and robustness of the decision-making process.

[0213] S50 is as follows:

[0214] The overall process is as follows Figure 6 As shown, it mainly includes operation-machine scoring and decision enhancement

[0215] S51: Global Features

[0216] After the dual attention network has been processed by multiple layers, the pooling operation is used to summarize the features of all operations and machines. First, average pooling is performed on the operation features and machine features respectively. The average pooling formula for the operation features is:

[0217]

[0218] h O Represents the global feature representation of the operation, O u It is a collection of all operations related to scheduling.

[0219] Similarly, the average pooling formula for machine features is:

[0220]

[0221] hM Represents the global feature representation of the machine, M u Is the set of all machines relevant to the schedule.

[0222] S52: Operation-Machine Scoring

[0223] The Actor-Critic architecture of deep neural networks is used to generate the selection probability of each operation-machine pair.

[0224] The Actor network is responsible for generating the decision distribution π θ (a t |s t ), that is, in a given state s t Next, for the candidate operation-machine pair a t =(O ij ,M k ). First, the Actor network generates a scalar μ(a t |s t ), and then convert it into the probability of selecting the pair through the Softmax function:

[0225]

[0226] Among them, ‖ represents the connection operation of the feature, MLP θ is a multilayer perceptron that processes input features and generates a scalar of selection probabilities.

[0227] The scalar is converted into a probability distribution through the Softmax function:

[0228]

[0229] This probability distribution is used to select the best operator-machine pair a at time step t. t .

[0230] The Critic network is responsible for evaluating the quality of the current decision and generating a value function V φ (s t ) to represent a given state s t The value of:

[0231] V φ (s t )=MLP φ (h G )

[0232] Among them, MLP φ is a multi-layer perceptron used to evaluate the value of a state.

[0233] By continuously interacting with the environment, the network is able to self-learn and optimize its decision-making strategies.

[0234] S53: Decision-making enhancement part

[0235] This module weights the operation-machine pair features to obtain a decision enhancement coefficient, and uses the decision enhancement coefficient to weight the generated operation-machine pair probability distribution.

[0236] First, the machine-operator interaction features are analyzed through the decision enhancement module. The initial interaction features are mapped to a 32-dimensional hidden layer using the formula:

[0237]

[0238] Among them, W 1 is the weight matrix, b 1 is the bias, and SELU is the scaled exponential linear unit activation function.

[0239] Next, the output of the hidden layer is mapped to the 1D output layer, and the final probability adjustment value is obtained through the Sigmoid function:

[0240] output=σ(W 2 z 1 +b 2 )

[0241] Among them, σ is the Sigmoid activation function, W 2 and b 2 are the weight matrix and bias respectively.

[0242] The adjustment value output of the decision enhancement module is a scalar in the range of (0,1), which acts on the selection probability of each operation-machine pair, thereby enhancing the accuracy of the decision. t |s t ) is defined as:

[0243]

[0244] The scalar is converted into a probability distribution through the Softmax function:

[0245]

[0246] Among them, output is the decision enhancement weight. The module will adaptively adjust the weight of the feature according to different environmental conditions to ensure that key features can play a greater role in the scheduling process.

[0247] S60: Scheduling plan generation

[0248] At each scheduling moment, the system selects the operation-machine pair with the highest probability from the probability distribution calculated under the current state. ij ,M k ). This process is carried out dynamically until all task operations are assigned. The specific steps are as follows:

[0249] Step 1: The system first calculates the matching probability between all currently pending operations and available machines.

[0250] Generate the corresponding probability distribution P(O ij ,M k );

[0251] Step 2: Select the one with the highest current probability in the probability distribution (O ij ,M k ) and assign the operation to the corresponding machine M k ;

[0252] Step 3: Update the scheduling status, including the completion of the operation, the load of the machine, and the list of pending operations;

[0253] Step 4: Repeat steps 1-3 until all operations are assigned to the corresponding machines and the scheduling is completed.

[0254] Through this selection mechanism, the system can make the best decision based on the latest status and probability distribution at each moment and dynamically update the scheduling plan.

[0255] The scheduling effect of the present invention is compared with the four most commonly used existing scheduling rules, including the following scheduling rules:

[0256] 1. First Come First Served (FIFO): FIFO is a simple and classic scheduling rule that selects the earliest available operation and assigns it to the first compatible machine for scheduling. This method prioritizes jobs that are ready for scheduling and ensures that the scheduling process proceeds in order.

[0257] 2. Most Operations Remaining (MOPNR): In the MOPNR rule, the candidate operation with the most remaining subsequent operations is prioritized and the machine that can process the operation immediately is assigned. This method is designed to speed up the overall completion of complex jobs with multiple subsequent operations.

[0258] 3. Shortest Processing Time (SPT): The SPT rule selects the operation-machine pair with the shortest processing time for scheduling. By giving priority to shorter tasks, this rule effectively reduces the number of unfinished jobs in the system, thereby optimizing overall scheduling efficiency.

[0259] 4. Most Work to Remain (MWKR): The MWKR rule selects the candidate operation with the largest remaining average processing time and assigns it to a machine that can process it immediately. This rule focuses on jobs with large loads and aims to reduce their impact on the overall system scheduling and ensure the balance of scheduling.

[0260] The test results on 8 different test sets are shown in the following table, where the performance of the SD1 dataset is mainly reflected in the scheduling effect of the model in small-scale and medium-scale FJSP instances, including indicators such as average completion time, relative error, and calculation time. The SD2 dataset examines the performance of the model on larger-scale problems, reflecting the robustness of the model in processing job scheduling with different processing time ranges.

[0261]

[0262] When compared with traditional priority scheduling rules, our method shows obvious advantages. Although traditional scheduling methods such as FIFO, MWKR and SPT usually have shorter computation time, their performance decreases significantly as the complexity of the problem increases. For example, in the 20×10 instances of the SD2 dataset, the maximum completion times of FIFO and MWKR are 1088.05 and 1040.69 respectively, while our method is only 550.11, showing significant superiority in handling large-scale complex scheduling tasks.

[0263] We compare the proposed method with existing deep reinforcement learning methods, focusing on evaluating the scheduling performance at different instance sizes. The two methods are based on Graph Neural Network (GNN) and deep reinforcement learning proposed by Song et al., and based on Dual Attention Network (DAN) and deep reinforcement learning proposed by Wang et al.

[0264] The method based on graph neural network and deep reinforcement learning proposed by Song et al. models the flexible job shop scheduling problem (FJSP) as a Markov decision process (MDP) and uses graph neural network (GNN) to capture the complex relationship between operations and machines. This method represents operations and machines as nodes in a heterogeneous graph and uses reinforcement learning to optimize the scheduling strategy. Test results on multiple benchmark datasets show that this method can achieve better results than traditional priority scheduling rules on specific tasks.

[0265] The method based on dual attention network and deep reinforcement learning proposed by Wang et al. uses a dual attention mechanism to extract deep features from the perspectives of operation and machine respectively, so as to more accurately capture the complex dependencies between them.

[0266] The optimal algorithm is marked in bold in the table. By comparing with the above scheduling methods, the present invention has achieved the best results in almost all examples. Its effectiveness and advantages in the flexible job shop scheduling problem have been verified, and it can better cope with complex production environments and resource allocation problems.

[0267] In order to verify the generalization ability of the proposed method, the model is first trained on a simulation dataset to fully explore the relationship between features and optimize the scheduling strategy. Subsequently, the model is tested on multiple public datasets to verify its generalization ability and effectiveness in practical applications. The public datasets cover different types of scheduling scenarios, including a variety of industrial environments and production tasks, allowing the model to be evaluated under complex conditions in the real world. The test results are shown in the table, showing the comparison between the scheduling method of the present invention and the traditional rule-based method:

[0268]

[0269] From the test results, it can be seen that the scheduling method proposed in this invention shows significant performance advantages on various data sets, especially in minimizing the total production time. This result not only verifies the learning ability of the model, but also highlights its superior generalization ability on different data sets. The knowledge and experience acquired by the model during the training process can be successfully transferred to unseen test data, which reflects its adaptability to complex scheduling environments and its strong practical value.

[0270] From the above comparison, it can be seen that our method not only outperforms traditional and deep reinforcement learning methods in performance, but also exhibits strong cross-distribution adaptability, which makes it have wide application potential in actual complex scheduling tasks.

Claims

1. A flexible job shop scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: 1) Build a flexible workshop scheduling environment; 2) Abstract the flexible workshop scheduling environment into a Disjunctive Graph; 3) Extract state features; 4) Encode state features; 5) Execute decisions based on the encoded features to obtain a scheduling probability distribution; 6) Generate a scheduling plan based on the probability distribution.

2. A flexible job shop scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The step 1) comprises the following steps: 1.1) Define the tasks in the flexible workshop and the machines that complete the tasks: The task set in the flexible workshop is represented as J = {J1, J2, ..., J n }, where each task J i Contains a set of operations to be completed (O i1 , O i2 , ..., O im ), the machine set is represented as M = {M1, M2, ..., M k }; 1.2) Define each task operation and constraints: Each task operation can be executed on multiple machines, but each operation can only be assigned to one machine to complete. For the same task, operation O ij Must be in the previous operation O i，j-1 Each machine can only process one task operation at a time, and cannot process other tasks until the operation is completed.

3. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step 2) comprises the following steps: 2.1) Construct a graph G = (N, C, D), where the node set N represents the operation of the task and each node corresponds to a specific operation O ij , the directed arc set C represents the order relationship between operations, and the undirected arc set D represents the set of operations that can be processed on the same machine; 2.2) Initialize the graph: At the beginning of the scheduling process, the graph G contains the operation nodes of all tasks and two types of arcs: the node set N includes all the operation nodes to be executed; the directed arc set C represents the order of operations within each task; the undirected arc set D connects the operations that can be executed on the same machine; 2.3) Transform the decision points and states of the graph: At each scheduling decision point, one of the undirected arcs is selected and given a direction to determine the order of operations on the machine. The scheduling graph is updated to trigger a state transition, fixing the undirected arc to a directed arc, thereby locking the operation order of the machine. By continuously assigning directions to the undirected arcs, a directed acyclic graph representing the final scheduling plan is eventually generated.

4. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step 3) is specifically as follows: At time t, the state s t Represented as a set of feature vectors, including the feature vector for each operation The feature vector for each machine and the feature vector for each operation-machine pair in, represents a real number, and the superscript value represents the number of features, so the state vector s t It is expressed as: Among them, machine characteristics Used to describe the current status of each machine and its related information in scheduling, operation characteristics Used to describe the status of each operation in the scheduling process and its importance in scheduling, operation-machine pair characteristics Used to represent relevant information when an operation is executed on a certain machine.

5. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step 4) comprises the following steps: 4.1) The state vector s t Input into the CrossNet network to obtain the CrossNet features; 4.2) The state vector s t Machine features in Input into the machine attention module to obtain machine features; 4.3) The state vector s t The operating characteristics of Input into the machine attention module to obtain the operation features.

6. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step 5) comprises the following steps: 5.1) Perform average pooling operations on machine features and operation features respectively; 5.2) Score the operation-machine pairs based on the features and obtain the probability distribution of the operation-machine pairs: 5.3) Weight the operation-machine pair features to obtain a decision enhancement coefficient, and use the decision enhancement coefficient to weight the operation-machine pair probability distribution.

7. A flexible job shop scheduling method based on deep reinforcement learning according to claim 6, characterized in that: The step 5.1) is specifically as follows: The specific method of average pooling machine features is as follows: Among them, h M Represents the global characteristics of the machine, M u is the set of all machines relevant to the schedule; The specific operation features are averaged pooled as follows: Among them, h O Represents the global characteristics of the operation, O u It is a collection of all operations related to scheduling.

8. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 6, characterized in that: The step 5.2) is specifically as follows: 5.2.1) Using the machine features, machine global features, operation features, operation global features, and CrossNet features as inputs to the Actor-Critic network to obtain a first operation-machine pair probability distribution; 5.2.2) Using the CrossNet features as the input of the Actor-Critic network to obtain the second operation-machine pair probability distribution; 5.2.3) Fusion of probability distributions of two operator-machine pairs.

9. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 6, characterized in that: Step 5.3) The following steps are involved: 5.3.1) Machine-operator interaction features After processing, the initial interaction features are mapped to a 32-dimensional hidden layer: in, z1 represents the hidden layer, W1 is the weight matrix, b1 is the bias, and SELU is the scaled exponential linear unit activation function; 5.3.2) Map the output of the hidden layer to the 1D output layer, and obtain the final probability adjustment value through the Sigmoid function: output=σ(W2z1+b2) Among them, σ is the Sigmoid activation function, W2 and b2 are the weight matrix and bias respectively; 5.3.3) Use the decision enhancement weight output to act on the selection probability of each operation-machine pair to obtain the adjusted scalar μ(a t |s t ): 5.3.4) Convert the scalar to a probability distribution through the Softmax function:

10. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step 6) comprises the following steps: 6.1) Calculate the matching probability between all currently pending operations and available machines, and generate the corresponding probability distribution P(O ij , M k ); 6.2) Select the one with the highest current probability in the probability distribution (O ij , M k ) and assign the operation to the corresponding machine M k ; 6.3) Update the scheduling status, including the completion of the operation, the load of the machine, and the list of pending operations; 6.4) Repeat steps 6.1) to 6.3) until all operations are assigned to the corresponding machines and the scheduling is completed.

Images