Flexible workshop scheduling optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of industrial manufacturing, and discloses a flexible workshop scheduling optimization method based on deep reinforcement learning, and the method comprises the following steps: constructing a graph neural network model, and carrying out the modeling of a workshop operation process of a flexible workshop scheduling problem; the flexible job-shop scheduling problem is converted into a Markov decision process, a deep reinforcement learning algorithm is used for dynamic decision making, and the Markov decision process comprises a state space, an action space, a reward function and a state transfer function; and restraining the action space of the deep reinforcement learning algorithm in combination with a heuristic scheduling rule. The flexible workshop scheduling is optimized through the deep reinforcement learning algorithm, and the scheduling strategy can be dynamically adjusted in real time according to the actual production state and task requirements of the workshop. Compared with a traditional scheduling method, the method does not need to preset a fixed scheduling rule, autonomously learns and decides through the intelligent agent, and adapts to a complex and changeable production environment.

Description

Technical Field

[0001] This invention relates to the field of industrial manufacturing technology, specifically to a flexible workshop scheduling optimization method based on deep reinforcement learning. Background Technology

[0002] Flexible shop floor scheduling is a highly challenging optimization problem in modern manufacturing. Its goal is to optimize production efficiency, reduce costs, and shorten delivery times by rationally arranging processes and machines under various production constraints. Traditional scheduling methods are mainly divided into exact methods and heuristic methods. Exact methods, such as mixed integer programming, can find theoretically optimal solutions, but their computational complexity is extremely high and the solution time is too long when dealing with large-scale problems, making it difficult to meet the real-time requirements of actual production. Heuristic methods, including genetic algorithms and simulated annealing algorithms, have certain advantages in solution speed, but they often rely on manually designed heuristic rules, making them prone to getting trapped in local optima. Furthermore, they lack sufficient flexibility and adaptability when facing dynamic changes in the shop floor production environment (such as equipment failures and urgent order insertions).

[0003] In recent years, deep reinforcement learning, as an emerging intelligent optimization technology, has been increasingly applied to the field of flexible shop floor scheduling. Through interactive learning between agents and the environment, deep reinforcement learning can dynamically adjust scheduling strategies to adapt to complex changes in the production environment. However, existing deep reinforcement learning-based scheduling methods still have some shortcomings. On the one hand, most deep reinforcement learning models require significant computational resources and time during training, and their generalization ability is limited when facing scheduling problems of different scales and types, making it difficult to quickly adapt to new production scenarios. On the other hand, existing methods often struggle to effectively balance multiple optimization objectives when dealing with multi-objective optimization problems, resulting in poor overall performance of the generated scheduling schemes in practical applications. Furthermore, traditional methods rely heavily on human experience, and human factors have a significant impact on scheduling decisions, which to some extent limits the scientific validity and accuracy of the scheduling schemes. Therefore, designing a flexible shop floor scheduling optimization method that can quickly respond to dynamic changes in the shop floor, efficiently solve large-scale problems, possess good generalization ability, and effectively balance multi-objective optimization is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a flexible shop floor scheduling optimization method based on deep reinforcement learning, which solves the deficiencies of existing flexible shop floor scheduling methods in terms of dynamic environment adaptability, multi-objective optimization balance, and efficiency in solving large-scale problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a flexible shop floor scheduling optimization method based on deep reinforcement learning, comprising the following steps:

[0006] A graph neural network model is constructed to model the workshop operation process of the flexible job shop scheduling problem;

[0007] The flexible job shop scheduling problem is transformed into a Markov decision process, and a deep reinforcement learning algorithm is used for dynamic decision-making. The Markov decision process includes a state space, an action space, a reward function, and a state transition function.

[0008] Heuristic scheduling rules are used to constrain the action space of the deep reinforcement learning algorithm to improve the solution efficiency.

[0009] We construct a diverse set of scheduling strategies, obtain more diverse datasets through Bayesian optimization, and introduce the K-nearest neighbor algorithm to select the most suitable strategy, thereby enhancing the model's generalization ability and solution quality.

[0010] Preferably, the construction of the graph neural network model specifically includes:

[0011] Define a directed graph, including a set of nodes and a set of edges. The set of nodes includes a set of processes, a set of machines, and a set of jobs. The set of edges includes undirected edges between machines and processes, directed edges between processes and jobs, directed edges between processes, directed edges between machines and jobs, directed edges between jobs, and directed edges between machines.

[0012] A graph attention network is used to process the directed graph to extract the feature information of nodes and edges, so as to realize the modeling of the workshop operation process.

[0013] Preferably, the transformation of the flexible job shop scheduling problem into a Markov decision process specifically includes:

[0014] State space: Each state is represented as a heterogeneous graph, including a set of processes, a set of machines, and a set of jobs, as well as the set of edges between them;

[0015] Action space: At each time step, the action set consists of all compatible job and machine pairs;

[0016] State transition: After an action is performed in a state, the graph will transition to a new state, where the operation has been executed and the heterogeneous graph will undergo the corresponding transformation.

[0017] Reward function: Defined as the difference between the completion time of the current state and the completion time of the next state, in order to minimize the completion time.

[0018] Preferably, the constraint of the action space by combining heuristic scheduling rules specifically includes:

[0019] Only the k earliest starting operations can be selected. If multiple operations have the same start time, all of them can be selected.

[0020] Only the k earliest operations that can be completed are allowed to be selected. By doing so, the agent can reduce the ineffective actions in the early exploration process, thereby improving the overall scheduling efficiency.

[0021] Preferably, the construction of a diverse set of scheduling strategies specifically includes:

[0022] Define a search space to determine the variables to be optimized, including the graph neural network structure, deep reinforcement learning algorithm parameters, scheduling rule types, or relevant parameters in the instance generation process;

[0023] Generate a validation set and calculate the optimal or near-optimal maximum completion time for each instance using an exact method;

[0024] The search space is sampled using Bayesian optimization to generate models with different configurations;

[0025] The K-nearest neighbors algorithm is used to cluster and filter the strategies, resulting in a small but high-performance set of diverse strategies.

[0026] Preferably, the deep reinforcement learning algorithm employs a proximal policy advantage algorithm, which includes generating a policy network and a value network, both of which are trained simultaneously using embedding representations computed by a graph neural network.

[0027] Preferably, the method further includes:

[0028] During the training phase, a set of differentiated policies is generated, and a robust and adaptable policy library is gradually built through iterative optimization.

[0029] During the inference phase, the strategy is applied to new instances to verify its effectiveness. This can be done by generating a unique solution based on a single strategy or by calling a subset of representative strategies in parallel to generate multiple solutions and selecting the best one.

[0030] A flexible shop floor scheduling optimization system based on deep reinforcement learning includes:

[0031] The graph neural network modeling module is used to model the workshop operation process for the flexible job shop scheduling problem;

[0032] The Markov Decision Process Transformation Module is used to transform the flexible job shop scheduling problem into a Markov decision process.

[0033] The deep reinforcement learning decision-making module is used to make dynamic decisions using deep reinforcement learning algorithms.

[0034] The action space constraint module is used to constrain the action space of deep reinforcement learning algorithms by combining heuristic scheduling rules;

[0035] The diversified strategy building module is used to construct a diverse set of scheduling strategies. It obtains a more diverse dataset through Bayesian optimization and introduces the K-nearest neighbor algorithm to select the most suitable strategy.

[0036] Preferably, the graph neural network modeling module uses a graph attention network to model the workshop operation process.

[0037] Preferably, the deep reinforcement learning decision module employs a proximate policy advantage algorithm, which includes generating a policy network and a value network, both of which are trained simultaneously using embedding representations computed by a graph neural network.

[0038] This invention provides a flexible shop floor scheduling optimization method based on deep reinforcement learning. It has the following beneficial effects:

[0039] 1. This invention optimizes flexible shop floor scheduling using deep reinforcement learning algorithms, dynamically adjusting scheduling strategies in real time based on the actual production status and task requirements of the shop floor. Compared to traditional scheduling methods, this method does not require pre-setting fixed scheduling rules; instead, it adapts to complex and ever-changing production environments through autonomous learning and decision-making by intelligent agents. This makes the scheduling scheme more flexible, effectively handling unexpected situations such as urgent order insertions and equipment failures, significantly improving shop floor production efficiency and resource utilization, reducing production costs, and bringing significant economic benefits to enterprises.

[0040] 2. This invention utilizes the adaptive learning capability of deep reinforcement learning algorithms to automatically extract valuable information from historical scheduling data and continuously optimize scheduling strategies. This data-driven optimization method avoids the over-reliance on human experience in traditional methods, reducing the impact of human factors on scheduling decisions. Simultaneously, this method can quickly respond to changes in workshop production conditions, such as equipment status updates and task priority adjustments, generating optimal or near-optimal scheduling schemes. This not only improves the scientific rigor and accuracy of scheduling decisions but also significantly shortens the time required to formulate scheduling plans, thereby enhancing the enterprise's production management level and market competitiveness. Attached Figure Description

[0041] Figure 1 This is an overall flowchart of the present invention;

[0042] Figure 2 This is a state transition flowchart of the present invention;

[0043] Figure 3 This is a diagram of the training method architecture of the present invention. Detailed Implementation

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Please see the appendix Figure 1 - Appendix Figure 3 This invention provides a flexible shop floor scheduling optimization method based on deep reinforcement learning, comprising the following steps:

[0046] A graph neural network model is constructed to model the workshop operation process of the flexible job shop scheduling problem;

[0047] Specifically, Graph Neural Networks (GNNs) have garnered significant attention over the past five years due to their ability to efficiently extract information from graph-structured data. GNNs can perform information transfer and prediction based on the edges between nodes. A directed graph G is defined as a tuple (V, E) of two sets, where V is the set of nodes and E is the set of edges. (i,j)∈E represents a directed edge from node i to node j. It is typically assumed that each node has an initial representation. Where d0 is the dimension of the initial features of the node. All edges are considered directed. Various GNN architectures have been proposed, one of the more commonly used being the Graph Attention Network (GAT). In GAT, each node updates its own representation by aggregating the representations of its neighbors. Specifically, let the representations of all nodes be a set H = {h1, h2, ..., h...}. n}, a certain node h i The representation is updated to h′ i The update method is as follows:

[0048] h i =f θ (h i ,AGGREGATE({h j |j∈N i}))

[0049] Where, N iLet f represent the set of nodes connected to node i, and AGGREGATE be the functions defining the GNN architecture. Many traditional GNN methods assign equal importance to all neighboring nodes, but in the original GNN model, the authors use a weighted average to assign importance to neighboring nodes N. i The representations are aggregated. This method learns the importance of node j to the features of node i through a scoring function. This scoring function is defined as follows:

[0050] e(h i ,h j =LeakyReLU(d T ·[Wh i ‖Wh j ])

[0051] in, and W∈Rd n ×d0 is the vector parameter that needs to be learned, d ′ h is a node representation vector i ′ The dimension is denoted by , and the symbol ‖ denotes the vector concatenation operation. These scores are then normalized using the softmax function across all neighbors of the node, resulting in the following:

[0052]

[0053] Finally, in order to obtain node h i The representations of neighboring nodes are weighted and summed according to their attention coefficients, and then processed by a non-linear activation function σ, as shown in the following formula:

[0054]

[0055] Recent research has shown that improved graph attention networks (GATs) perform better by reordering the internal operations within the attention mechanism, resulting in a more expressive attention mechanism. The original GAT suffers from the problem that the weight vector w and attention weight vector α are applied consecutively, causing them to mathematically degenerate into a linear transformation. To address this, GATv2 first applies a non-linear function to the concatenated features before using the attention vector α for scoring, thus generating a more discriminative score for each "node key" pair. The updated scoring formula is shown below:

[0056] e(h i ,h j ) = a T LeakyReLU(W·[h i ||h j ])

[0057] The weight matrix W is divided into two parts, one part is used to process the features of the target node (W). i One part (W2) is used to process the features of neighboring nodes, allowing for the consideration of their influence separately in the attention mechanism, thus enabling more flexible learning of the relationships between nodes. To this end, this invention will consider embedding the shop floor scheduling state using the GATv2 network.

[0058] Building a graph neural network model specifically includes:

[0059] Define a directed graph, including a set of nodes and a set of edges. The set of nodes includes a set of processes, a set of machines, and a set of jobs. The set of edges includes undirected edges between machines and processes, directed edges between processes and jobs, directed edges between processes, directed edges between machines and jobs, directed edges between jobs, and directed edges between machines.

[0060] A graph attention network is used to process the directed graph and extract the feature information of nodes and edges to model the workshop operation process.

[0061] Specifically, to extract the state information used to generate the policy, we employ a heterogeneous graph neural network based on the GATv2 attention mechanism. This allows us to construct a GNN architecture with more powerful and dynamic attention capabilities without significantly increasing computational cost. This capability is crucial for problems like the Flexible Job Shop Scheduling Problem (FJSSP). For each type of node, the initial representation is as follows: a job node is represented as h. ji ∈R dj The operation node is represented as Machine nodes are represented as Where d j \d o d M These represent the initial feature dimensions of the operation, task, and machine, respectively.

[0062] Operation embedding: Each operation node receives information from its subsequent operations and the machine capable of processing that operation. Therefore, two attention weights must be computed. For each operation that satisfies... and The operation node, its preceding operation o ij and subsequent operations o ij+1 Weights between Calculated as:

[0063]

[0064] in and It is aimed at satisfying The linear transformation parameters d′ obtained through operational learning oThis is the dimension of the operation node embedding representation. To calculate the attention relationship between the machine and the operation, considering the features of these edges, we utilize... Indicates operation o ij and machines The edges between, and This is the initial representation of the edge feature. Operation node o ij and machine nodes The weights between them are calculated as follows:

[0065]

[0066] in and These are the learnable attention parameter vector, the operator node weight matrix, the machine node weight matrix, and the weight matrix of the edges connecting the operator and the machine. After normalizing the attention weights of the operator and its successors, itself, and with each machine using the softmax function, we can obtain the normalized attention coefficients. and And use this to calculate the new representation vector of the operation node. The calculation method is as follows:

[0067] Machine Embedding: Machine nodes can receive information through two connection methods: connections from the operator and connections from other machines. The purpose of information exchange between machines is to understand each other's states and prioritize tasks based on those states. The calculation method for the first weight is similar to... Similarly, denoted as To calculate the attention weights that a machine node should assign to other machine nodes, for all m... i ,m j ∈M, calculate the following:

[0068]

[0069] in, This represents a column vector used to embed two machine nodes into a concatenated array. and These are two linear transformation matrices. After normalizing the attention coefficients... The calculation method is as follows.

[0070]

[0071] Job embedding: Job embedding means that computations originate from operation information, machine information, and other job information. M j,i Defined as the time t at which a task j can be assigned. iA collection of machines, This represents the initial feature edges. In general, attention weights are computed using a method similar to the embedding approach described previously.

[0072] The information ultimately yields h. ji New representation:

[0073] The three embedding processes described above illustrate the first layer of GNN computation. This process needs to be repeated multiple times to generate embedding representations. Each layer uses the job, operation, and machine embedding representations generated in the previous layer.

[0074] The flexible job shop scheduling problem is transformed into a Markov decision process, and a deep reinforcement learning algorithm is used for dynamic decision-making. The Markov decision process includes a state space, an action space, a reward function, and a state transition function.

[0075] Transforming the flexible job shop scheduling problem into a Markov decision process specifically includes:

[0076] State space: Each state is represented as a heterogeneous graph, including a set of processes, a set of machines, and a set of jobs, as well as the set of edges between them;

[0077] Action space: At each time step, the action set consists of all compatible job and machine pairs;

[0078] Specifically, the action space: at each time step t, the action set A4 consists of all compatible job and machine pairings. When a job is selected, the first unscheduled operation within that job is chosen. This action set corresponds to the set of edges w connecting jobs and machines. t As described in the next section, to improve learning efficiency and stabilize inference, some of these actions will be excluded from the policy's optional actions based on scheduling rules. Job nodes offer two main advantages. First, they significantly reduce the action space because the number of jobs is typically much smaller than the number of operations. Second, they concentrate problem-related information on one type of node, making the learning process more efficient and inference faster.

[0079] State transition: After an action is performed in a state, the graph will transition to a new state, where the operation has been executed and the heterogeneous graph will undergo the corresponding transformation.

[0080] Specifically, state transition: in state s r Perform action a r After that, it will transition to a new state s. t+1The operation has been executed. During each state transition, the heterogeneous graph undergoes three transformations. First, the node with the executed operation and all its connected edges are removed. Second, the edges of the executed operation are removed, and new edges are created with the next operation to be executed. Finally, the characteristics of all nodes and edges are updated.

[0081] Reward function: Defined as the difference between the completion time of the current state and the completion time of the next state, in order to minimize the completion time.

[0082] Specifically, the reward function: The agent's goal is to minimize the completion time. One way to minimize the total completion time is to define the reward function at step t as r(st,at,st+1)=C(st)-C(st+1), where C(st) represents the completion time at step t. If the discount factor x is set to 1, the cumulative reward becomes... If the completion time is zero or constant in the first step, maximizing the cumulative reward is equivalent to minimizing C(s). |a| ), which is the completion time of the final state.

[0083] Heuristic scheduling rules are used to constrain the action space of deep reinforcement learning algorithms to improve solution efficiency;

[0084] Constraining the action space using heuristic scheduling rules specifically includes:

[0085] Only the k earliest starting operations can be selected. If multiple operations have the same start time, all of them can be selected.

[0086] Only the k earliest operations that can be completed are allowed to be selected, thereby reducing the ineffective actions of the agent in the early exploration process and improving the overall scheduling efficiency.

[0087] We construct a diverse set of scheduling strategies, obtain more diverse datasets through Bayesian optimization, and introduce the K-nearest neighbor algorithm to select the most suitable strategy, thereby enhancing the model's generalization ability and solution quality.

[0088] Constructing a diverse set of scheduling strategies specifically includes:

[0089] Define a search space to determine the variables to be optimized, including the graph neural network structure, deep reinforcement learning algorithm parameters, scheduling rule types, or relevant parameters in the instance generation process;

[0090] Generate a validation set and calculate the optimal or near-optimal maximum completion time for each instance using an exact method;

[0091] By sampling the search space using Bayesian optimization, models with different configurations can be generated;

[0092] The K-nearest neighbors algorithm is used to cluster and filter the strategies, resulting in a small but high-performance set of diverse strategies.

[0093] The deep reinforcement learning algorithm employs the proximal policy advantage algorithm, which involves generating a policy network and a value network, both of which are trained simultaneously using embedding representations computed by a graph neural network.

[0094] The method also includes:

[0095] During the training phase, a set of differentiated policies is generated, and a robust and adaptable policy library is gradually built through iterative optimization.

[0096] During the inference phase, the strategy is applied to new instances to verify its effectiveness. This can be done by generating a unique solution based on a single strategy or by calling a representative subset of strategies in parallel to generate multiple solutions and selecting the best one.

[0097] To generate policies for decision-making, this invention considers utilizing the well-known proximity policy advantage algorithm. The method includes generating a policy network (Actor) π. s (a t |s t ) and a value network (Critic) V θ (s t Both networks use embedding representations computed by the GNN, but in different ways. For the Actor network, the edge set w is used. τ Prediction is performed based on the embedding. For each The machine embedding, job embedding, and edge feature embedding are concatenated and then fed into a multilayer perceptron (MLP1). A scheduling rule-based mask and softmax function are then applied to model the probability distribution of all possible job and machine combinations. For the Critic network, all job embeddings are fed into another multilayer perceptron (MLP2) with the activation function tamh, and then average pooling is performed to obtain a single state-value function.

[0098] To train these two networks effectively, n steps are performed on the networks. εz In round-based interaction, in each interaction round, the agent interacts with the environment by sampling from the probability distribution πθ(at|st) and stores the experience in a buffer. The training method architecture is as follows: Figure 3 The following is an example. Every nt rounds, we use this experience to train the network for K epochs with a batch size of b. Furthermore, every ng rounds, a new set of synthetic instances with a dimension of nins is generated for the agent to interact with. The specific algorithm training process is as follows:

[0099]

[0100]

[0101] To enhance the performance of deep reinforcement learning in flexible job shop scheduling, this section proposes two optimization strategies: first, introducing scheduling rules to constrain the action space, reducing unnecessary exploration processes and improving decision-making efficiency; second, constructing a diverse set of scheduling strategies to enhance the model's generalization ability and solution quality.

[0102] Action Space Constraint Method Based on Scheduling Rules: Scheduling rules have been widely used in flexible job shop scheduling problems. However, traditional methods usually use scheduling rules alone, which has significant limitations and is difficult to effectively cope with large-scale scheduling problems with high complexity. Combining scheduling rules with deep reinforcement learning can provide an effective benchmark for agent decision-making, especially showing more significant advantages in large-scale complex instances. In high-complexity, large-scale scheduling instances, the action space dimension is high, which seriously restricts the exploration efficiency of the agent. To this end, this invention introduces two simple heuristic scheduling rules to effectively constrain the action space, reduce the invalid actions of the agent in the early exploration process, and thus improve the overall scheduling efficiency. Specifically, the two rules are: (1) Only the k earliest starting operations are allowed to be selected (if there are multiple operations with the same start time, all of them are allowed to be selected); (2) Only the k earliest completing operations are allowed to be selected. By constraining the action space in the above way, the selection of actions that cause large gaps in the scheduling process can be effectively avoided, which helps to obtain a better maximum completion time. In addition, the compression of the action space can also significantly reduce the size of the state space that the agent needs to explore, thereby accelerating the model training process. The setting of parameter k directly determines the degree of freedom of the agent in the decision-making process. The two simple heuristic rules adopted in this invention aim to avoid the problem of increased inference time due to excessive computational complexity, thereby improving the overall scheduling performance while ensuring solution efficiency.

[0103] Constructing diverse scheduling strategies: This invention considers a simplified solution set generation method based on parallel execution of diverse models, replacing the traditional sampling strategy. Traditional sampling strategies, relying on a single strategy to generate solutions, may lead to highly similar solutions, thus limiting the diversity of solutions. Therefore, this invention aims to construct a small but high-performance diverse strategy set, enabling it to achieve good performance on different types of instances. The focus of this method is not on generating similar strategies, but on the differences between strategies.

[0104] The method consists of two main steps: (1) constructing a set of candidate policies for the inference stage; and (2) selecting the most representative policies from these. To achieve policy diversity, this invention employs Bayesian optimization to efficiently search the hyperparameter space to generate models with different configurations; subsequently, the K-nearest neighbor algorithm is used to cluster and filter the policies. The complete algorithm for generating diverse scheduling policies is shown in Algorithm 2.

[0105]

[0106]

[0107] First, define a search space D to identify the variables to be optimized. These variables may involve the graph neural network structure, PPO algorithm parameters, scheduling rule types used to limit the action space, or relevant parameters in the instance generation process. A proper definition of this search space is crucial to ensuring the diversity of generation strategies.

[0108] Next, a validation set V of size nv is generated, and the optimal or near-optimal maximum completion time for each instance is calculated using an exact method. The results are stored in set V. M This process is implemented using the method `getvalmk(V)`. To control computational costs, the validation set size should not be too large; typically, small to medium-sized instances are chosen for validation and training. Subsequently, the search space is sampled using the function `sample(D)` to obtain a set of hyperparameters `d`, and the policy model is trained based on these hyperparameters. The training process is implemented using the method `generatepolicy(d)`. The generated policy is evaluated on the validation set `V`, and the relative gap between its maximum completion time and the optimal solution is calculated. For a given instance `v` in the validation set, the gap of policy `p` is defined as...

[0109]

[0110] Where pv represents the maximum completion time of policy p on instance v. This is the result obtained by the exact method. The evaluation process is implemented by the method evalpolicy(V,VM,p). After generating the first policy, all its evaluation results are stored in the set CG as the current best benchmark. This policy is also added to the candidate policy set C, providing a basis for comparison in the generation of subsequent policies. Next, a Bayesian optimization process is performed, with nso iterations. In each iteration je{1,2,…,nso}, a set of hyperparameters di is first sampled from the search space, and the corresponding policy p is generated. jSubsequently, the gap set Gpi of the policy on the validation set is calculated, and the maximum difference between it and the current best gap is found. This difference serves as the objective function for Bayesian optimization, guiding the policy towards a direction with greater improvement potential. If the difference is positive, it indicates that the new policy performs better on at least one instance, so it is added to the candidate policy set, and the current best difference is updated.

[0111] A flexible shop floor scheduling optimization system based on deep reinforcement learning includes:

[0112] The graph neural network modeling module is used to model the workshop operation process for the flexible job shop scheduling problem; the graph neural network modeling module uses a graph attention network to model the workshop operation process.

[0113] The Markov Decision Process Transformation Module is used to transform the flexible job shop scheduling problem into a Markov decision process.

[0114] The deep reinforcement learning decision module is used to make dynamic decisions using deep reinforcement learning algorithms. The deep reinforcement learning decision module adopts the proximal policy advantage algorithm, which includes generating a policy network and a value network. Both networks are trained simultaneously using the embedding representations calculated by graph neural networks.

[0115] Specifically, the graph neural network modeling module utilizes graph attention networks (GAT) to model the shop floor workflow, effectively capturing the complex relationships between process nodes and machine nodes. The Markov decision process transformation module converts the FJSP into a Markov decision process, providing a decision framework for deep reinforcement learning. The deep reinforcement learning decision module employs the proximal policy advantage (PPO) algorithm to generate policy and value networks, and trains them using embedded representations computed by graph neural networks to achieve dynamic decision-making. This approach, combining graph neural networks and deep reinforcement learning, effectively handles the complex relationships and dynamic decision-making requirements in FJSP, improving scheduling efficiency and quality.

[0116] The action space constraint module is used to constrain the action space of deep reinforcement learning algorithms by combining heuristic scheduling rules;

[0117] The diversified strategy building module is used to construct a diverse set of scheduling strategies. It obtains a more diverse dataset through Bayesian optimization and introduces the K-nearest neighbor algorithm to select the most suitable strategy.

[0118] Specifically, the action space constraint module constrains the action space of the deep reinforcement learning algorithm by combining heuristic scheduling rules, thereby reducing invalid or inefficient action selections and improving the algorithm's search efficiency and decision quality. The diversified policy construction module builds a set of multiple scheduling policies, using Bayesian optimization techniques to obtain a more diverse dataset, enhancing the model's adaptability to different scheduling scenarios. Furthermore, this module introduces the K-nearest neighbor algorithm to select the most suitable policy from the diverse policy set for the current state, achieving better scheduling decisions. This approach, combining action space constraints and diversified policy construction, aims to improve the flexibility and robustness of the scheduling system while ensuring rapid, high-quality scheduling decisions in complex and ever-changing production environments.

[0119] The overall process of this invention is divided into a training phase and an inference phase. In the training phase, a set of differentiated strategies is first generated; in the inference phase, these strategies are applied to new instances to verify their effectiveness.

[0120] Depend on Figure 1 As shown in the diagram, the outermost layer represents the policy configuration selection, the middle layer represents the policy training process, and the innermost layer represents the single-instance solution method. Two strategies can be used in the inference phase: first, generating a unique solution based on a single policy; second, invoking a representative subset of policies in parallel to generate multiple solutions and selecting the best one. The specific steps of the training phase are as follows:

[0121] Validation set construction: First, a validation set is constructed, and the optimal solution for each instance is obtained by constrained programming (CP) as the benchmark for subsequent model performance evaluation.

[0122] Hyperparameter domain setting: Establish a hyperparameter domain to guide policy generation, which covers the distribution control of instance features (number of processes, number of machines, processing time, etc.), heterogeneous graph neural network (GNN) architecture configuration, and various parameterization schemes for the training process. Sampling in this domain ensures that each policy is trained on an independently distributed set of instances, thereby enhancing the diversity between policies.

[0123] Candidate policy pool initialization: Construct a set of candidate policies for the inference phase. Initially, a baseline policy is generated through hyperparameter domain sampling and included in this set to provide a performance reference for subsequent policies.

[0124] Optimization Objectives and Methods: The aim is to generate a diverse set of policies tailored to different instance specializations to achieve comprehensive coverage of the instance space. To this end, an optimization framework centered on policy improvement is adopted: in each iteration, policies superior to the current candidate set are identified and retained, progressively building a robust and adaptable policy library.

[0125] Iterative process: In each optimization cycle, the hyperparameter domain is first sampled, and then...* The strategy is trained in rounds. Each round, instances are randomly generated and transformed into a heterogeneous graph, which is then processed by the HGATv2 network to obtain the probability distribution of job-machine allocation. Subsequently, sampling is performed guided by scheduling rules (DR) to collect training samples. Every n rounds, the network parameters are updated once using the PPO loss function based on the accumulated samples.

[0126] Policy Evaluation: After training for n rounds, the new policy is tested on the validation set, and the performance metric is the relative gap between the new policy and the optimal solution in CP. If the new policy outperforms the current candidate set on any validation instance, it is added to the candidate set, and the policy generation model is updated accordingly.

[0127] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A flexible job shop scheduling optimization method based on deep reinforcement learning, characterized in that, The method comprises the following steps: a graph neural network model is constructed to model the workshop operation process of a flexible job shop scheduling problem; the flexible job shop scheduling problem is converted into a Markov decision process, and a deep reinforcement learning algorithm is used for dynamic decision-making, wherein the Markov decision process comprises a state space, an action space, a reward function and a state transition function; heuristic scheduling rules are combined to constrain the action space of the deep reinforcement learning algorithm, so as to improve the solving efficiency; a diversified scheduling strategy set is constructed, a more diversified data set is obtained through Bayesian optimization, and a K nearest neighbor algorithm is introduced to select the most suitable strategy, so as to enhance the generalization ability and solving quality of the model.

2. The flexible job shop scheduling optimization method based on deep reinforcement learning according to claim 1, wherein, The graph neural network model comprises the following steps: a directed graph is defined, comprising a node set and an edge set, wherein the node set comprises a set of operations, a set of machines and a set of jobs, and the edge set comprises undirected edges between machines and operations, directed edges between operations and jobs, directed edges between operations, directed edges between machines and jobs, directed edges between jobs and directed edges between machines; a graph attention network is used to process the directed graph to extract feature information of nodes and edges, so as to model the workshop operation process. 3.The flexible job shop scheduling optimization method based on deep reinforcement learning of claim 1, wherein, The flexible job shop scheduling problem is converted into a Markov decision process, which comprises the following steps: state space: each state is represented as a heterogeneous graph, comprising a set of operations, a set of machines and a set of jobs, and a set of edges between them; action space: at each time step, the action set is composed of all compatible pairs of jobs and machines; state transition: after the action is performed in the state, it will be converted to a new state, in which the operation has been performed, and the heterogeneous graph will undergo corresponding transformation; reward function: defined as the difference between the completion time of the current state and the completion time of the next state, to achieve the goal of minimizing the completion time.

4. The flexible job shop scheduling optimization method based on deep reinforcement learning according to claim 1, wherein, The action space is constrained by combining heuristic scheduling rules, which comprises the following steps: only the k operations with the earliest start time are allowed to be selected, and if there are multiple operations with the same start time, all of them are allowed to be selected; only the k operations with the earliest completion time are allowed to be selected, which reduces the invalid actions of the agent in the early exploration process, thereby improving the overall scheduling efficiency.

5. The flexible job shop scheduling optimization method based on deep reinforcement learning according to claim 1, wherein, The diversified scheduling strategy set is constructed, which comprises the following steps: a search space is defined to determine the variables to be optimized, including the graph neural network structure, the deep reinforcement learning algorithm parameters, the scheduling rule type or the related parameters in the instance generation process; a validation set is generated, and the optimal or approximately optimal maximum completion time of each instance is calculated by an exact method; the search space is sampled by Bayesian optimization to generate models with different configurations; K nearest neighbor algorithm is used to cluster and filter the strategies, and finally a diversified strategy set with small size but excellent performance is obtained.

6. The flexible job shop scheduling optimization method based on deep reinforcement learning according to claim 1, wherein, The deep reinforcement learning algorithm uses the proximal policy optimization algorithm, which comprises generating a policy network and a value network, and both networks are trained using the embedding representation calculated by the graph neural network.

7. The flexible job shop scheduling optimization method based on deep reinforcement learning according to claim 1, wherein, The method further comprises the following steps: in the training phase, a group of different strategies are generated, and a robust and adaptive strategy library is gradually constructed through iterative optimization; In the reasoning phase, the policy is applied to a brand new instance to verify its effectiveness, which can adopt a strategy based on a single policy to generate a unique solution or call a representative subset of strategies in parallel to produce multiple solutions and choose the best one.

8. A flexible job shop scheduling optimization system based on deep reinforcement learning, according to any one of claims 1-7, characterized in that, The method comprises the following steps: a graph neural network modeling module for modeling the workshop operation process of the flexible job shop scheduling problem; a Markov decision process transformation module for transforming the flexible job shop scheduling problem into a Markov decision process; a deep reinforcement learning decision module for dynamic decision making by using a deep reinforcement learning algorithm; an action space constraint module for constraining the action space of the deep reinforcement learning algorithm in combination with heuristic scheduling rules; a diversified strategy construction module for constructing a diversified scheduling strategy set, obtaining a more diversified data set through Bayesian optimization, and introducing a K-nearest neighbor algorithm to select the most suitable strategy.

9. The deep reinforcement learning based flexible job shop scheduling optimization system of claim 8, wherein, The graph neural network modeling module models the workshop operation process by using a graph attention network.

10. The flexible job shop scheduling optimization system based on deep reinforcement learning of claim 8, wherein, The deep reinforcement learning decision module adopts a proximal policy optimization algorithm, which includes generating a policy network and a value network, and both networks are trained using the embedding representation calculated by the graph neural network.

Images