A job shop scheduling method and system based on competitive meta-reinforcement learning

Abstract

This invention discloses a job shop scheduling method and system based on competitive meta-reinforcement learning. First, a task library containing multiple scheduling task types is constructed, and multiple scheduling models are randomly initialized. In each training cycle, a task environment is randomly selected for each scheduling model. A support set is generated through the interaction between the behavioral policy network and the environment, and the inner-loop adaptive update of the model parameters is performed. Then, a query set is generated through interaction with a new environment, and the outer-loop meta-parameters are updated. Finally, the learning performance of each scheduling model is compared based on a competitive evaluation function, identifying the best-performing scheduling model in this training round, and adjusting the parameters of other models to align with its parameters. This process is repeated until training is complete and multiple high-quality scheduling models are obtained. This invention helps scheduling models quickly adapt to different tasks, improves their autonomous decision-making and continuous optimization capabilities, and is of great significance for solving complex and dynamic scheduling problems in reality.

Description

Technical Field

[0001] This invention relates to the field of discrete manufacturing technology, specifically to a job shop scheduling method and system based on competitive meta-reinforcement learning. Background Technology

[0002] Shop floor scheduling is of significant practical importance in modern production activities. Reasonable scheduling decisions can effectively improve production efficiency and reduce production costs. In the global wave of "smart manufacturing" industrial transformation, the discrete manufacturing industry, along with the rise of artificial intelligence technology, has also ushered in new challenges and opportunities. Traditional methods for solving shop floor scheduling problems mainly include exact methods and approximate methods. Exact methods, such as branch and bound and mixed integer programming, aim to obtain the optimal solution through rigorous calculation. While accurate, these methods typically require high time and computational costs, and are therefore less commonly used in practice. Approximate methods, such as priority scheduling rule algorithms and intelligent optimization algorithms, are easier to implement and have faster computation speeds, thus being more widely used. However, these algorithms often require targeted rule or parameter design to ensure performance when facing different task scenarios. With the increasing prevalence of smart manufacturing technology, modern production lines frequently require scheduling adjustments under small sample conditions, such as temporary adjustments to production tasks, emergency manpower allocation, or changes in production type. In such complex and dynamic scenarios, how to enable the scheduling decision model to quickly adapt to new small sample data and effectively cope with new scheduling tasks remains a pressing problem to be solved.

[0003] In recent years, deep reinforcement learning methods have played an important role in an increasing number of fields, including combinatorial optimization, such as job shop scheduling. By modeling the scheduling task as a Markov decision process, reinforcement learning methods can help the decision model explore and learn in a simulated job shop environment, thereby providing end-to-end scheduling decision solutions. Currently, deep reinforcement learning has been widely applied to solve job shop scheduling problems and has achieved excellent performance. Luo et al. (Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Applied Soft Computing, 2020) combined deep Q-networks with scheduling rules to design a deep reinforcement learning-based scheduling method that can quickly obtain robust solutions in job shop scheduling problems. Zhang et al. (Learning to dispatch for job shop scheduling via deep reinforcement learning. Advances in neural information processing systems, 2020) designed a method that uses a deep reinforcement learning model to learn and generate high-quality priority scheduling rules. It extracts state features based on graph neural networks and can achieve fast end-to-end decision-making. Furthermore, to enhance adaptability to different types of scheduling tasks or environments, meta-learning techniques can be introduced on top of reinforcement learning methods. Meta-learning, also known as "learning how to learn," is a cutting-edge machine learning technique. Meta-reinforcement learning is a combination of meta-learning and reinforcement learning. Its core idea is to teach an agent how to learn, that is, to extract generalizable knowledge or strategies through learning on multiple related tasks, enabling the agent to quickly learn and adapt when facing new tasks. Zhang et al. proposed a new deep reinforcement learning method based on the concept of meta-learning in their paper (Meta-learning-based deep reinforcement learning for multiobjectiveoptimization problems. IEEE Transactions on Neural Networks and Learning Systems, 2022). This method trains a meta-model and fine-tunes it, enabling the model to quickly adapt to new weight vector configurations, thus generating high-quality Pareto front solutions with fewer updates.

[0004] While the scheduling methods listed above can be used to solve the job shop scheduling problem, they still have some limitations. These limitations primarily manifest in poor generalization and adaptability to different types of scheduling tasks and environments. Therefore, there is room for improvement in aspects such as model cross-task adaptation speed, solution efficiency, and scheduling quality. Consequently, there is an urgent need to propose a new method and system that supports training job shop scheduling models in different types of tasks and environments. This system should efficiently train the model while enhancing its adaptability to different scheduling tasks, enabling autonomous decision-making and continuous optimization under sudden task changes. Summary of the Invention

[0005] In view of this, the present invention provides a job shop scheduling method and system based on competitive meta-reinforcement learning, which can improve generalization ability, support job shop scheduling model training in different types of tasks and environments, enhance the model's adaptability to different scheduling tasks, and realize autonomous decision-making and continuous optimization under the condition of sudden task changes.

[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0007] A job shop scheduling method based on competitive meta-reinforcement learning includes:

[0008] A diverse task library is generated by randomly setting different problem sizes, processing time distributions, and problem types, resulting in a task library containing various types of scheduling task environments. Each scheduling task consists of a set of jobs and a set of machines, with each job containing a series of operations. The problem size refers to the product of the number of machines and jobs in a job shop scheduling problem. The processing time distribution refers to the random distribution type used for sampling the processing time required for an operation to be completed on the corresponding machine during the initialization of the job shop environment. The problem type refers to the type of problem set during the initialization of the job shop environment.

[0009] Construct a scheduling model, which includes a policy network π set up by a decision model based on the Actor-Critic framework. θ Behavioral Strategy Network π θold And evaluation network v φ Policy Network π θ The parameters θ and the evaluation network v φ The parameter φ in the model constitutes the parameter Θ = {θ, φ} of the scheduling model;

[0010] Training multiple scheduling models in parallel:

[0011] During training, for each scheduling model, a scheduling task is randomly selected from the task library. Using the aforementioned behavioral policy network π θold Generate support sets by interacting with the environment Based on support set Meta-learning training is performed, the inner loop loss is calculated, and the parameters of the scheduling model are updated based on the inner loop loss; the updated parameters are denoted as Θ. i ′={θ i ′,φ i ′};i represents the sequence number of the scheduled task;

[0012] When targeting a specific scheduling task The number of inner loop iterations has reached the target. For each scheduling model, the updated parameters Θ on each task are used. i Constructing a policy network with new parameters Generate query sets by interacting with the environment Based on query set Perform meta-learning training, calculate the outer loop loss and meta-gradient, and update the outer loop meta-update of the scheduling model;

[0013] Select the best scheduling model from the current training cycle and mark it as the winner; the rest are the losers. The parameters of the winner remain unchanged; the parameters of all losers are updated in the direction of the parameters of the winner.

[0014] By performing training over multiple cycles, a scheduling model with good learning performance is finally obtained.

[0015] Preferably, the parameters of all losers are updated in the direction of the parameters of the winners as follows: Let the parameters of the losers be Θ. l The parameter of the preferred one is Θ w Then update the parameters of the loser to Θ. l +γ(Θ w -Θ l ); where γ is the adjustment factor.

[0016] Preferably, the types and quantities of scheduled tasks in the task library are updated periodically.

[0017] Preferably, the workshop environment adopts the following modeling: disjunctive structure modeling; in the disjunctive graph, each operation is represented by a node, the sequential relationship between nodes is represented by directed arcs, and undirected arcs connect operations that need to be processed by the same machine; the scheduling process marks the direction of all undirected arcs in the disjunctive graph, that is, indicates the execution order of operations that need to be processed by the same machine.

[0018] The present invention also provides a job shop scheduling system based on competitive meta-reinforcement learning, including a task library, a scheduling model, a meta-reinforcement learning training module, and an optimization module;

[0019] The task library is used to set up various types of scheduling tasks corresponding to different job shop environments by randomly setting different problem sizes, processing time distributions, and problem types. Each scheduling task consists of a set of jobs and a set of machines, with each job containing a series of operations. The problem size refers to the product of the number of machines and the number of jobs in a job shop scheduling problem. The processing time distribution refers to the random distribution type on which the processing time required for an operation to be completed on the corresponding machine is randomly sampled during the initialization of the job shop environment. The problem type refers to the type of problem set for solving the problem during the initialization of the job shop environment.

[0020] The scheduling model is based on the decision-making model constructed using the Actor-Critic framework, including the policy network π. θ Behavioral Strategy Network π θold and evaluation network ν φ Policy Network π θ The parameters θ and the evaluation network ν φ The parameter φ in the model constitutes the parameter Θ = {θ, φ} of the scheduling model; there are multiple scheduling models to be trained.

[0021] The meta-reinforcement learning training module trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library. Using the aforementioned behavioral policy network π θold Generate support sets by interacting with the environment Based on support set Meta-learning training is performed, the inner loop loss is calculated, and the parameters of the scheduling model are updated based on the inner loop loss; the updated parameters are denoted as Θ′. i ={θ′ i ,φ′ i}, where i represents the sequence number of the scheduled task; when targeting a specific scheduled task The number of inner loop iterations has reached the target. For each scheduling model, the updated parameters Θ′ on each task are used. i Constructing a policy network with new parameters Generate query sets by interacting with the environment Based on query set Perform meta-learning training, calculate the outer loop loss and meta-gradient, and update the outer loop meta-update of the scheduling model;

[0022] The optimization module is used to evaluate the scheduling models after each scheduling model has completed its parameter update. The scheduling model with the best evaluation result is marked as the best in this stage, and the rest are the failures. The parameters of the best remain unchanged; the parameters of all failures are updated in the direction of the parameters of the best.

[0023] The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, ultimately resulting in a scheduling model with good learning performance.

[0024] Preferably, when the optimization module updates the parameters of the loser towards the parameters of the optimizer, the parameter of the loser is set to Θ. l The parameter of the preferred one is Θ w Then update the parameters of the loser to Θ. l +γ(Θ w -Θ l ); where γ is the adjustment factor.

[0025] Beneficial effects:

[0026] (1) The job shop scheduling scheme based on competitive meta-reinforcement learning designed in this invention, and the job shop scheduling method based on competitive meta-reinforcement learning, apply the meta-reinforcement learning method to the training of the scheduling model, so that the model can summarize general knowledge in different types of scheduling task data and enhance its adaptability to sudden new tasks, and realize autonomous decision-making and continuous optimization under the condition of sudden task changes.

[0027] (2) By training multiple scheduling models in parallel and using competitive parameter updates, the learning efficiency and training stability of the models are further improved.

[0028] (3) The present invention constructs a diversified task library corresponding to diverse task scenarios. By using data from different types of tasks in training, the model is made adaptable to specific structural problems, which enhances the ability to quickly adapt to and make scheduling decisions in different work workshops. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the operation structure of a job shop scheduling method and system based on competitive meta-reinforcement learning provided by the present invention;

[0030] Figure 2 This is a schematic diagram of a diversified scheduling task library in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0031] Figure 3 This is a schematic diagram of the disjunctive graph-based job shop environment modeling method in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0032] Figure 4 This is a schematic diagram of a decision model based on a disjunctive graph and an Actor-Critic framework in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention.

[0033] Figure 5This is a schematic diagram illustrating the specific operation process of a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0034] Figure 6 This is a diagram illustrating the operational effect of an embodiment of a job shop scheduling method based on competitive meta-reinforcement learning provided by this invention.

[0035] Figure 7 This is a block diagram of a job shop scheduling system based on competitive meta-reinforcement learning provided by the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] This invention provides a job shop scheduling scheme based on competitive meta-reinforcement learning. By applying meta-reinforcement learning to the training of the scheduling model, the model summarizes general knowledge from different types of scheduling task data and enhances its adaptability to sudden new tasks. Furthermore, by training multiple scheduling models in parallel and using competitive parameter updates, the learning efficiency and training stability of the models are further improved.

[0038] This invention also constructs a diverse task library, consisting of a set of different types of job shop scheduling tasks. These tasks have varying problem sizes, processing time distributions, and problem types, corresponding to different job shop environments. Through this diversified approach, a wide range of scheduling task types are incorporated into the training data of the scheduling decision model. This allows the model to adapt to job shop scheduling decisions—a type of problem with specific structural characteristics—by summarizing different types of data during meta-training, enabling it to quickly adapt to sudden changes in new tasks. Furthermore, by continuously updating the task types and quantities in the task library, the autonomous decision-making capability of the scheduling model can be continuously optimized.

[0039] Therefore, the focus of this invention is to design a diversified scheduling task library, a decision model based on the Actor-Critic framework, and a training method for a scheduling decision model based on competitive meta-reinforcement learning.

[0040] The diversified scheduling task library provided by this invention describes a scheduling task based on three characteristics: problem size, processing time distribution, and problem type. It treats scheduling tasks with different problem sizes, processing time distributions, or problem types as different types of tasks. For example... Figure 2As shown, the problem size refers to the product of the number of machines and the number of jobs in a job shop scheduling problem, such as 20×10, 50×20, etc. Processing time refers to the time required for an operation to be completed on its corresponding machine, and this time is sampled from a random distribution during the initialization of the shop environment; this distribution is the processing time distribution, such as [1,99], [1,199], etc. Problem type refers to the type of problem set during the initialization of the shop environment. For example, in the J / / Cmax problem, the machine sequence for each job is randomly generated; while in the J / 2SETS / Cmax problem, the machines are divided into two equal subsets, and each job must complete its operation on all machines in one subset before it can be transferred to the machines in the other subset for further processing. When generating a diverse scheduling task library, different problem sizes, processing time distributions, and problem types are randomly set to ensure that the task library contains shop environments corresponding to various types of tasks, which are used for meta-reinforcement learning training of the scheduling model.

[0041] This invention employs a disjunctive graph-based job shop environment modeling method, using a graph structure to describe the job shop environment. For example... Figure 3 As shown, in the disjunctive graph, each operation is represented by a node, and the order of operations between nodes is represented by directed arcs to ensure that operations are processed in the prescribed order. Undirected arcs connect operations that need to be processed by the same machine. The scheduling process then marks the direction of all undirected arcs in the disjunctive graph, indicating the execution order of operations that need to be processed by the same machine. Using a graph structure to describe the environment, the relationships between different workpieces and processing sequences are represented in a point-arc format, facilitating data input to the neural network. The decision model obtains the data needed for training, such as state transition processes, by interacting with the disjunctive graph-based job shop environment.

[0042] See Figure 4 The decision-making model based on the Actor-Critic framework provided by this invention uses two deep neural networks as the policy network Actor and the evaluation network Critic, respectively. The Actor is responsible for learning a policy function and selecting scheduling actions based on the current workshop state. The policy network Actor is further divided into a policy network π. θ Behavioral Strategy Network π θold The two parts are trained alternately, with the parameters of the behavior policy network being assigned to the policy network. The Critic is responsible for evaluating the value of scheduled actions and providing reward signals to guide the Actor's actions. Specifically, taking the simplest AC framework as an example, the Actor module needs to learn a policy function π. θLet (a|s) represent the probability distribution of choosing action a in state s. This policy is typically represented by a parameterized neural network, with its parameters denoted as θ. The Actor's goal is to maximize the expected cumulative reward, i.e., to maximize the expected reward of the policy function. Since the probability of the Actor choosing an action based on the current state is π... θ If (a|s), then it can be updated according to the policy gradient method: Where α is the learning rate for policy gradient updates, and G t Let be the expected value of the accumulated reward from time t until some future time. Let s be the gradient of the objective function with respect to the network parameters θ. t Let a be the state at time t. t Let be the action at time t. The Critic model, on the other hand, needs to learn a Q-function Q. φ (s,a) is used to evaluate the value of the state-action pair (s,a), and it can also be represented by a parameterized neural network, denoted as φ. The goal of the Critic is to estimate the value of the current state or action and provide a reasonable reward signal to guide the Actor's action selection. Therefore, according to the value function method, the Critic's update method is as follows: Where δt is the temporal difference (TD) error. Let be the gradient of the objective function with respect to the network parameters φ. In this invention, the updates of Actor and Critic will depend on the inner and outer loop losses.

[0043] The scheduling model training method based on competitive meta-reinforcement learning provided by this invention is applied to different types of scheduling tasks, aiming to enhance the generalization ability of the scheduling model and improve its adaptation speed and decision quality when facing new tasks. The training method includes five steps: task library initialization, scheduling model initialization, adaptive update of inner-loop parameters for specific tasks, meta-update of outer-loop parameters across multiple tasks, and competitive parameter update between scheduling models based on preset competitive indicators. Specifically, task library initialization creates different workshop environments (i.e., different environmental data) for different tasks based on different problem sizes, processing time distributions, and problem types; multiple scheduling models are initialized simultaneously and trained in parallel; inner-loop updates use the support set portion of the environmental data in the task library; outer-loop meta-updates use the query set portion of the data; and competitive updates directly correct the scheduling model parameters without requiring environmental data.

[0044] See Figure 1 and Figure 5 The present invention provides a job shop scheduling method based on competitive meta-reinforcement learning, comprising the following steps:

[0045] Step S1: Prepare the diverse task library T and scheduling model Θ = {θ, φ} required for the meta-training process.

[0046] The task library T contains various types of scheduling tasks with different problem sizes, data distributions, and problem types. The scheduling model includes a policy network π. θ Behavioral Strategy Network π θold and evaluation network ν φ , π θ and π θold In the scheduling model Θ, θ represents the Actor; the evaluation network v φ In the scheduling model Θ, φ represents Critic. Updates to Θ will include updates to both θ and φ. The competition evaluation function V is also defined. c .

[0047] Multiple scheduling models are trained in parallel, their parameters are randomly initialized, and training data is set, including the number of operation nodes n×m, the processing time distribution range, and the dataset structure type. Algorithm hyperparameters are set, including the inner loop learning rate α, the outer loop learning rate β, the competitive learning rate γ, the maximum training period N, and the maximum number of inner loop updates K.

[0048] Step S2: If the training period t reaches the preset maximum training period N, then training ends. Otherwise, for each scheduling model, a task is randomly selected from the task library D. i represents the sequence number of the scheduled task, which consists of a set of jobs J and a set of machines M. Each job contains a series of operations O. For each task... Using behavioral policy network π θold Interact with the corresponding environment and collect data including state transition processes (s n,t ,a n,t ,r n,t ,s n,t+1 ) and generalized dominance estimation A n,t Collect the data and add it to the support set. In China, it supports training sets similar to those in traditional deep learning algorithms, allowing the model to be updated based on gradient descent using a specific loss function.

[0049] Step S3, using support sets Calculate the inner loop loss And conduct specific tasks Update the adaptive parameters: Θ′ represents the updated parameter, Θ′ i ={θ′ i ,φ′ iThe inner loop iteration count k is incremented by 1. If k reaches the maximum inner loop update count K, then proceed to S4. Otherwise, return and re-execute S3.

[0050] Specifically, firstly, using support sets Calculate the policy loss, which measures the new policy π. θ Compared to the old strategy π θold Differences between them:

[0051]

[0052] In the above formula, clip(·) is the clipping function, used to limit the strategy ratio value to the range [1-∈, 1+∈], where ∈ is the set range value, and A n,t This is the generalized advantage estimate at time t in the nth training period.

[0053] Next, calculate the value function loss, which aims to reduce the value function v. φ Predicted value vs. actual value A n,t The difference between them:

[0054]

[0055] Recalculate the entropy loss to encourage the policy to explore more actions:

[0056]

[0057] Among them, a n,t and s n,t Let S(·) represent the state of the action at time t in the nth training epoch, and let S(·) be the entropy function used to measure the randomness of the policy. The higher the entropy value, the stronger the randomness of the policy, which helps to explore more potentially effective actions; the lower the entropy value, the stronger the determinism of the policy, which helps to obtain higher rewards by utilizing the current policy.

[0058] Finally, the inner ring loss is obtained. Among them, c v and s S This is an adjustment factor. Then, a specific task is performed. The following adaptive parameter updates: This represents the gradient of the inner-loop loss with respect to the scheduling model parameter Θ.

[0059] Step S4: Each scheduling model updates its adaptive parameters Θ′ based on the inner loop for its specific task. i This parameter is then passed to the policy network to form the policy network. Using this policy network It interacts with the corresponding environment, collects data including state transition processes and generalized dominance estimation, and forms a query set. Then, based on the query set Through a calculation process similar to step S3, the corresponding outer loop loss and elementary gradient can be obtained. And update the outer loop elements of the scheduling model:

[0060] S5. After each scheduling model completes its meta-update, the scheduling models are evaluated, and the best scheduling model is selected.

[0061] In this step, after the parameters of each model are updated in this iteration cycle, the competitive evaluation function V can be used. c The evaluation criteria, such as test accuracy and training loss, are used to assess the training effectiveness and compare the learning performance of different scheduling models. For example, the competitive evaluation function could use the maximum completion time obtained from the test data. For evaluating the learning performance, the model's test accuracy on the test task can be used as a competitive metric.

[0062] Based on the evaluation results, the best-performing scheduling model is labeled as the winner, and the remaining scheduling models are labeled as losers. For the winner, its parameter Θ is preserved. w For the loser, its parameters are adjusted to approximate those of the winner. The parameter update method to approximate the winner's parameters can be: Θ l ←Θ l +γ(Θ w -Θ l ), where γ is the adjustment factor.

[0063] S6, repeatedly execute operations from S2 to S5 until the number of cycles reaches the maximum training period N, and finally obtain multiple job shop scheduling decision models with strong adaptability.

[0064] like Figure 6 As shown, Figure 6This paper presents the results of an instance of the job shop scheduling method based on competitive meta-reinforcement learning provided by this invention in job shop scheduling environments with different problem sizes and processing time distributions. In the experiment, the scheduling model was trained in an environment with a fixed problem size of 20×20 and processing time distributions ranging from [1,29] to [1,169]. Testing was conducted in environments with problem sizes ranging from 20×15 to 50×20 and processing time distributions ranging from [1,199]. The job shop scheduling method based on competitive meta-reinforcement learning (ML-FSJSS) provided by this invention was compared with four classic heuristic rules and one reinforcement learning-based scheduling method. The results show that the job shop scheduling method based on competitive meta-reinforcement learning provided by this invention outperforms the compared algorithms in terms of solution quality. Furthermore, in an environment with 1000 operation nodes (i.e., a problem size of 50×20), the model can quickly respond to new tasks and make scheduling decisions within 2.3 seconds. The results show that the method proposed in this invention can help the scheduling model quickly learn and adapt to new tasks, realize autonomous decision-making and continuous optimization under sudden task changes, and fully demonstrate the effectiveness of the method.

[0065] Based on the above method, the present invention also provides a job shop scheduling system based on competitive meta-reinforcement learning, which includes a task library, a scheduling model, a meta-reinforcement learning training module, and an optimization module.

[0066] The task library is used to set up various types of scheduling tasks to correspond to different workshop environments by randomly setting different problem sizes, processing time distributions, and problem types.

[0067] The scheduling model is based on the decision-making model constructed using the Actor-Critic framework, including the policy network π. θ Behavioral Strategy Network π θold And evaluation network v φ There are multiple scheduling models to be trained.

[0068] The meta-reinforcement learning training module trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library. Using the aforementioned behavioral policy network π θold Interact with the corresponding environment to collect data and form a support set. Based on support set Meta-learning training is performed, the inner loop loss is calculated, and the adaptive parameters are updated based on the inner loop loss. When the number of inner loop iterations reaches a certain threshold, for each scheduling model, the updated adaptive parameters Θ′ on each task are used. i Constructing a policy network with new parameters Interact with the corresponding environment to collect data and form a query set. Based on query set Perform meta-learning training, calculate the outer loop loss and meta-gradient, and update the outer loop meta-update of the scheduling model;

[0069] The optimization module evaluates the scheduling models after each model has completed its parameter update. The scheduling model with the best evaluation result is marked as the winner in this stage, and the rest are losers. The parameters of the winner remain unchanged. The parameters of all losers are updated in the direction of the winner's parameters. In a preferred embodiment, when updating the parameters of the losers in the direction of the winner's parameters, the optimization module updates the parameters of the losers to Θ. l ←Θ l +γ(Θ w -Θ l The parameter for the loser is Θ. l The parameter of the preferred one is Θ w γ is the adjustment factor.

[0070] The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, ultimately resulting in a scheduling model with good learning performance.

[0071] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A job shop scheduling method based on competitive meta-reinforcement learning, characterized in that, include: A diverse task library is generated by randomly setting different problem sizes, processing time distributions, and problem types, resulting in a task library containing various types of scheduling task environments. Each scheduling task consists of a set of jobs and a set of machines, with each job containing a series of operations. The problem size refers to the product of the number of machines and jobs in a job shop scheduling problem. The processing time distribution refers to the random distribution type used for sampling the processing time required for an operation to be completed on the corresponding machine during the initialization of the job shop environment. The problem type refers to the type of problem set during the initialization of the job shop environment. Construct a scheduling model, which includes a policy network π set up by a decision model based on the Actor-Critic framework. θ Behavioral Strategy Network π θold and evaluation network ν φ Policy Network π θ The parameters θ and the evaluation network v φ The parameter φ in the model constitutes the parameter Θ = {θ, φ} of the scheduling model; Training multiple scheduling models in parallel: During training, for each scheduling model, a scheduling task is randomly selected from the task library. Using the aforementioned behavioral policy network π θold Generate support sets by interacting with the environment Based on support set Meta-learning training is performed, the inner loop loss is calculated, and the parameters of the scheduling model are updated based on the inner loop loss; the updated parameters are denoted as Θ′. i ={θ′ i ,φ′ i }; i represents the sequence number of the scheduled task; When targeting a specific scheduling task The number of inner loop iterations has reached the target. For each scheduling model, the updated parameters Θ′ on each task are used. i Constructing a policy network with new parameters Generate query sets by interacting with the environment Based on query set Perform meta-learning training, calculate the outer loop loss and meta-gradient, and update the outer loop meta-update of the scheduling model; Select the best scheduling model from the current training cycle and mark it as the winner; the rest are the losers. The parameters of the winner remain unchanged; the parameters of all losers are updated in the direction of the parameters of the winner. By performing training over multiple cycles, a scheduling model with good learning performance is finally obtained.

2. The method as described in claim 1, characterized in that, The parameters of all losers are updated towards the parameters of the winners as follows: Let the parameter of the loser be Θ. l The parameter of the preferred one is Θ w Then update the parameters of the loser to Θ. l +γ(Θ w -Θ l ); where γ is the adjustment factor.

3. The method as described in claim 1, characterized in that, Regularly update the types and quantities of scheduled tasks in the task library.

4. The method as described in claim 1, characterized in that, The workshop environment is modeled using a disjunctive structure. In the disjunctive graph, each operation is represented by a node, and the sequential relationship between nodes is represented by directed arcs. Undirected arcs connect operations that need to be processed by the same machine. The scheduling process marks the direction of all undirected arcs in the disjunctive graph, that is, it indicates the execution order of operations that need to be processed by the same machine.

5. A job shop scheduling system based on competitive meta-reinforcement learning, characterized in that, Includes a task library, scheduling model, meta-reinforcement learning training module, and optimization module; The task library is used to set up various types of scheduling tasks corresponding to different job shop environments by randomly setting different problem sizes, processing time distributions, and problem types. Each scheduling task consists of a set of jobs and a set of machines, with each job containing a series of operations. The problem size refers to the product of the number of machines and the number of jobs in a job shop scheduling problem. The processing time distribution refers to the random distribution type on which the processing time required for an operation to be completed on the corresponding machine is randomly sampled during the initialization of the job shop environment. The problem type refers to the type of problem set for solving the problem during the initialization of the job shop environment. The scheduling model is based on the decision-making model constructed using the Actor-Critic framework, including the policy network π. θ Behavioral Strategy Network π θold And evaluation network v φ Policy Network π θ The parameters θ and the evaluation network v φ The parameter φ in the model constitutes the parameter Θ = {θ, φ} of the scheduling model; there are multiple scheduling models to be trained. The meta-reinforcement learning training module trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library. Using the aforementioned behavioral policy network π θold Generate support sets by interacting with the environment Based on support set Meta-learning training is performed, the inner loop loss is calculated, and the parameters of the scheduling model are updated based on the inner loop loss; the updated parameters are denoted as Θ′. i ={θ′ i ,φ′ i }, where i represents the sequence number of the scheduled task; when targeting a specific scheduled task The number of inner loop iterations has reached the target. For each scheduling model, the updated parameters Θ′ on each task are used. i Constructing a policy network with new parameters Generate query sets by interacting with the environment Based on query set Perform meta-learning training, calculate the outer loop loss and meta-gradient, and update the outer loop meta-update of the scheduling model; The optimization module is used to evaluate the scheduling models after each scheduling model has completed its parameter update. The scheduling model with the best evaluation result is marked as the best in this stage, and the rest are the failures. The parameters of the best remain unchanged; the parameters of all failures are updated in the direction of the parameters of the best. The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, ultimately resulting in a scheduling model with good learning performance.

6. The system as described in claim 5, characterized in that, When the optimization module updates the parameters of the losers towards the parameters of the optimizers, let the parameters of the losers be Θ. l The parameter of the preferred one is Θ w Then update the parameters of the loser to Θ. l +γ(Θ w -Θ l ); where γ is the adjustment factor.

Images