Job-shop scheduling method and system based on competitive element reinforcement learning

Abstract

The invention discloses a job shop scheduling method and system based on competitive meta-reinforcement learning, and the method comprises the steps: constructing a task library containing a plurality of different scheduling task types, and randomly initializing a plurality of scheduling models; in each training period, a task environment is randomly extracted for each scheduling model, a support set is generated through interaction of a behavior strategy network and the environment, and inner ring adaptive updating of model parameters is carried out. Then, interacting with a new environment to generate a query set, and updating outer ring element parameters; and finally, comparing learning effects of the scheduling models based on a competition evaluation function, distinguishing the scheduling model which is evaluated optimally in the training, and enabling parameters of other models to be close to each other in the parameter direction. And repeating the above process until the training is completed and a plurality of high-quality scheduling models are obtained. According to the method, the scheduling model is helped to quickly adapt to different tasks, the autonomous decision-making and continuous optimization capabilities of the scheduling model are improved, and the method has very important significance for solving complex and dynamic scheduling problems in reality.

Description

Technical Field

[0001] The present invention relates to the field of discrete manufacturing technology, and in particular to a job shop scheduling method and system based on competitive meta-reinforcement learning. Background Art

[0002] The workshop scheduling problem has important practical significance in modern production activities. Through reasonable scheduling decisions, production efficiency can be effectively improved and production costs can be reduced. In the global wave of "intelligent manufacturing" industrial transformation, the discrete manufacturing industry has also ushered in new challenges and opportunities with the rise of artificial intelligence technology. Traditional workshop scheduling problem-solving methods mainly include exact methods and approximate methods. Among them, exact methods such as branch and bound method and mixed integer programming aim to obtain the optimal solution through rigorous calculation. Although these methods are accurate, they usually require high time and computing costs, so they are rarely used in practice. Approximate methods, such as priority scheduling rule algorithms and intelligent optimization algorithms, are easy to implement and have fast computing speeds, so they are more widely used. However, when facing different task scenarios, these algorithms often need to carry out targeted rule or parameter design to ensure performance. With the increasing popularity of intelligent manufacturing technology, modern production lines often need to make scheduling adjustments in small sample situations, such as temporary adjustment of production tasks, emergency deployment of manpower, or change of production types. In this complex and dynamic scenario, how to make the scheduling decision model quickly adapt to small sample new data and effectively respond to new scheduling tasks is still an urgent problem to be solved.

[0003] In recent years, deep reinforcement learning methods have played an important role in more and more fields, and combinatorial optimization, including job shop scheduling problems, is one of them. By modeling the scheduling task as a Markov decision process, reinforcement learning methods can help decision models explore and learn in a simulated workshop environment, thereby achieving end-to-end scheduling decision solutions. At present, deep reinforcement learning has been widely used to solve workshop scheduling problems and has achieved outstanding performance. Luo et al. (Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Applied Soft Computing, 2020) combined deep Q networks with scheduling rules and designed a scheduling method based on deep reinforcement learning, which can quickly obtain robust solutions in job shop scheduling problems. Zhang et al. (Learning to dispatch for job shop scheduling via deep reinforcement learning. Advances in neural information processing systems, 2020) designed a method using deep reinforcement learning models to learn and generate high-quality priority scheduling rules. It extracts state features based on graph neural networks and can achieve end-to-end fast decision-making. Furthermore, in order to enhance the adaptability to different types of scheduling tasks or environments, meta-learning technology can be introduced on the basis of reinforcement learning methods. Meta-learning, also known as "learning how to learn", is a cutting-edge machine learning technology. The meta-reinforcement learning method is a combination of meta-learning and reinforcement learning. Its core idea is to let the agent learn how to learn, that is, to extract generalizable general knowledge or strategies through learning on multiple related tasks, so that the agent can quickly learn and adapt when facing new tasks. Zhang et al. proposed a new deep reinforcement learning method based on the concept of meta-learning in the literature (Meta-learning-based deep reinforcement learning for multiobjective optimization problems. IEEE Transactions on Neural Networks and Learning Systems, 2022). This method trains a meta-model and fine-tunes it so that the model can quickly adapt to the new weight vector configuration, so that only a small number of updates are required to generate a high-quality Pareto frontier solution set.

[0004] Although the scheduling methods listed above can be used to solve the job shop scheduling problem, there are still some problems, mainly manifested in the poor generalization and adaptability to different types of scheduling tasks and environments. Therefore, there is room for improvement in the model's cross-task adaptation speed, solution efficiency, and scheduling quality. Therefore, it is urgent to propose a new method and system that supports the training of job shop scheduling models in different types of tasks and environments, while efficiently training the model, enhancing the model's adaptability to different scheduling tasks, and achieving autonomous decision-making and continuous optimization in the case of task mutations. Summary of the invention

[0005] In view of this, the present invention provides a job shop scheduling method and system based on competitive meta-reinforcement learning, which can improve generalization ability, support job shop scheduling model training in different types of tasks and environments, enhance the model's adaptability to different scheduling tasks, and realize autonomous decision-making and continuous optimization in the case of task mutation.

[0006] In order to solve the above technical problems, the present invention is implemented as follows.

[0007] A job shop scheduling method based on competitive meta-reinforcement learning, comprising:

[0008] Generate a diverse task library, and by randomly setting different problem scales, processing time distributions and problem types, the task library contains workshop environments corresponding to various types of scheduling tasks; the scheduling task consists of a job set and a machine set, and each job contains a series of operations; the problem scale refers to the product of the number of machines and the number of jobs included in a job shop scheduling problem; the processing time distribution refers to the random distribution type based on which the processing time required for an operation to be processed on the corresponding machine is randomly sampled when the workshop environment is initialized; the problem type refers to the setting type of the problem to be solved when the workshop environment is initialized;

[0009] Build a scheduling model, which includes a policy network π set by a decision model based on the Actor-Critic framework θ 、Behavior strategy network π θold and evaluation network v φ ; Strategy network π θ The parameters θ and the evaluation network v in φ The parameters φ in the scheduling model constitute the parameters Θ = {θ, φ};

[0010] Train multiple scheduling models in parallel:

[0011] During training, for each scheduling model, a scheduling task is randomly selected from the task library. Use the behavioral strategy network π θold Interact with the environment to generate support sets Based on support set Perform meta-learning training, calculate the inner loop loss, and update the parameters of the scheduling model based on the inner loop loss; the updated parameters are recorded as Θ i ′={θ i ′,φ i '}; i represents the sequence number of the scheduled task;

[0012] When targeting a specific scheduling task The number of inner loop iterations reaches the target. For each scheduling model, each uses the updated parameters Θ on each task. i 'Construct a policy network with new parameters Interact with the environment to generate query sets Based on query set Perform meta-learning training, calculate outer-loop loss and meta-gradient, and perform outer-loop meta-updates on the scheduling model;

[0013] The best scheduling model in this cycle training is selected and marked as the preferred one, and the rest are failures; the parameters of the preferred one remain unchanged; the parameters of all failures are updated in the direction of the parameters of the preferred one;

[0014] After multiple cycles of training, a scheduling model with good learning effect is finally obtained.

[0015] Preferably, the parameters of all the losers are updated towards the parameters of the preferred ones as follows: Let the parameters of the losers be Θ l , the preferred parameter is Θ w ; then update the loser's parameter to Θ l +γ(Θ w -Θ l ), where γ is the adjustment factor.

[0016] Preferably, the types and quantities of scheduled tasks in the task library are updated regularly.

[0017] Preferably, the job shop environment adopts; disjunctive structure modeling; in the disjunctive graph, each operation is represented by a node, the sequence relationship between nodes is represented by directed arcs, and undirected arcs connect operations that need to be processed by the same machine; the scheduling process indicates the direction of all undirected arcs in the disjunctive graph, that is, indicates the execution order of operations that need to be processed by the same machine.

[0018] The present invention also provides a job shop scheduling system based on competitive meta-reinforcement learning, including a task library, a scheduling model, a meta-reinforcement learning training module, and an optimization module;

[0019] The task library is used to set a variety of different types of scheduling tasks corresponding to different workshop environments by randomly setting different problem scales, processing time distributions and problem types; the scheduling task consists of a job set and a machine set, and each job contains a series of operations; the problem scale refers to the product of the number of machines and the number of jobs included in a job workshop scheduling problem; the processing time distribution refers to the random distribution type based on which the processing time required for an operation to be processed on the corresponding machine is randomly sampled when the workshop environment is initialized; the problem type refers to the setting type of the problem to be solved when the workshop environment is initialized;

[0020] Scheduling model, based on the decision model of the Actor-Critic framework, including the policy network π θ 、Behavior strategy network π θold and evaluation network ν φ ; Strategy network π θ The parameters θ and the evaluation network ν in φ The parameters φ in the scheduling model constitute the parameters Θ = {θ, φ}; there are multiple scheduling models to be trained;

[0021] Meta-reinforcement learning training module, which trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library Use the behavioral strategy network π θold Interact with the environment to generate support sets Based on support set Perform meta-learning training, calculate the inner loop loss, and update the parameters of the scheduling model based on the inner loop loss; the updated parameters are recorded as Θ′ i ={θ′ i ,φ′ i}, i represents the sequence number of the scheduled task; when a specific scheduled task The number of inner loop iterations reaches the target. For each scheduling model, each uses the updated parameter Θ′ on each task. i Constructing a policy network with new parameters Interact with the environment to generate query sets Based on query set Perform meta-learning training, calculate outer-loop loss and meta-gradient, and perform outer-loop meta-updates on the scheduling model;

[0022] The optimization module is used to evaluate the scheduling model after each scheduling model completes parameter update. The scheduling model with the best evaluation result is marked as the preferred one in this stage, and the rest are failures. The parameters of the preferred one remain unchanged; the parameters of all failures are updated in the direction of the parameters of the preferred one;

[0023] The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, and finally a scheduling model with better learning effect is obtained.

[0024] Preferably, when the optimization module updates the parameters of the loser to the parameters of the optimization, the parameters of the loser are θ l , the preferred parameter is Θ w ; then update the loser's parameter to Θ l +γ(Θ w -Θ l ), where γ is the adjustment factor.

[0025] Beneficial effects:

[0026] (1) The job shop scheduling scheme based on competitive meta-reinforcement learning designed by the present invention and the job shop scheduling method based on competitive meta-reinforcement learning, by applying the meta-reinforcement learning method to the training of the scheduling model, enable the model to summarize common knowledge in different types of scheduling task data and enhance the adaptability to sudden new tasks, thereby achieving autonomous decision-making and continuous optimization under sudden task conditions.

[0027] (2) By training multiple scheduling models in parallel and using competitive parameter updates, the learning efficiency and training stability of the model are further improved.

[0028] (3) The present invention constructs a diversified task library corresponding to various task scenarios. By using data of different types of tasks in training, the model has the adaptability to specific structural problems, thereby enhancing the ability to quickly adapt and make scheduling decisions in scheduling tasks in different job shops. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the operation structure of a job shop scheduling method and system based on competitive meta-reinforcement learning provided by the present invention;

[0030] Figure 2 It is a schematic diagram of a diversified scheduling task library in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0031] Figure 3 It is a schematic diagram of a job shop environment modeling method based on a disjunctive graph in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0032] Figure 4 It is a schematic diagram of a decision model based on an Actor-Critic framework based on a disjunctive graph in a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0033] Figure 5It is a schematic diagram of a specific operation flow of a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention;

[0034] Figure 6 It is an operating effect diagram of an embodiment of a job shop scheduling method based on competitive meta-reinforcement learning provided by the present invention.

[0035] Figure 7 It is a block diagram of a job shop scheduling system based on competitive meta-reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0037] The present invention provides a job shop scheduling solution based on competitive meta-reinforcement learning. By applying the meta-reinforcement learning method to the training of the scheduling model, the model can summarize common knowledge in different types of scheduling task data and enhance its adaptability to sudden new tasks. In addition, by training multiple scheduling models in parallel and using competitive parameter updates, the learning efficiency and training stability of the model are further improved.

[0038] The present invention also constructs a diversified task library, which is composed of a group of different types of job shop scheduling tasks. The scheduling tasks have different problem scales, different processing time distributions, and different problem types, so that they correspond to different workshop environments. Through diversified settings, a wide range of scheduling task types are incorporated into the training data of the scheduling decision model, so that the model can summarize different types of data during the meta-training process and have adaptability to problems with specific structural characteristics such as job shop scheduling decisions, so as to quickly adapt to sudden changes in new tasks. By continuously updating the task types and quantities in the task library, the continuous optimization of the autonomous decision-making ability of the scheduling model can also be achieved.

[0039] Therefore, the design focus of the present invention is to design a diversified scheduling task library, a decision model based on the Actor-Critic framework, and a scheduling decision model training method based on competitive meta-reinforcement learning.

[0040] The diversified scheduling task library provided by the present invention describes a scheduling task based on three characteristics: problem scale, processing time distribution and problem type, and regards scheduling tasks with different problem scales, different processing time distributions or different problem types as different types of tasks. Figure 2As shown in the figure, the problem size refers to the product of the number of machines and the number of jobs in a job shop scheduling problem, such as 20×10, 50×20, etc. The processing time refers to the time required for an operation to be processed on the corresponding machine, and this time is sampled from a random distribution when the shop environment is initialized. This distribution is the processing time distribution, such as [1,99], [1,199], etc. The problem type refers to the setting type of the problem to be solved when the shop environment is initialized. For example, in the J / / Cmax problem, the machine sequence of each job is randomly generated; while in the J / 2SETS / Cmax problem, the machines are divided into two subsets of equal size. Each job must complete the operation on all machines in one subset before it can be transferred to the machine in the other subset for further processing. When generating a diversified scheduling task library, by randomly setting different problem sizes, processing time distributions and problem types, the task library contains shop environments corresponding to various types of tasks, which are used to perform meta-reinforcement learning training on the scheduling model.

[0041] The present invention adopts a job shop environment modeling method based on a disjunctive graph to describe the job shop environment using a graph structure. Figure 3 As shown in the figure, in the disjunctive graph, each operation is represented by a node, and the order relationship between nodes is represented by directed arcs to ensure that the operations are processed in the specified order, while the undirected arcs connect the operations that need to be processed by the same machine. The scheduling process is to indicate the direction of all undirected arcs in the disjunctive graph, that is, to indicate the execution order of operations that need to be processed by the same machine. The environment is described by using a graph structure, and the relationships between different workpieces, processing sequences, etc. are represented in the form of points-arcs, which is convenient for inputting data into the neural network. The decision model obtains the data required for training such as the state transition process by interacting with the job shop environment based on the disjunctive graph.

[0042] See also Figure 4 The decision model based on the Actor-Critic framework provided by the present invention uses two deep neural networks as the policy network Actor and the evaluation network Critic, where Actor is responsible for learning a policy function and selecting a scheduling action according to the current workshop status. The policy network Actor is further divided into the policy network π θ 、Behavior strategy network π θold The two parts are trained alternately, and the parameters of the behavior strategy network are assigned to the strategy network. The Critic is responsible for evaluating the value of the scheduling action and providing a reward signal to guide the Actor's action. Specifically, taking the simplest AC framework as an example, the Actor module needs to learn a strategy function π θ(a|s), which represents the probability distribution of selecting action a in state s. This strategy is usually represented by a parameterized neural network, and its parameters are recorded as θ. The goal of the actor is to maximize the expected cumulative reward, that is, to maximize the expected reward of the policy function. Since the probability of the actor selecting an action according to the current state is π θ (a|s), it can be updated according to the policy gradient method: Where α is the learning rate of policy gradient update, G t is the expected value of the accumulated reward from time t to a certain moment in the future, is the gradient of the objective function with respect to the network parameter θ, s t is the state at time t, a t is the action at time t. The Critic model needs to learn a Q function Q φ (s, a), used to evaluate the value of the state-action pair (s, a), which can also be represented by a parameterized neural network, denoted as φ. The goal of Critic is to estimate the value of the current state or action and provide a reasonable reward signal to guide the Actor's action selection. According to the value function method, the update method of Critic is as follows: Where, δt is the temporal difference (TD) error, is the gradient of the objective function to the network parameter φ. In the present invention, the update of Actor and Critic will depend on the inner loop and outer loop losses.

[0043] The scheduling model training method based on competitive meta-reinforcement learning provided by the present invention is carried out for different types of scheduling tasks, aiming to enhance the generalization ability of the scheduling model and improve the adaptation speed and decision-making quality of the model when facing new tasks. The process of the training method includes five steps: task library initialization, scheduling model initialization, adaptive update of inner loop parameters for specific tasks, outer loop parameter meta-update across multiple tasks, and competitive parameter update between scheduling models based on preset competitive indicators. Among them, the task library initialization will form workshop environments corresponding to different tasks according to different problem scales, processing time distributions and problem types, that is, different environmental data; multiple scheduling models will be initialized at the same time and trained in parallel; the inner loop update uses the support set part of the environmental data in the task library; the outer loop meta-update uses part of the query set data; and the competitive update directly modifies the scheduling model parameters without the need for environmental data.

[0044] See also Figure 1 and Figure 5 , a job shop scheduling method based on competitive meta-reinforcement learning provided by an embodiment of the present invention comprises the following steps:

[0045] Step S1, prepare the diverse task library T and scheduling model Θ = {θ, φ} required for the meta-training process.

[0046] The task library T contains many different types of scheduling tasks with different problem sizes, different data distributions, and different problem types. The scheduling model includes the policy network π θ 、Behavior strategy network π θold and evaluation network ν φ , π θ and π θold Corresponding to θ in the scheduling model Θ, it is Actor; the evaluation network v φ Corresponding to φ in the scheduling model Θ, is Critic. When updating Θ below, it includes updating θ and φ. At the same time, set the competition evaluation function V c .

[0047] Train multiple scheduling models in parallel, randomly initialize their parameters, set training data, including the number of operation nodes n×m, processing time distribution range, data set structure type, etc.; set algorithm hyperparameters, including inner loop learning rate α, outer loop learning rate β, competitive learning rate γ, maximum training cycle N and maximum number of inner loop updates K.

[0048] Step S2: If the training cycle t reaches the preset maximum training cycle N, the training is terminated. Otherwise, for each scheduling model, a task is randomly selected from the task library D. i represents the serial number of the scheduled task, which consists of a set of jobs J and a set of machines M. Each job contains a series of operations O. Using the Behavioral Strategy Network π θold Interact with the corresponding environment and collect data including the state transfer process (s n,t ,a n,t ,r n,t ,s n,t+1 ) and the generalized advantage estimate A n,t And other data, and add it to the support set The support set is similar to the training set in traditional deep learning algorithms. Through a specific loss function, the model can be updated based on gradient descent.

[0049] Step S3, using the support set Calculate the inner loop loss and conduct specific tasks Update the adaptive parameters: Θ′ represents the updated parameter, Θ′ i ={θ′ i ,φ′ iThe number of inner loop iterations k increases by 1. If k reaches the maximum number of inner loop updates K, then go to S4. Otherwise, return and re-execute S3.

[0050] Specifically, we first use the support set Compute the policy loss, which measures the new policy π θ With the old strategy π θold Differences:

[0051]

[0052] In the above formula, clip(·) is a clipping function, which is used to limit the strategy ratio value to the range of [1-∈, 1+∈], ∈ is the set range value, and A n,t is the generalized advantage estimate at time t in the nth training cycle.

[0053] Then calculate the value function loss, which aims to reduce the value function v φ The predicted value and the true value A n,t The gap between:

[0054]

[0055] We then calculate the entropy loss to encourage the policy to explore more actions:

[0056]

[0057] Among them, a n,t and n,t are the states of the actions at time t in the nth training cycle, and S(·) is the entropy function, which is used to measure the randomness of the strategy. The higher the entropy value, the stronger the randomness of the strategy, which helps to explore more possible effective actions; the lower the entropy value, the stronger the certainty of the strategy, which helps to obtain higher rewards using the current strategy.

[0058] Finally, the inner ring loss is obtained Among them, c v and S is the adjustment coefficient. Then perform specific tasks The adaptive parameters are updated as follows: is the gradient of the inner loop loss with respect to the scheduling model parameters Θ.

[0059] Step S4: each scheduling model is based on the adaptive parameter θ′ after the inner loop is updated for each specific task. i , pass the parameter to the policy network to form a policy network Use this strategy network Interact with the corresponding environment, collect data including state transition process and generalized advantage estimation, and form a query set Then, based on the query set Through a calculation process similar to step S3, the corresponding outer loop loss and meta-gradient can be obtained And update the outer ring element of the scheduling model:

[0060] S5, after each scheduling model completes the meta-update, each scheduling model is evaluated and the best scheduling model is selected.

[0061] In this step, after the model parameters are updated in this round of iteration, the competitive evaluation function V c The competition evaluation function can use the maximum completion time value obtained from the test data as an indicator, such as test accuracy, training loss, etc., to evaluate the training effect and compare the learning effects of various scheduling models. For example, the competition evaluation function can use the maximum completion time value obtained from the test data. For the evaluation of learning effects, for example, the test accuracy of the model on the test task can be used as a competition indicator.

[0062] According to the evaluation results, the scheduling model with the best evaluation performance is marked as the winner, and the remaining scheduling models are losers. For the winner, keep its parameter Θ w For the loser, its parameters are moved closer to the winner's parameters. The parameter update method to move closer to the winner's parameters can be: l ←Θ l +γ(Θ w -Θ l ). Where γ is the adjustment factor.

[0063] S6, repeatedly execute S2 to S5 operations until the number of cycles reaches the maximum training cycle N, and finally obtain multiple job shop scheduling decision models with strong adaptability.

[0064] like Figure 6 As shown, Figure 6The results of an example of the operation of a competitive meta-reinforcement learning-based job shop scheduling method provided by the present invention in a job shop scheduling environment with different problem sizes and different processing time distributions. In the experiment, the scheduling model was trained in an environment with a fixed problem size of 20×20 and a processing time distribution ranging from [1,29] to [1,169], while the test was conducted in an environment with a problem size ranging from 20×15 to 50×20 and a processing time distribution of [1,199]. The competitive meta-reinforcement learning-based job shop scheduling method (ML-FSJSS) provided by the present invention was compared with four classic heuristic rules and a scheduling method based on reinforcement learning. It can be seen from the results that the competitive meta-reinforcement learning-based job shop scheduling method provided by the present invention is superior to the comparison algorithm in terms of solution quality. At the same time, in an environment with 1000 operating nodes (i.e., a problem size of 50×20), the model can also quickly respond to new tasks and make scheduling decisions within 2.3 seconds. The results show that the method proposed in this invention can help the scheduling model quickly learn and adapt to new tasks, realize autonomous decision-making and continuous optimization under sudden task changes, and fully demonstrate the effectiveness of the method.

[0065] Based on the above method, the present invention also provides a job shop scheduling system based on competitive meta-reinforcement learning, which includes a task library, a scheduling model, a meta-reinforcement learning training module, and an optimization module;

[0066] The task library is used to set up various types of scheduling tasks corresponding to different workshop environments by randomly setting different problem sizes, processing time distributions and problem types.

[0067] Scheduling model, based on the decision model of the Actor-Critic framework, including the policy network π θ 、Behavior strategy network π θold and evaluation network v φ ; There are multiple scheduling models to be trained.

[0068] Meta-reinforcement learning training module, which trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library Use the behavioral strategy network π θold Interact with the corresponding environment and collect data to form a support set Based on support set Perform meta-learning training, calculate the inner loop loss, and update the adaptive parameters based on the inner loop loss; when the number of inner loop iterations reaches the target, for each scheduling model, each uses the adaptive parameter Θ′ updated on each task i Constructing a policy network with new parameters Interact with the corresponding environment to collect data to form a query set Based on query set Perform meta-learning training, calculate outer-loop loss and meta-gradient, and perform outer-loop meta-updates on the scheduling model;

[0069] The optimization module is used to evaluate the scheduling model after each scheduling model completes the parameter update. The scheduling model with the best evaluation result is marked as the preferred one in this stage, and the rest are failures; the parameters of the preferred one remain unchanged; the parameters of all failures are updated in the direction of the parameters of the preferred one; in a preferred embodiment, when the optimization module updates the parameters of the failures in the direction of the parameters of the preferred one, the parameters of the failures are updated to Θ l ←Θ l +γ(Θ w -Θ l ), where the loser's parameter is Θ l , the preferred parameter is Θ w , γ is the adjustment factor.

[0070] The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, and finally a scheduling model with better learning effect is obtained.

[0071] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. A job shop scheduling method based on competitive meta-reinforcement learning, characterized in that: include: Generate a diverse task library, and by randomly setting different problem scales, processing time distributions and problem types, the task library contains workshop environments corresponding to various types of scheduling tasks; the scheduling task consists of a job set and a machine set, and each job contains a series of operations; the problem scale refers to the product of the number of machines and the number of jobs included in a job shop scheduling problem; the processing time distribution refers to the random distribution type based on which the processing time required for an operation to be processed on the corresponding machine is randomly sampled when the workshop environment is initialized; the problem type refers to the setting type of the problem to be solved when the workshop environment is initialized; Build a scheduling model, which includes a policy network π set by a decision model based on the Actor-Critic framework θ 、Behavior strategy network π θold and evaluation network ν φ ; Strategy network π θ The parameters θ and the evaluation network v in φ The parameters φ in the scheduling model constitute the parameters Θ = {θ, φ}; Train multiple scheduling models in parallel: During training, for each scheduling model, a scheduling task is randomly selected from the task library. Use the behavioral strategy network π θold Interact with the environment to generate support sets Based on support set Perform meta-learning training, calculate the inner loop loss, and update the parameters of the scheduling model based on the inner loop loss; the updated parameters are recorded as Θ′ i ={θ′ i ,φ′ i }; i represents the sequence number of the scheduled task; When targeting a specific scheduling task The number of inner loop iterations reaches the target. For each scheduling model, each uses the updated parameters Θ′ on each task. i Constructing a policy network with new parameters Interact with the environment to generate query sets Based on query set Perform meta-learning training, calculate outer-loop loss and meta-gradient, and perform outer-loop meta-updates on the scheduling model; The best scheduling model in this cycle training is selected and marked as the preferred one, and the rest are failures; the parameters of the preferred one remain unchanged; the parameters of all failures are updated in the direction of the parameters of the preferred one; After multiple cycles of training, a scheduling model with good learning effect is finally obtained.

2. The method according to claim 1, characterized in that The parameters of all losers are updated towards the parameters of the preferred ones as follows: Let the parameters of the losers be Θ l , the preferred parameter is Θ w ; then update the loser's parameter to Θ l +γ(Θ w -Θ l ), where γ is the adjustment factor.

3. The method according to claim 1, characterized in that Regularly update the types and quantities of scheduled tasks in the task library.

4. The method according to claim 1, characterized in that The job shop environment is modeled using a disjunctive structure. In the disjunctive graph, each operation is represented by a node, and the sequence relationship between nodes is represented by directed arcs, while undirected arcs connect operations that need to be processed by the same machine. The scheduling process indicates the direction of all undirected arcs in the disjunctive graph, that is, indicates the execution order of operations that need to be processed by the same machine.

5. A job shop scheduling system based on competitive meta-reinforcement learning, characterized in that: Includes task library, scheduling model, meta-reinforcement learning training module, and optimization module; The task library is used to set a variety of different types of scheduling tasks corresponding to different workshop environments by randomly setting different problem scales, processing time distributions and problem types; the scheduling task consists of a job set and a machine set, and each job contains a series of operations; the problem scale refers to the product of the number of machines and the number of jobs included in a job workshop scheduling problem; the processing time distribution refers to the random distribution type based on which the processing time required for an operation to be processed on the corresponding machine is randomly sampled when the workshop environment is initialized; the problem type refers to the setting type of the problem to be solved when the workshop environment is initialized; Scheduling model, based on the decision model of the Actor-Critic framework, including the policy network π θ 、Behavior strategy network π θold and evaluation network v φ ; Strategy network π θ The parameters θ and the evaluation network v in φ The parameters φ in the scheduling model constitute the parameters Θ = {θ, φ}; there are multiple scheduling models to be trained; Meta-reinforcement learning training module, which trains multiple scheduling models in parallel; during training, for each scheduling model, a scheduling task is randomly selected from the task library Use the behavioral strategy network π θold Interact with the environment to generate support sets Based on support set Perform meta-learning training, calculate the inner loop loss, and update the parameters of the scheduling model based on the inner loop loss; the updated parameters are recorded as Θ′ i ={θ′ i ,φ′ i }, i represents the sequence number of the scheduled task; when a specific scheduled task The number of inner loop iterations reaches the target. For each scheduling model, each uses the updated parameter Θ′ on each task. i Constructing a policy network with new parameters Interact with the environment to generate query sets Based on query set Perform meta-learning training, calculate outer-loop loss and meta-gradient, and perform outer-loop meta-updates on the scheduling model; The optimization module is used to evaluate the scheduling model after each scheduling model completes parameter update. The scheduling model with the best evaluation result is marked as the preferred one in this stage, and the rest are failures. The parameters of the preferred one remain unchanged; the parameters of all failures are updated in the direction of the parameters of the preferred one; The meta-reinforcement learning training module and the optimization module are trained for multiple cycles, and finally a scheduling model with better learning effect is obtained.

6. The system according to claim 5, characterized in that When the optimization module updates the parameters of the loser to the parameters of the optimizer, the parameters of the loser are θ l , the preferred parameter is Θ w ; then update the loser's parameter to Θ l +γ(Θ w -Θ l ), where γ is the adjustment factor.

Images