A distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning

Abstract

The application discloses a kind of distributed mixed flow shop scheduling method based on multi-agent deep reinforcement learning, belongs to the technical field of scheduling production workshop in manufacturing, to solve the problem that the technical generally exists slow response, the problem of low solving efficiency of distributed mixed flow shop scheduling.This application is for the optimization goal of minimum maximum completion time and minimum total energy consumption for distributed mixed flow shop scheduling problem, the method first constructs multi-agent neural network model by regarding each machine as agent, then uses the model to calculate and solve a large number of distributed mixed flow production examples, and uses experience database to save the action, reward and state change during training, then the experience database is randomly sampled to train each neural network, the model is tested using the verification example set during training, finally the trained model is used to solve the distributed mixed flow shop scheduling problem.

Description

Technical Field

[0001] This invention relates to a distributed hybrid flow shop scheduling method, specifically a scheduling optimization solution method based on multi-agent deep reinforcement learning, belonging to the field of scheduling and production planning technology in manufacturing workshops. Background Technology

[0002] Manufacturing is the foundation of my country's economic development, and building a globally competitive manufacturing sector is essential for my country to become a world-class power. Among these efforts, intelligent manufacturing, as a key focus in building a manufacturing powerhouse, directly impacts the quality of my country's manufacturing industry. Distributed manufacturing fully utilizes the resources of multiple enterprises or factories, employing rational division of labor and collaboration. Through the rational allocation, optimized combination, and sharing of resources, it achieves high-efficiency and high-return product manufacturing at low cost and low risk. However, distributed production scheduling requires the coordinated consideration of multiple highly coupled sub-problems. Ensuring the feasibility of the generated scheduling scheme while achieving collaborative and global optimization of various decision-making processes leads to higher computational complexity. Therefore, considering how to allocate workpieces to factories and formulate production scheduling plans within each factory to minimize the maximum completion time and reduce total energy consumption across all factories, thereby achieving high-efficiency, high-quality, low-cost, and low-energy workshop operations under a distributed hybrid flow production model, has become a very real and urgent problem for manufacturing enterprises, and a hot topic in academic research for many years.

[0003] Currently, methods commonly used to solve distributed hybrid production scheduling include metaheuristic algorithms, neighborhood search operators, and various evolutionary algorithms. However, these methods are difficult to respond quickly to problems with large amounts of information, such as distributed hybrid production scheduling. In other words, existing technologies for solving distributed hybrid production scheduling generally suffer from slow response speed and low solution efficiency. Summary of the Invention

[0004] To address the problems of slow response speed and low solution efficiency in existing technologies for scheduling distributed hybrid flow shops, this invention provides a distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning.

[0005] The present invention discloses a distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, which includes the following steps:

[0006] Step S1: Initialize the multi-agent system and build neural networks for each agent;

[0007] Step S2: Initialize neural network parameters and optimizer, and initialize sample replay set;

[0008] Step S3: training the neural network, the steps of training the network are as follows:

[0009] Step of selecting a distributed hybrid flow shop scheduling example and reading its information;

[0010] Step of inputting the production state of the current machine into the decision network to obtain the decision network output, and inputting the workpiece production information into the evaluation network to obtain the workpiece feature output;

[0011] Step of selecting actions for each agent by comparing the decision network output and the workpiece feature output;

[0012] Step of inputting the current production state and the actions taken by each agent into the evaluation network to obtain the evaluation network output;

[0013] Step of each agent executing the selected action and updating the production state;

[0014] Step of calculating the reward value obtained by each agent and storing samples;

[0015] Step of calculating the loss function value of the decision network according to the evaluation network output and updating the parameters of the decision network, and calculating the loss function value of the evaluation network according to the reward value of each agent and updating the parameters of the evaluation network;

[0016] Step of soft updating the parameters of the target decision network and the target evaluation network;

[0017] Step S4: determining whether the number of neural network training iterations meets the termination condition given by the initialized parameters in step S2, yes, then executing step S5; otherwise, returning to execute step S3;

[0018] Step S5: using the trained neural network to solve the distributed hybrid flow shop scheduling problem, decoding the obtained scheduling scheme to obtain the production Gantt chart.

[0019] Preferably, the process of initializing multiple agents and building neural networks in step S1 is as follows:

[0020] Step S1-1, regarding each machine as an agent;

[0021] Step S1-2, building a decision network, a target decision network, an evaluation network and a target evaluation network for each agent;

[0022] Step S1-3, the initial parameters of the decision network and the evaluation network of each agent are generated by a random strategy, and the parameters of the target decision network and the target evaluation network are the same as those of the decision network and the evaluation network.

[0023] Preferably, the step S2 of initializing the neural network parameters, the optimizer comprises:

[0024] The decision network learning rate actor_lr, the evaluation network learning rate critic_lr, the sample size memory_size, the training batch size batch_size, the validation data set size num_validation, the upper limit of the number of training epochs num_epsiode, the loss function weight γ and the soft update weight τ.

[0025] The initialization of the sample replay set comprises:

[0026] The original workpiece state sample old_job_state_memory, the new workpiece state sample new_job_state_memory, the original machine state sample old_machine_state_memory, the new machine state sample new_machine_state_memory, the original observation sample old_observation_memory, the new observation sample new_observation_memory, the action sample action_memory and the reward sample reward_memory.

[0027] Preferably, the step S3 of training the neural network comprises the following steps: firstly, a distributed hybrid flow shop scheduling example is selected, and its information is read; secondly, the current production state of each machine is input into the decision network of the corresponding agent to obtain the decision network output, and the production information of the workpiece is input into the evaluation network to obtain the workpiece feature output; subsequently, the action of each agent is selected by comparing the decision network output and the workpiece feature output, that is, the next process of each machine is selected; then, the production state of all machines, the production information of all workpieces and the action of each agent are input into the evaluation network to obtain the evaluation network output; then, each agent executes the selected action, and the production state is updated; subsequently, the reward value obtained by each agent is calculated; then, the original workpiece state, the original machine state, the original observation, the new workpiece state, the new machine state, the new observation, the action and the reward are stored as a sample; then, the loss function value of the decision network is calculated according to the evaluation network output, and the parameters of the decision network are updated based on this; the loss function value of the evaluation network is calculated according to the reward value of each agent, and the parameters of the evaluation network are updated based on this; finally, the parameters of the target decision network and the target evaluation network are soft updated to obtain the specific implementation process of the decision network output and the workpiece feature output.

[0028] The step S3A-1 of inputting the machine production state vector s iThe decision network of the corresponding agent is inputted, and the output result of the decision network of the agent i is denoted as wherein m is the number of machines, and θ i is the parameter of the decision network.

[0029] In step S3A-2, the vector [p j1 ,p j2 ,...,p jk ] including the machining time information of each process of the workpiece is inputted into the evaluation network, and the workpiece feature y j is obtained through the calculation of the RNN part of the evaluation network, wherein p jk is the machining time of the kth process of the workpiece j, and y j is the workpiece feature of the workpiece j. The step is repeated until the features of all workpieces are obtained.

[0030] Preferably, in step S3, the action of each agent is selected by comparing the output of the decision network and the output of the workpiece feature, and the specific implementation process is as follows:

[0031] In step S3B-1, for each agent, the set of workpieces that can be machined is screened according to the factory and the process stage where the agent is located, and in combination with the machining progress of each workpiece.

[0032] In step S3B-2, the workpiece with the closest value is selected as the next process of the agent to be machined by comparing the output value of the decision network of the agent with the feature value of the machinable workpiece, and if the set of machinable workpieces is empty, the action selected by the agent is empty, i.e. no workpiece to be machined.

[0033] In step S3B-3, it is judged whether all agents have completed the action selection, yes, then step S3B-4 is executed, otherwise step S3B-1 is executed.

[0034] In step S3B-4, it is checked whether there is a conflict in the actions selected by all agents, i.e. whether there are multiple agents selecting the same workpiece as the next process, yes, then step S3B-5 is executed, otherwise step S3B-6 is executed.

[0035] In step S3B-5, for the workpiece with conflict, one with the closest value between the output value of the decision network and the feature value is selected from all machines that select it as the next process, and the remaining machines will reselect the process according to the description of step S3B-2, and return to step S3B-4.

[0036] In step S3B-6, the next process to be machined by each agent is returned.

[0037] Preferably, in step S3, the evaluation network output is obtained by inputting the current production state and the action taken by each agent into the evaluation network, and the specific implementation process is as follows:

[0038] Step S3C-1, the workpiece processing time information, the current state information x of all machines and the action vector taken by each agent are spliced and integrated as the input of the evaluation network of each agent;

[0039] Step S3C-2, the evaluation network of each agent obtains the evaluation score of the corresponding decision network selecting the corresponding workpiece as the workpiece to be processed under the current production state through calculation wherein, w i is the parameter of the evaluation network of agent i;

[0040] Step S3C-3, each evaluation network transmits the evaluation score calculated by it to the decision network of the corresponding agent.

[0041] Preferably, the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning in step S3, characterized in that each agent in step S3 performs the selected action and updates the production state, and the specific implementation process is as follows:

[0042] Step S3D-1, for each agent, if the selected process is the first process of a workpiece, the start processing time of the process is the earliest available time of the machine corresponding to the agent, otherwise, the start processing time of the process is the maximum of the earliest available time of the machine corresponding to the agent and the completion time of the previous process of the process, and the start processing time of the selected process is updated according to the above description;

[0043] Step S3D-2, the end processing time of the selected process is updated as the sum of the start processing time and the processing time required by the process;

[0044] Step S3D-3, it is judged whether the processing time and the completion time of the process selected by all agents have been updated, if yes, the subsequent steps are executed, otherwise, the step S3D-1 is returned.

[0045] Preferably, the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning in step S3, characterized in that the reward value obtained by each agent is calculated in step S3, and the original state, the action taken by each agent, the reward value, and the new state are stored as a sample, and the specific implementation process is as follows:

[0046] Step S3E-1, if the action selected by the agent is a workpiece, the reward value r obtained by the agent in the current state selecting the corresponding workpiece for processing is calculated according to the following formula i :

[0047]

[0048] where total load i is the total working time of the machine corresponding to agent i after completing the processing of the selected job, total load i is the total working time of the machine corresponding to agent i before starting the processing of the selected job, earliest avail i is the earliest available time of the machine corresponding to agent i after completing the processing of the selected job, earliest avail i is the earliest available time of the machine corresponding to agent i before starting the processing of the selected job, max i∈{1,2,...,m} (earliest avail i ) represents the maximum value of the earliest available time of all machines after completing the processing of the selected job.

[0049] Step S3E-2, if the action selected by the agent is empty, the reward value r i is calculated according to the following formula:

[0050] Steps S3E-1 and S3E-2 are repeated until the reward values of all agents are calculated.

[0051] Step S3E-3, the original job state, the original machine state and the original observation sample of all agents before starting the processing of the selected job are stored in the original job state sample old_job_state_memory, the original machine state sample old_machine_state_memory and the original observation sample old_observation_memory respectively, the new job state, the new machine state and the new observation sample of all agents after completing the processing of the selected job are stored in the new job state sample new_job_state_memory, the new machine state sample new_machine_state_memory and the new observation sample new_observation_memory respectively, and the actions taken by all agents and the rewards obtained are stored in the action sample action_memory and the reward sample reward_memory respectively.

[0052] Preferably, the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning in step S3, characterized in that, the loss function value of the decision network is calculated according to the output of the evaluation network, and the parameters of the decision network are updated, the loss function value of the evaluation network is calculated according to the reward value of each agent, and the parameters of the evaluation network are updated, and the specific implementation process is as follows:

[0053] Step S3F-1, the loss function of the decision network is calculated according to the following formula

[0054] wherein Mean(·) denotes taking the average;

[0055] Step S3F-2, decision network by backpropagation of the loss function , the parameters θ i are updated;

[0056] Step S3F-3, the loss function of the evaluation network is calculated according to the following formula

[0057] wherein MSE(·) denotes the mean square error, γ is the loss function weight, a1,..., a m ∈action_memory are all the actions taken by the agents, x'∈new_job_state_memory∩new_machine_state_memory is the new state after the agents execute the actions, denotes the target evaluation network of the agent i, w i ' is the parameter of the target evaluation network of the agent i, denotes the target decision output of the target decision network of the agent i under the new observation sample s' i , θ i ' is the parameter of the target decision network of the agent i;

[0058] Step S3F-4, evaluation network by backpropagation of the loss function , the parameters w i are updated.

[0059] Preferably, the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning in step S3, characterized in that the parameters of the target decision network and the target evaluation network are soft updated in step S3, and the specific implementation process is as follows:

[0060] Step S3G-1, the parameters θ i ' of the target decision network are updated according to the following formula:

[0061] θ i ' = τ × θ i ' + (1 - τ) × θ i , wherein τ is the soft update weight;

[0062] Step S3G-2, the parameters w i ' of the target decision network are updated according to the following formula:

[0063] w i '=τ×w i '+(1-τ)×w i .

[0064] The application can start from the basic principle of distributed hybrid flow shop scheduling, consider the factory allocation limit of workpieces, the sequence limit of process processing and the feasibility limit of processing machines, first build a multi-agent neural network model by taking each machine as an intelligent agent, then use the model to calculate and solve a large number of distributed hybrid flow production examples, save the action, reward and state change during the training process using the experience library, then train each neural network by randomly sampling the experience library, use the verification example set to test the model during the training process, and finally use the trained model to solve the distributed hybrid flow shop scheduling problem. The method is a multi-agent collaborative optimization scheduling strategy of centralized training and decentralized decision making, which can greatly improve the efficiency of solving the distributed hybrid flow shop scheduling problem, so as to obtain a job shop scheduling scheme that minimizes the maximum completion time and total energy consumption at the same time in a very short time, and improve the efficiency and reliability of workshop production and processing under the distributed hybrid flow production mode. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flow chart of the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to the application;

[0066] Figure 2 is a schematic diagram of the multi-agent neural network framework built;

[0067] Figure 3 is a flow chart of selecting actions for each agent according to the application;

[0068] Figure 4 is a training curve diagram of the neural network in the 3x10x2 example according to the application;

[0069] Figure 5 is a Gantt chart for solving the 3x10x2 example according to the application. DETAILED DESCRIPTION

[0070] The application provides a distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, and belongs to the technical field of scheduling and production of manufacturing workshops. For a distributed hybrid flow shop scheduling problem with the optimization objectives of minimum maximum completion time and minimum total energy consumption, the method first builds a multi-agent neural network model by taking each machine as an agent, then uses the model to calculate and solve a large number of distributed hybrid flow production examples, saves the action, reward and state change during the training process by using an experience library, trains each neural network by randomly sampling the experience library, uses a verification example set to test the model during the training process, and finally uses the trained model to solve the distributed hybrid flow shop scheduling problem. The method is a multi-agent collaborative optimization scheduling strategy with centralized training and decentralized decision-making, which can greatly improve the efficiency of solving the distributed hybrid flow shop scheduling problem, so as to obtain a job shop scheduling scheme that simultaneously minimizes the maximum completion time and the total energy consumption in a very short time, and improve the efficiency and reliability of workshop production under the distributed hybrid flow production mode.

[0071] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0072] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0073] The application will be further described below with reference to the drawings and specific embodiments, but is not limited to the application.

[0074] Specific embodiment one: the following will be described with reference to the drawings Figures 1 to 5 The application provides a distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, and belongs to the technical field of scheduling and production of manufacturing workshops. For a distributed hybrid flow shop scheduling problem with the optimization objectives of minimum maximum completion time and minimum total energy consumption, the method first builds a multi-agent neural network model by taking each machine as an agent, then uses the model to calculate and solve a large number of distributed hybrid flow production examples, saves the action, reward and state change during the training process by using an experience library, trains each neural network by randomly sampling the experience library, uses a verification example set to test the model during the training process, and finally uses the trained model to solve the distributed hybrid flow shop scheduling problem. The method is a multi-agent collaborative optimization scheduling strategy with centralized training and decentralized decision-making, which can greatly improve the efficiency of solving the distributed hybrid flow shop scheduling problem, so as to obtain a job shop scheduling scheme that simultaneously minimizes the maximum completion time and the total energy consumption in a very short time, and improve the efficiency and reliability of workshop production under the distributed hybrid flow production mode.

[0075] Step S1: initialize the multi-agent, and build a neural network for each agent;

[0076] The distributed hybrid flow shop scheduling problem can be described as follows: n jobs need to be completed on m machines, each job needs to be processed in s processes, and all machines are distributed in F different factories. For each process, there is one or more available machines in each factory. In the process of making production plans, for each job, the decision maker needs to assign it to a factory first, then assign a processing machine to each process of the job in the factory, and decide the processing order of the processes on each machine in the factory. Therefore, to solve the distributed hybrid flow shop scheduling problem, we need to solve the factory allocation, machine selection, and process ordering sub-problems, and finally minimize the maximum completion time and total energy consumption of all factories.

[0077] The process of initializing multiple agents and building a neural network in step S1 is as follows:

[0078] Step S1-1, each machine is regarded as an agent;

[0079] Step S1-2, a decision network, a target decision network, an evaluation network, and a target evaluation network are built for each agent;

[0080] Step S1-3, the initial parameters of the decision network and the evaluation network of each agent are generated by a random strategy, and the parameters of the target decision network and the target evaluation network are the same as those of the decision network and the evaluation network, respectively;

[0081] Taking a data set containing 3 factories and 10 jobs as an example, each job includes 2 processes, and the processing time required by each process of each job is shown in Table 1. The number of machines in different factories and the energy consumption of each machine per unit time are shown in Table 2. Among them, factory F1 has 1 machine that can process the first process and 3 machines that can process the second process. Similarly, factory F2 has 3 machines that can process both processes, and factory F3 has 3 and 2 machines, respectively.

[0082] Table 1

[0083]

[0084] Table 2

[0085]

[0086] Taking the data set given in Table 1 and Table 2 as an example, the neural network model built by this method includes 15 agents, each agent contains a decision network, a target decision network, an evaluation network, and a target evaluation network, so the neural network model contains 60 neural networks.

[0087] Step S2: initialize neural network parameters, optimizer, initialize sample replay set.

[0088] The initializing of neural network parameters, optimizer includes:

[0089] The decision network learning rate actor_lr, the evaluation network learning rate critic_lr, the sample size memory_size, the training batch size batch_size, the validation dataset size num_validation, the training epoch upper limit num_epsiode, the loss function weight γ and the soft update weight τ.

[0090] Taking the dataset given in Table 1 and Table 2 as an example, the method sets the decision network learning rate actor_lr to 0.00001, the evaluation network learning rate critic_lr to 0.00001, the sample size memory_size to 10000, the training batch size batch_size to 32, the validation dataset size num_validation to 100, the training epoch upper limit num_epsiode to 600, the loss function weight γ to 0.95, and the soft update weight τ to 0.01, and uses the Adam optimizer to train all neural networks.

[0091] The initializing of sample replay set includes:

[0092] The original workpiece state sample old_job_state_memory, the new workpiece state sample new_job_state_memory, the original machine state sample old_machine_state_memory, the new machine state sample new_machine_state_memory, the original observation sample old_observation_memory, the new observation sample new_observation_memory, the action sample action_memory and the reward sample reward_memory.

[0093] Step S3: train the neural network, and the steps of training the network are as follows:

[0094] The step of selecting a distributed hybrid flow shop scheduling example and reading its information;

[0095] The step of inputting the production state of the current machine into the decision network to obtain the decision network output, and inputting the workpiece production information into the evaluation network to obtain the workpiece feature output;

[0096] The step of selecting an action for each agent by comparing the decision network output and the workpiece feature output;

[0097] a step of inputting the current production state and the action taken by each agent into the evaluation network to obtain an evaluation network output;

[0098] a step of each agent performing the selected action and updating the production state;

[0099] a step of calculating the reward value obtained by each agent and storing a sample;

[0100] a step of calculating a loss function value of the decision network according to the evaluation network output and updating the parameters of the decision network, and calculating a loss function value of the evaluation network according to the reward value of each agent and updating the parameters of the evaluation network;

[0101] a step of soft updating the parameters of the target decision network and the target evaluation network;

[0102] The step of training the neural network in step S3: first, select a distributed hybrid flow shop scheduling example and read its information, second, input the current production state of each machine into the decision network of the corresponding agent to obtain a decision network output, input the production information of the workpiece into the evaluation network to obtain a workpiece feature output, then, select an action for each agent by comparing the decision network output and the workpiece feature output, that is, select the next process for each machine, then, input the production state of all machines, the production information of all workpieces and the action of each agent into the evaluation network to obtain an evaluation network output, then, each agent performs the selected action and the production state is updated, then, calculate the reward value obtained by each agent, then, store the original workpiece state, the original machine state, the original observation, the new workpiece state, the new machine state, the new observation, the action and the reward as a sample, then, calculate the loss function value of the decision network according to the evaluation network output and update the parameters of the decision network based on this, calculate the loss function value of the evaluation network according to the reward value of each agent and update the parameters of the evaluation network based on this, finally, soft update the parameters of the target decision network and the target evaluation network, and the specific implementation process of the decision network output and the workpiece feature output is as follows:

[0103] Step S3A-1, input the machine production state vector s i into the decision network of the corresponding agent, and the output result of the decision network of agent i is denoted as where m is the number of machines, θ i is the parameter of the decision network;

[0104] Step S3A-2, input the vector [p j1 ,p j2 ,...,pjk The workpiece features y are obtained by calculating the RNN part of the evaluation network j where p is the processing time of the kth process of workpiece j, and y is the workpiece feature of workpiece j jk j The step is repeated until the features of all workpieces are obtained.

[0105] For example, the feature y1 of workpiece J1 is calculated by the RNN part of the evaluation neural network according to the input processing time information vector [32, 37], and the features of the remaining workpieces are obtained in the same way.

[0106] The actions of each agent are selected by comparing the output of the decision network and the workpiece feature output in step S3, and the specific implementation process is as follows:

[0107] Step S3B-1, for each agent, according to its factory and process stage, and combined with the processing progress of each workpiece, the set of workpieces that can be processed is screened;

[0108] Step S3B-2, by comparing the output value of the decision network of the agent with the feature value of the workpieces that can be processed, the workpiece with the closest value is selected as the next process of the agent to be processed, and if the set of workpieces that can be processed is empty, the action selected by the agent is empty, i.e. no workpiece to be processed;

[0109] Step S3B-3, determine whether all agents have completed action selection, yes, then execute step S3B-4, otherwise execute step S3B-1;

[0110] Step S3B-4, check whether the actions selected by all agents conflict, i.e. whether there are multiple agents selecting the same workpiece as the next process, yes, then execute step S3B-5, otherwise execute step S3B-6;

[0111] Step S3B-5, for the workpiece with conflict, select the one with the closest output value of the corresponding decision network and its feature value from all machines that select it as the next process, and the remaining machines will select the process according to the description of step S3B-2, and return to step S3B-4;

[0112] Step S3B-6, return the next process of each agent to be processed.

[0113] The current production state and the actions taken by each agent are input into the evaluation network to obtain the evaluation network output in step S3, and the specific implementation process is as follows:

[0114] ​Step S3C-1, the workpiece processing time information, the current state information x of all machines and the action vector taken by each agent are spliced and integrated as the input of the evaluation network of each agent;

[0115] Step S3C-2, the evaluation network of each agent obtains the evaluation score of the corresponding decision network selecting the corresponding workpiece as the workpiece to be processed in the current production state through calculation Wherein, w i is the parameter of the evaluation network of agent i;

[0116] Step S3C-3, each evaluation network transmits the evaluation score calculated by it to the decision network of the corresponding agent.

[0117] In step S3, each agent executes the selected action to update the production state, and the specific implementation process is as follows:

[0118] Step S3D-1, for each agent, if the selected process is the first process of a workpiece, the start processing time of the process is the earliest available time of the machine corresponding to the agent, otherwise, the start processing time of the process is the maximum value of the earliest available time of the machine corresponding to the agent and the completion time of the previous process of the process, and the start processing time of the selected process is updated according to the above description;

[0119] Step S3D-2, the end processing time of the selected process is updated as the sum of the start processing time and the processing time required by the process;

[0120] Step S3D-3, it is judged whether the processing time and the completion time of the process selected by all agents have been updated, if yes, the subsequent step is executed, otherwise, the step S3D-1 is returned.

[0121] In step S3, the reward value obtained by each agent is calculated, and the original state, the action taken by each agent, the reward value and the new state are stored as a sample, and the specific implementation process is as follows:

[0122] Step S3E-1, if the action selected by the agent is a workpiece, the reward value r obtained by the agent in the current state selecting the corresponding workpiece for processing is calculated according to the following formula i :

[0123]

[0124] Wherein, total_load i is the total working time of the machine corresponding to agent i after completing the processing of the selected workpiece, total_load iearliest_avail i earliest_avail i max i∈{1,2,...,m} earliest_avail i ) represents the maximum value of the earliest available time after completing the processing of the selected workpiece among all machines;

[0125] If the action selected by the agent is empty, the reward value r is calculated according to the following formula: i :

[0126] Steps S3E-1 and S3E-2 are repeated until the reward values of all agents are calculated;

[0127] In step S3E-3, the original workpiece state, the original machine state and the original observation sample of all agents before starting the processing of the selected workpiece are stored in the original workpiece state sample old_job_state_memory, the original machine state sample old_machine_state_memory and the original observation sample old_observation_memory respectively, the new workpiece state, the new machine state and the new observation sample of all agents after completing the processing of the selected workpiece are stored in the new workpiece state sample new_job_state_memory, the new machine state sample new_machine_state_memory and the new observation sample new_observation_memory respectively, and the actions taken by all agents and the rewards obtained are stored in the action sample action_memory and the reward sample reward_memory respectively.

[0128] In step S3, the loss function value of the decision network is calculated according to the output of the evaluation network, and the parameters of the decision network are updated, the loss function value of the evaluation network is calculated according to the reward value of each agent, and the parameters of the evaluation network are updated, and the specific implementation process is as follows:

[0129] In step S3F-1, the loss function of the decision network is calculated according to the following formula:

[0130] Wherein, Mean(·) represents the average value;

[0131] In step S3F-2, the loss function of the decision network is calculated. backpropagation, the parameters θ i in the evaluation network are updated.

[0132] Step S3F-3, the loss function of the evaluation network is calculated according to the following formula

[0133] wherein, MSE(·) represents the mean square error, γ is the loss function weight, a1,..., a m ∈action_memory is the action taken by all agents, x'∈new_job_state_memory∩new_machine_state_memory is the new state after the action is executed by all agents, represents the target evaluation network of the agent i, w i ' is the parameter of the target evaluation network of the agent i, represents the target decision output of the target decision network of the agent i under the new observation sample s' i i ' is the parameter of the target decision network of the agent i;

[0134] Step S3F-4, the evaluation network is updated through the backpropagation of the loss function , and the parameters w i in the evaluation network are updated.

[0135] The soft update of the parameters of the target decision network and the target evaluation network in step S3 is implemented as follows:

[0136] Step S3G-1, the parameters θ i ' of the target decision network are updated according to the following formula:

[0137] θ i '=τ×θ i '+(1-τ)×θ i , wherein τ is the soft update weight;

[0138] Step S3G-2, the parameters w i ' of the target decision network are updated according to the following formula:

[0139] w i '=τ×w i '+(1-τ)×w i .

[0140] Step S4: determining whether the neural network training algebra satisfies the termination condition given by the initialized parameters in step S2, if yes, executing step S5; if no, returning to execute step S3.​

[0141] Step S5: using the trained neural network to solve the distributed hybrid flow shop scheduling problem, decoding the obtained scheduling scheme to obtain a production Gantt chart.

[0142] In order to verify the feasibility of the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, the model built in the present application is trained on an example set with a production scale of 3*10*2, and the trained model is tested on 100 verification examples with a scale of 3*10*2.

[0143] The average values of the maximum completion time and the total energy consumption obtained by the multi-agent deep reinforcement learning model built in the present application for solving 100 verification examples are shown in the curve of the change with the training generation number as shown in Figure 4 As can be seen from the curve trend, the target function values obtained by the model proposed in the present application are decreasing with the increase of the training generation number, and the convergence of the model is thus verified.

[0144] In order to verify the performance of the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, the trained model is compared with four classical priority scheduling rules, first in first out (FIFO), most operation remain (MOR), shortest processing time (SPT) and most work remain (MWKR) on 100 test examples with a scale of 3*10*2, wherein, in order to reduce the randomness of the experimental results, the four priority scheduling rules are repeatedly run 10 times on each example, and the average value is taken as the running result.

[0145] The average values of the maximum completion time and the total energy consumption obtained by the model proposed in the present application and the four priority scheduling rules for solving 100 test examples are shown in Table 3, and the scheduling scheme obtained by the method proposed in the present application on one of the test examples is shown in Figure 5The Gantt chart of the three distributed factories is shown in Figure 6, which shows the scheduling of workpieces in three distributed factories, each rectangle represents a process, and the number marked on the rectangle represents the workpiece number of the process, and the vertical coordinate of each Gantt chart represents the machine number, M11 represents the first machine capable of processing the first process, M12 represents the second machine capable of processing the first process, and so on. For example, workpiece 1, workpiece 5 and workpiece 7 are allocated to factory 1 for processing, the first process of workpiece 1 is processed on the second machine capable of processing the process in factory 1, and the second process is processed on the fourth machine capable of processing the process in factory 1. The scheduling scheme Gantt chart of the three factories is combined, which is the overall Gantt chart of the distributed hybrid flow shop scheduling example.

[0146] Table 3

[0147]

[0148] As can be seen from Table 3, compared with FIFO, MOR, SPT and MWKR, the scheduling result of the method proposed in the application is more prominent in terms of target value, the comprehensive performance is more excellent, the optimization effect is better, and the stability is higher. In summary, the distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning disclosed in the application can effectively shorten the construction period and ensure the production efficiency under the distributed hybrid flow shop production mode.

[0149] Although the application is described herein with reference to particular embodiments, it should be understood that these examples are merely set forth for purposes of example and illustration. As such, many modifications and variations will be apparent to those skilled in the art, and it is contemplated to be within the scope of the claimed application to embrace all such modifications, variations and arrangements. It is intended that the scope of the application be defined by the claims appended hereto rather than by the description appearing in the specification. It is intended that the features of the individual embodiments described herein can be used in other embodiments.

Claims

1. A distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning, characterized in that, The method comprises the following steps: Step S1: initializing multiple agents and building a neural network for each agent; Step S2: initializing neural network parameters, an optimizer, and initializing a sample replay set; Step S3: training the neural network, and the steps of training the network are as follows: a step of selecting a distributed hybrid flow shop scheduling example and reading information thereof; a step of inputting the production state of the current machine into the decision network to obtain a decision network output and inputting workpiece production information into the evaluation network to obtain a workpiece feature output; the specific implementation process is as follows: Step S3A 1. Machine production state vector including total working time, total idle time, earliest available time and total energy consumption information of each machine Input the decision network of the corresponding agent, and the agent The output result of the decision network of the agent is denoted as Wherein, M is the number of machines, is the parameter of the decision network; Step S3A 2, a vector including machining time information of each process of the workpiece is input into the evaluation network, and the workpiece feature is obtained through the calculation of the RNN part in the evaluation network , wherein, is the workpiece , and the machining time of the first process of the workpiece , and the workpiece feature of the workpiece is repeated until the features of all workpieces are obtained; a step of selecting an action for each agent by comparing the decision network output and the workpiece feature output; the specific implementation process is as follows: Step S3B 1. For each agent, according to the factory it is in and the process stage, and in combination with the processing progress of each workpiece, a set of workpieces that can be processed is screened; Step S3B 2. By comparing the output value of the intelligent agent decision network with the characteristic value of the workpiece, the workpiece with the closest value is selected as the next process of the intelligent agent, and if the set of workpieces is empty, the action selected by the intelligent agent is empty, i.e. no workpiece to be processed. Step S3B 3, if yes, go to step S3B 4, else go to step S3B 1; Step S3B 4. Check if there is a conflict in the actions selected by all agents, i.e. if more than one agent has selected the same workpiece as the next workpiece, yes, then perform step S3B 5, no, then perform step S3B 6; Step S3B 5. For the workpiece with conflict, select one with the closest value to the output value of the corresponding decision network from all the selected machines for the work process, and the rest of the machines will select the work process according to the description of step S3B 2, return to step S3B 4; Step S3B 6. Return to the next process to be processed by each intelligent agent. a step of inputting the current production state and the action taken by each agent into the evaluation network to obtain an evaluation network output; a step of each agent executing the selected action and updating the production state; a step of calculating the reward value obtained by each agent and storing samples; the specific implementation process is as follows: Step S3E 1. If the action selected by the agent is a certain workpiece, the reward value obtained by the agent for processing the corresponding workpiece in the current state is calculated according to the following formula : wherein, is the agent total working time of the corresponding machine after finishing the processing of the selected workpiece, is the agent total working time of the corresponding machine before starting the processing of the selected workpiece, is the agent earliest available time of the corresponding machine after finishing the processing of the selected workpiece, is the agent earliest available time of the corresponding machine before starting the processing of the selected workpiece, denotes the maximum value of the earliest available time of all machines after finishing the processing of the selected workpiece; Step S3E 2. If the action selected by the agent is empty, the reward value is calculated according to the following formula : , repeat step S3E 1 with S3E 2, until the reward value of all agents is calculated; Step S3E 3. Store the original workpiece state, original machine state and original observation sample of all agents before starting processing the selected workpiece into original workpiece state sample , original machine state sample and original observation sample , respectively.

4. Store the new workpiece state, new machine state and new observation sample of all agents after completing processing the selected workpiece into new workpiece state sample , new machine state sample and new observation sample , respectively.

5. Store the actions taken by all agents and the rewards obtained into action sample and reward sample , respectively. a step of calculating the loss function value of the decision network according to the evaluation network output and updating the parameters of the decision network, and calculating the loss function value of the evaluation network according to the reward value of each agent and updating the parameters of the evaluation network; a step of soft updating the parameters of the target decision network and the target evaluation network; Step S4: determining whether the number of neural network training iterations meets the termination condition given by the initialized parameters in step S2; if yes, executing step S5; otherwise, returning to execute step S3; Step S5: using the trained neural network to solve the distributed hybrid flow shop scheduling problem, and decoding the obtained scheduling scheme to obtain a production Gantt chart.

2. The distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The process of initializing multiple agents and building a neural network for each agent in step S1 is as follows: Step S1 1. Each machine is considered as an agent. Step S1 2. Build a decision network, a goal decision network, an evaluation network and a goal evaluation network for each agent. Step S1 3. The initial parameters of the decision network and the evaluation network of each agent are generated by a random strategy, and the parameters of the target decision network and the target evaluation network are the same as those of the decision network and the evaluation network, respectively.

3. The distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that, Initializing neural network parameters and an optimizer in step S2 comprises: Decision network learning rate , Evaluation network learning rate , Sample size , Training batch size , Validation dataset size , Training epoch upper limit , Loss function weight and soft update weight; Initializing a sample replay set comprises: original workpiece state sample , new workpiece state sample , original machine state sample , new machine state sample , original observation sample , new observation sample , action sample , and reward sample .

4. The distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S3, the current production state and the action taken by each agent are input into the evaluation network to obtain an evaluation network output, and the specific implementation process is as follows: Step S3C 1, workpiece processing time information, all machine current state information And each agent to take the action vector to splice integration, as the input of each agent evaluation network; Step S3C 2. The evaluation network of each agent obtains the evaluation score of the corresponding decision network selecting the corresponding workpiece as the workpiece to be processed in the current production state through calculation , wherein, is the parameter of the evaluation network of the agent . Step S3C 3. Each evaluation network passes its computed evaluation score to the decision network of the corresponding agent.

5. The distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to claim 4, characterized in that, In step S3, each agent executes the selected action and updates the production state, and the specific implementation process is as follows: Step S3D 1. For each agent, if the selected process is the first process of a workpiece, the start processing time of the process is the earliest available time of the machine corresponding to the agent, otherwise, the start processing time of the process is the maximum of the earliest available time of the machine corresponding to the agent and the completion time of the predecessor process of the process, and the start processing time of the selected process is updated according to the above description. Step S3D 2. update the end processing time of the selected process to be the sum of the start processing time and the processing time required for the process; Step S3D 3. Determine whether the processing time and the completion time of the process selected by all the intelligent agents have been updated, yes, then execute the following step, otherwise, return to step S3D 1.

6. The method of claim 1, wherein, In step S3, the loss function value of the decision network is calculated according to the evaluation network output, and the parameters of the decision network are updated, the loss function value of the evaluation network is calculated according to the reward value of each agent, and the parameters of the evaluation network are updated, and the specific implementation process is as follows: Step S3F 1. Calculate the loss function of the decision network according to the following formula : wherein denotes averaging. Step S3F 2, decision network by backpropagation of the loss function the parameters in it are updated ; Step S3F 3. The loss function of the evaluation network is calculated according to the following formula : wherein, denotes the mean squared error, is a loss function weight, is the action taken by all agents, is the new state after all agents perform the action, denotes the target evaluation network of the agent , is the parameter of the target evaluation network of the agent , denotes the target decision output of the target decision network of the agent on the new observation sample , is the parameter of the target decision network of the agent ; Step S3F 4, Evaluate network The parameters in the network are updated by backpropagation through the loss function .

7. The distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning according to claim 6, characterized in that, In step S3, the parameters of the target decision network and the target evaluation network are soft updated, and the specific implementation process is as follows: Step S3G 1. The parameters of the target decision network are updated according to the following equation:

1. The parameters of the target decision network are updated according to the following equation: wherein, is a soft update weight; Step S3G 2. The parameters of the target decision network are updated according to the following equation: : 。

Images