Production line scheduling method and production line scheduling device

Abstract

The present disclosure proposes a production line scheduling method and a production line scheduling device, which relate to the field of production scheduling. By constructing a production line production process model of a production line instance and its flexible job shop scheduling problem, the flexible job shop scheduling problem is solved using a taboo search algorithm. In the solution process, the parameters of the taboo search algorithm are corrected using reinforcement learning, so that the algorithm can search the solution space more widely within a limited time, effectively improve the quality of the solution, and thus make the production line scheduling plan determined according to the solution result more reasonable, and the production line scheduling plan can be quickly and dynamically formulated or adjusted.

Description

Technical Field

[0001] The present disclosure relates to the field of production scheduling, and in particular to a production line scheduling method and a production line scheduling device. Background Art

[0002] Cigarette production is an important part of tobacco enterprises' production operations, especially under the requirements of lean production management. It is crucial to shorten the completion time by improving scheduling methods. Due to the volatility of cigarette production plan demand and the suddenness of production resource changes, a method is needed to quickly and effectively schedule demand tasks in a reasonable and dynamic manner. Summary of the invention

[0003] The disclosed embodiments construct a production line production process model of a production line instance and its flexible job shop scheduling problem, and use a taboo search algorithm to solve the flexible job shop scheduling problem. During the solution process, reinforcement learning is used to correct the parameters of the taboo search algorithm, so that the algorithm can search the solution space more widely within a limited time, effectively improving the quality of the solution, thereby making the production line scheduling plan determined according to the solution result more reasonable, and the production line scheduling plan can be quickly and dynamically formulated or adjusted.

[0004] Some embodiments of the present disclosure provide a production line scheduling method, including:

[0005] Construct a production line production process model for a production line instance;

[0006] Constructing a flexible job shop scheduling problem for a production line production process model;

[0007] The tabu search algorithm is used to solve the flexible job shop scheduling problem. During the solution process, reinforcement learning is used to modify the parameters of the tabu search algorithm.

[0008] According to the solution of the flexible job shop scheduling problem, the production line scheduling plan is determined.

[0009] In some embodiments, the production line production process model includes: n workpieces J = {J 1 ,J 2 ,…,J n}On m machines M={M 1 ,M 2 ,…,M m}Processing process, each workpiece J i Process several processes according to the set process route ij ,j∈{1,2,…,r i}, where r i Indicates workpiece J i The total number of processes.

[0010] In some embodiments, the objective function of the flexible job shop scheduling problem is to minimize the maximum completion time of all workpieces; the constraints of the flexible job shop scheduling problem include: the completion time of each workpiece is less than or equal to the maximum completion time; each processing operation can only be processed by one machine in the set of available machines; the completion time of the process is not less than the sum of the start time and the processing time; and, on the same machine, the end time of the previous process plus the preparation time of the subsequent process is not greater than the start time of the subsequent process.

[0011] In some embodiments, solving the flexible job shop scheduling problem using a taboo search algorithm includes:

[0012] Generate an initial solution randomly and set the taboo table to be empty;

[0013] Determine whether the termination condition is met. If so, output the optimal solution and end the taboo search algorithm. If not, use the neighborhood function of the current solution to generate several neighborhood solutions and determine several candidate solutions from them.

[0014] Determine whether the candidate solution meets the contempt criterion. If so, replace the current solution with the best state that meets the contempt criterion as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution, and continue the iterative search process. If not, determine the taboo status of each object corresponding to the candidate solution, select the best state of the non-taboo object in the candidate solution set as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution;

[0015] Among them, the parameters of the taboo search algorithm modified by reinforcement learning include: the length of the taboo table and the number of candidate solutions.

[0016] In some embodiments, modifying parameters of the tabu search algorithm using reinforcement learning includes:

[0017] Initialize the parameters of the online network, the parameters of the target network, the parameters of the tabu search algorithm, the set of actions that the agent can choose, and define the state of each tabu search sub-cycle;

[0018] Select a production line instance and get the initial state s 0 ;

[0019] Based on the initial state s 0 Select action a 0 , according to action a 0 Modify the parameters of the tabu search algorithm and run the tabu search algorithm;

[0020] Determine the evolution state of the tth sub-cycle of the tabu search algorithm to obtain the state s t ;

[0021] Based on the status t Select the corresponding action a t , according to action a t Corrected parameters of the tabu search algorithm;

[0022] Run a tabu search sub-cycle and determine the evolution state of the tabu search algorithm's t+1th sub-cycle to obtain the state s t+1 ;

[0023] Calculate action a t The reward value r at the current state t ;

[0024] Storage experience(s) t ,a t ,r t ,s t+1 ) to the experience playback memory D;

[0025] Randomly sample B experiences (s) from the experience replay memory D t′ ,a t′ ,r t′ ,s t′+1 ), calculate the target Q value based on the target network, use the gradient descent method to update the parameters of the online network, and use the parameters of the online network to update the parameters of the target network every C time steps.

[0026] In some embodiments, the number of actions in the action set is determined jointly based on the number of intervals within the value range of the taboo table length of the taboo search algorithm and the number of intervals within the value range of the number of candidate solutions, and each action corresponds to an interval within the value range of the taboo table length of the taboo search algorithm and an interval within the value range of the number of candidate solutions.

[0027] In some embodiments, the evolution state of the t-th sub-cycle of the tabu search algorithm is determined to obtain the state s t Including: using the t-th sub-cycle evolution state and several previous sub-cycle evolution states of the taboo search algorithm to represent the state s t , wherein the evolutionary state of each sub-cycle is characterized by the static feature vector and dynamic feature vector corresponding to the sub-cycle, the static feature vector is configured to include: the total number of machines, the total number of workpiece types, the total number of processes, the mean number of processes, and the standard deviation of the number of processes, and the dynamic feature vector is configured to include: the number of current taboo search sub-cycles, the optimal fitness value of the sub-cycle, the ratio of the fitness value to the initial value, the mean value of the optimal solution within the most recent preset number of times, the standard deviation, and the ratio of the mean value of the optimal solution to the initial solution.

[0028] In some embodiments, based on the state s t Select the corresponding action a tIncluding: when selecting an action, randomly select an action with the probability of greedy rate, and select the state s with the probability of (1-greedy rate) t The action that maximizes the expected Q value is taken, where the greedy rate gradually decreases with the increase of training times.

[0029] In some embodiments, the calculation action a t The reward value r at the current state t Including: Determine the reward value r based on the difference between the minimum fitness value in the current training and the fitness value of the t-th sub-cycle in the current training t .

[0030] In some embodiments, calculating the target Q value includes: if s t′+1 is the final state, and the reward value r t′ Determine the target Q value for the t′th sub-cycle; if s t′+1 Not the final state, determined based on the online network in state s t′+1 Take the action a that maximizes the expected Q value t′+1 , based on the target network, determine the state s t′+1 Next action a t′+1 The Q value of the t′+1th sub-cycle is calculated based on the reward value r t′ The target Q value of the t′th sub-cycle is determined by summing up the Q value of the t′+1th sub-cycle under the discount factor.

[0031] In some embodiments, determining the production line scheduling plan includes: determining the production line scheduling plan according to the process of processing each workpiece on each machine so as to minimize the maximum completion time of all workpieces.

[0032] In some embodiments, the production line example comprises a tobacco production line example.

[0033] Some embodiments of the present disclosure provide a production line scheduling device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a production line scheduling method based on instructions stored in the memory.

[0034] Some embodiments of the present disclosure provide a computer-readable storage medium having computer instructions stored thereon, which implement a production line scheduling method when executed by a processor.

[0035] Some embodiments of the present disclosure provide a computer program product, including computer instructions, which implement a production line scheduling method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0037] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0038] Figure 1 A schematic flow chart showing a production line scheduling method according to some embodiments of the present disclosure.

[0039] Figure 2 A schematic flow chart showing a production line scheduling method according to some other embodiments of the present disclosure.

[0040] Figure 3 A schematic diagram showing a tabu search algorithm modified by reinforcement learning according to some embodiments of the present disclosure.

[0041] Figure 4 A schematic diagram showing a production line scheduling device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0043] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0044] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0045] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0046] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0047] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0048] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0050] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0051] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0052] In addition, in order to avoid obscuring the present disclosure due to unnecessary details, only the processing steps and / or device structures closely related to at least the scheme according to the present disclosure are shown in the drawings, and other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0053] The terms used in the present disclosure are explained below.

[0054] Deep Q-Network (DQN) is an algorithm that combines deep learning and reinforcement learning. It is mainly used to solve optimization problems in the decision-making process, especially in areas that require sequential decision-making. DQN uses deep neural networks to approximate the Q function. The Q function is a function that maps a state (or state-action pair) to an expected reward. Traditional Q learning algorithms use tables to store the Q value of each state (or state-action pair), but this method is no longer applicable when the state space is very large or continuous. DQN approximates this Q function through neural networks, so that it can handle more complex problems.

[0055] To improve the stability of training, DQN uses two neural networks: one is the online network for training, and the other is the target network for calculating the target Q value. The weights of the target network are updated regularly to reduce fluctuations in training.

[0056] State: In DQN, the process of interaction between the agent and the environment can be divided into a series of discrete time steps. At each time step, the agent observes the current state of the environment, which can be any information describing the environment.

[0057] Action: The agent must choose an action at each time step to affect the environment and obtain rewards. The action can be a finite set of choices, depending on the problem in the specific scenario.

[0058] Reward: At each time step, after the agent performs an action, the environment will give the agent a reward signal to indicate whether the action is good or bad. The reward can be positive (indicating good behavior) or negative (indicating bad behavior), or even zero.

[0059] The Q-value function represents the expected cumulative reward for taking a specific action in a given state. The Q-value is usually expressed as Q(s,a), where s represents the state and a represents the action.

[0060] Experience Replay is used to improve the learning process. Experience Replay is a method that stores the agent's experience in a replay buffer and randomly samples it for training. This helps reduce sample correlation and improves the stability of learning.

[0061] Figure 1 A schematic diagram showing a flow chart of a production line scheduling method according to some embodiments of the present disclosure is shown. Figure 1 As shown, the production line scheduling method of this embodiment includes the following steps.

[0062] In step 110, a production line production process model of the production line instance and a flexible job shop scheduling problem of the production line production process model are constructed.

[0063] There are different production line instances in different business production fields. For example, in the production fields of tobacco, food, electronic products, automobiles, pharmaceuticals, textiles, etc., there are corresponding tobacco production line instances (such as cigarette production line instances), food processing production line instances, electronic product manufacturing production line instances, automobile manufacturing production line instances, pharmaceutical production line instances, textile production line instances, etc., but not limited to the examples given.

[0064] According to the production characteristics of the production line instance, a corresponding production line production process model is formulated. Through the production process model of the production line, production information such as the workpiece set, machine set, and the processing procedures of each workpiece can be clarified.

[0065] Flexible Job Shop Scheduling Problem (FJSP) is a combinatorial optimization problem that can be applied to manufacturing and industrial production scheduling. FJSP aims to solve how to arrange the processing sequence and time of each job on the machine in a workshop with multiple jobs and multiple machines to optimize the job objectives, such as minimizing the completion time and maximizing the machine utilization rate.

[0066] In step 120, the flexible job shop scheduling problem is solved using a tabu search algorithm, and during the solving process, the parameters of the tabu search algorithm are corrected using reinforcement learning.

[0067] The parameters of the taboo search algorithm modified by reinforcement learning include, for example, the length of the taboo table and the number of candidate solutions.

[0068] The improved tabu search algorithm based on reinforcement learning allows tabu search parameters such as the length of the tabu table and the number of candidate solutions to be dynamically adjusted, allowing the algorithm to search the solution space more extensively within a limited time and effectively improve the quality of the solution. The improved tabu search algorithm has the advantages of fast calculation speed, strong global search capability, and adaptive adjustment of control parameters.

[0069] In step 130, a production line scheduling solution is determined based on the solution of the flexible job shop scheduling problem.

[0070] The production line scheduling plan is determined based on the processing steps of each workpiece on each machine indicated by the solution results.

[0071] The disclosed embodiments construct a production line production process model of a production line instance and its flexible job shop scheduling problem, and use a taboo search algorithm to solve the flexible job shop scheduling problem. During the solution process, reinforcement learning is used to correct the parameters of the taboo search algorithm, so that the algorithm can search the solution space more widely within a limited time, effectively improving the quality of the solution, thereby making the production line scheduling plan determined according to the solution result more reasonable, and the production line scheduling plan can be quickly and dynamically formulated or adjusted.

[0072] Considering that the production scheduling problem of tobacco packaging production line is similar to the FJSP problem, which has the NP-hard property, as well as the problems of inserting orders, overtime, and task cancellation that often occur on the production line, the actual production currently mainly adopts manual scheduling or some simple scheduling rules for scheduling. Among them, NP-hard problems refer to problems to which all NP problems can be reduced within the complexity of polynomial time, where "NP" stands for nondeterministic polynomial time, which means that although algorithms to solve these problems in polynomial time have not yet been found, if a solution is given, its correctness can be verified in polynomial time. Since the improved taboo search algorithm has the advantages of fast calculation speed, strong global search ability, and adaptive adjustment of control parameters, this embodiment takes minimizing the maximum completion time as the optimization goal, and proposes an improved taboo search algorithm. The improved taboo search algorithm realizes a job shop scheduling method that dynamically adjusts the length of the taboo table and the number of candidate solutions. This embodiment dynamically adjusts the parameters of each cycle of the taboo search algorithm, so that the algorithm searches the solution space more widely within a limited time, greatly improves the search efficiency, and effectively improves the quality of the solution.

[0073] Figure 2 A schematic diagram showing a flow chart of a production line scheduling method according to some other embodiments of the present disclosure. Figure 2 As shown, the production line scheduling method of this embodiment includes the following steps.

[0074] Step 210: Randomly generate multiple production line instances that meet the characteristics of the actual production line, and build production process models for each production line, abstract the actual production process into a FJSP problem that considers the switching time, and use the taboo search algorithm to solve the flexible job shop scheduling problem. During the solution process, use reinforcement learning to modify the parameters of the taboo search algorithm, see steps 220-2120 for details.

[0075] The production process model of the production line includes: n workpieces J = {J 1 ,J 2 ,…,J n}On m machines M={M 1 ,M 2 ,…,M m}Processing process, each workpiece J i Process several processes according to the set process route ij ,j∈{1,2,…,r i}, where r i Indicates workpiece J i The total number of processes. Each process O ij Can be processed on one or more machines (such as machine tools). ijRepresentative workpiece J i The set of machines that can process the jth process. i The preparation time for the jth process on machine k is p ijk . Workpiece J i The processing time of the jth process on machine k is t ijk . Workpiece J i Whether the jth process of ijk It is necessary to determine the appropriate workpiece processing sequence and which machine to use for each process of each workpiece. i Indicates workpiece J i The completion time of the last process.

[0076] The objective function of the flexible job shop scheduling problem is to minimize the maximum completion time of all workpieces. The constraints of the flexible job shop scheduling problem include: the completion time of each workpiece is less than or equal to the maximum completion time; each processing operation can only be processed by one machine in the set of available machines; the completion time of the process is not less than the sum of the start time and the processing time; and the end time of the previous process plus the preparation time of the next process on the same machine is not greater than the start time of the next process.

[0077] The objective function of the flexible job shop scheduling problem is expressed as:

[0078] f=minC max (1)

[0079] The constraints of the flexible job shop scheduling problem are expressed as follows:

[0080]

[0081] ∑ k∈M x ijk =1 (3)

[0082] C ijk ≥S ijk +t ijk × ijk (4)

[0083] S ijk ≥C ij-1k +p ijk (5)

[0084] Formula (1) represents the objective function of the problem, that is, to maximize the completion time C of all jobs. max Minimize, min represents the minimization function. Formula (2) represents the completion time C of each workpiece i All less than or equal to the maximum completion time C maxFormula (3) restricts each processing operation to be processed by only one machine in the available machine set. Formula (4) indicates that the completion time of the process is not less than the sum of the start time and the processing time. Formula (5) indicates that the end time of the previous process plus the preparation time of the next process on the same machine is not greater than the start time of the next process.

[0085] Wherein, solving the flexible job shop scheduling problem using the taboo search algorithm includes the following steps S1-S6.

[0086] S1: Randomly generate an initial solution and set the taboo table to empty;

[0087] S2: Determine whether the termination condition (or convergence condition) is met. If so, output the optimal solution and end the tabu search algorithm. If not, execute step S3.

[0088] S3: Generate several neighborhood solutions using the neighborhood function of the current solution, and determine several candidate solutions from them;

[0089] S4: Determine whether the candidate solution satisfies the contempt criterion. If so, replace the current solution with the best state that satisfies the contempt criterion as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution, and then go to step S6 to continue the iterative search process. If not, execute step S5;

[0090] S5: Determine the taboo status of each object corresponding to the candidate solution, select the best state of the non-taboo object in the candidate solution set as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution;

[0091] S6: Go to step S2 and continue the iterative search process.

[0092] The parameters of the taboo search algorithm modified by reinforcement learning include, for example, the length of the taboo table and the number of candidate solutions.

[0093] Step 220: Randomly initialize the online network (Q θ The parameters of the target network (i.e., θ) are initialized using the parameters of the online network. - ←θ), initialize the parameters of the taboo search algorithm {L list ,N c}, where L list Represents the length of the taboo table in the taboo search algorithm, N c Represents the number of candidate solutions generated each time in the taboo search algorithm. The initialization agent can select the action set A and define the state s under each taboo search sub-cycle i i .

[0094] The number of actions in the action set is determined jointly based on the number of intervals within the value range of the taboo table length of the taboo search algorithm and the number of intervals within the value range of the number of candidate solutions. Each action corresponds to an interval within the value range of the taboo table length of the taboo search algorithm and an interval within the value range of the number of candidate solutions.

[0095] The following example illustrates the action set. Taboo search algorithm parameters {L list ,N c} is initialized with 50 actions, which are merged into action set A. list , the commonly used value range is 5 to 55, and the range is divided into 5 intervals, for example. The corresponding action randomly selects a value in the interval as the length of the taboo table for this cycle. For example, if the action is a 1 When L list Randomly select between 5 and 15. This method is also applicable to N c , whose value is usually in the range of 10 to 510, which is divided into 10 intervals. The two parameters are combined to form 50 different actions corresponding to different interval selections. The agent determines the key parameter L of the current sub-cycle by selecting actions. list and N c .

[0096] The state s is represented by the evolution state of the ith sub-cycle and several previous sub-cycles using the taboo search algorithm. 5 , wherein the evolutionary state of each sub-cycle is characterized by the static feature vector and dynamic feature vector corresponding to the sub-cycle, the static feature vector is configured to include: the total number of machines, the total number of workpiece types, the total number of processes, the mean number of processes, the standard deviation of the number of processes, etc., and the dynamic feature vector is configured to include: the number of current taboo search sub-cycles, the optimal fitness value of the sub-cycle, the ratio of the fitness value to the initial value, the mean value of the optimal solution within the most recent preset number of times, the standard deviation, the ratio of the mean value of the optimal solution to the initial solution, etc.

[0097] For example, according to the taboo search algorithm and the scale and characteristics of the problem to be solved, 5 static features and 6 dynamic features are extracted to jointly form a state feature vector describing the current sub-cycle state. The static features include the total number of machines M, the total number of workpiece types R, the total number of processes N r , mean value of number of processes J ave , Standard deviation of number of processes J std , the corresponding static eigenvector can be expressed as: T t =[M,R,N r ,J ave ,J std ]; Dynamic features include the number of current taboo search sub-cycles t, the optimal fitness value B of this cycle 1 , the ratio of fitness value to initial value B2 , the average value of the last five optimal solutions is B 3 , standard deviation B 4 and the ratio of the 5th average to the initial solution B 5 , the corresponding dynamic feature vector can be expressed as: E t =[t,B 1 ,B 2 ,B 3 ,B 4 ,B 5 ]. Define the state feature vector of the algorithm in the tth generation as G t =[T t ,E t At each decision point t, according to the evolutionary state of the taboo search algorithm in the tth sub-cycle and the 5 sub-cycle states before this cycle [G t-4 ,G t-3 ,G t-2 ,G t-1 ,G t ] to adjust the algorithm parameters in the t+1th sub-cycle. Finally, the state of the decision point t is defined as s t =[G t-4 ,G t-3 ,G t-2 ,G t-1 ,G t ]. Similarly, the state s under the taboo search sub-cycle i i =[G i-4 ,G i-3 ,G i-2 ,G i-1 ,G i ].

[0098] Step 230: Randomly select a production line instance and obtain the initial state s 0 .

[0099] According to the above state s i The acquisition method is to obtain the initial state s when i = 0 0 .

[0100] Step 240: Select action a based on the initial state 0 , according to action a 0 Modify the parameters of the tabu search algorithm and run the tabu search algorithm.

[0101] When selecting an action, randomly select an action with the probability of the greedy rate, and select the action in the initial state s with the probability of (1-greedy rate) 0 The action that maximizes the expected Q value is taken, where the greedy rate gradually decreases with the increase of training times.

[0102] For example, the greedy method is used as the action selection strategy, which can be expressed as the following formula.

[0103]

[0104] Where r is a random number between 0 and 1, and ε represents the greedy rate, which decreases gradually with the increase of training times. n is the current iteration number. This action selection strategy compromises between "exploration" and "exploitation" based on a probability. Specifically, each time an action is selected: exploration is performed with a probability of ε, and an action is randomly selected with a uniform probability, represented as a(random); exploitation is performed with a probability of 1-ε, and an action is selected in state s t The action that maximizes the expected Q value is denoted as max a Q(s t ,a), when t=0, choose the initial state s 0 The action that maximizes the expected Q value is denoted as max a Q(s 0 ,a).

[0105] As mentioned above, according to the way the action set is generated, each action corresponds to an interval within the value range of the taboo table length of the taboo search algorithm and an interval within the value range of the number of candidate solutions. Therefore, when the action acts on the parameters of the taboo search algorithm, a value is randomly selected from a corresponding interval within the value range of the taboo table length as the taboo table length of this cycle. For example, if the action is a 1 When L list Randomly select between 5 and 15. This method is also applicable to N c , for example, the action is a 1 When N c Randomly select between 10 and 60.

[0106] Step 250: Determine the evolution state of the t-th sub-cycle of the tabu search algorithm to obtain the state s t ;

[0107] The state s is represented by the evolution state of the tth sub-cycle and the evolution states of several previous sub-cycles using the tabu search algorithm. t , where the evolution state of each sub-cycle is characterized by the static feature vector and dynamic feature vector corresponding to the sub-cycle. The static feature vector is configured to include: the total number of machines, the total number of workpiece types, the total number of processes, the mean number of processes, and the standard deviation of the number of processes. The dynamic feature vector is configured to include: the number of current taboo search sub-cycles, the optimal fitness value of the sub-cycle, the ratio of the fitness value to the initial value, the mean value of the optimal solution within the most recent preset number of times, the standard deviation, and the ratio of the mean value of the optimal solution to the initial solution. For details, see the above, for example, s t =[Gt-4 ,G t-3 ,G t-2 ,G t-1 ,G t ].

[0108] Step 260: Based on the state s t Select action a t , according to action a t Fix parameters of the tabu search algorithm.

[0109] When selecting an action, randomly select an action with the probability of the greedy rate, and select the state s with the probability of (1-greedy rate) t The action that maximizes the expected Q value is selected, where the greediness rate gradually decreases as the number of training times increases. For example, see the aforementioned greedy method as an action selection strategy.

[0110] As mentioned above, according to the way the action set is generated, each action corresponds to an interval within the value range of the taboo table length of the taboo search algorithm and an interval within the value range of the number of candidate solutions. Therefore, when the action acts on the parameters of the taboo search algorithm, a value is randomly selected from a corresponding interval within the value range of the taboo table length as the taboo table length of this cycle. For example, if the action is a 1 When L list Randomly select between 5 and 15. This method is also applicable to N c , for example, the action is a 1 When N c Randomly select between 10 and 60.

[0111] Step 270: Run a tabu search sub-cycle to determine the evolution state of the tabu search algorithm sub-cycle t+1 to obtain the state s t+1 .

[0112] Among them, the state s t+1 The method of obtaining the status s can refer to the above t , status i The method of obtaining is not described here.

[0113] Step 280: Calculate action a based on the formula t The reward value r at the current state t .

[0114] Action a t The reward value r at the current state tThe reward function is calculated by the optimal solution of a single cycle and the optimal solution of multiple training processes. The agent discovers which actions can generate greater reward values ​​in the current environment through the change of reward values ​​after multiple attempts. Obviously, this reward has positive and negative values. If the optimal value of the current generation t is less than the t-1 generation, the agent will experience a n If the optimal value of the current t-th generation is less than the optimal value of the previous cycle, the agent will also receive a positive reward value, otherwise it will receive a negative value. The reward value r is determined based on the difference between the minimum fitness value in the current training and the fitness value of the t-th sub-cycle in the current training. t .

[0115] The following is an example of a reward function after executing an action, Among them, minf(x l ) represents the minimum fitness value among the l training runs currently. Represents the fitness value of the t-th sub-cycle in the l-th training.

[0116] Step 290: Store experience(s t ,a t ,r t ,s t+1 ) to the experience replay memory D.

[0117] The experience replay memory is also called the replay buffer or experience pool. t ,a t ,r t ,s t+1 ) The four variables are stored in the experience replay memory D in the form of an array, and the experience records of different instances are also stored in the same experience replay memory D.

[0118] Step 2100: Randomly sample B experiences (s t′ ,a t′ ,r t′ ,s t′+1 ), calculate the target Q value, use the gradient descent method to update the parameters θ of the online network, and use the parameters of the online network to update the parameters θ of the target network every C time steps - .

[0119] If s t′+1 is the terminal state, and the reward value r t′ Determine the target Q value of the t′th sub-cycle (set to y t′ ). If s t′+1 Not the final state, based on the online network Qθ Determine in state s t′+1 The action that maximizes the expected Q value (set to a t′+1 , that is, argmax a′ Q θ (s t′+1 ,a′)), based on the target network Q θ- Determine in state s t ′+1 Next action a t′+1 The Q value of the t′+1th sub-cycle (i.e., Q θ- Ps t′+1 ,argmax a′ Q θ (s t′+1 ,a′)Q), according to the reward value r t′ and the Q value of the t′+1th sub-cycle under the discount factor γ, determine the target Q value of the t′th sub-cycle (set as y t′ ). For example, the target Q value of each sample is calculated using the following formula.

[0120]

[0121] The long short-term memory (LSTM) recurrent neural network is used to transform the state vector s of multiple generations t A vector of fixed length The LSTM network can add the latest 5 generations of state information G t to the cell state. Therefore, the final hidden state Captures all state s t The iteration state [G t-4 ,G t-3 ,G t-2 ,G t-1 ,G t ]. Then, based on the experience replay, the batch experience data is uniformly randomly sampled in the memory D, the loss function is calculated, and the gradient descent algorithm is used to update the online network (also called the policy network).

[0122] In some embodiments, the online network parameter θ is updated using a gradient descent method, which can be expressed as: θ←B -1 ∑(y t′ -Q θ (s t′ ,a t′ )) 2 , where B represents B experiences, y t′ represents the target Q value of the t′th sub-cycle, Q θ (s t′ ,a t′) indicates online network Q θ Determined in state s t′ Next action a t′ The Q value of (the Q value is used in reinforcement learning to measure the expected cumulative reward obtained by taking an action in a given state).

[0123] In some embodiments, the target network parameter θ is updated every C time steps. - , for example, by using the replication method, the online network parameter θ is assigned to the target network parameter θ - , for example, can be expressed as: θ - ←θ.

[0124] Step 2110: Repeat steps 250 to 2100 until the number of sub-cycles T of the tabu search algorithm is met.

[0125] Step 2120: Repeat steps 230 to 2110 until the number of training cycles L is met.

[0126] Step 2130: Determine the production line scheduling plan based on the solution of the flexible job shop scheduling problem.

[0127] The production line scheduling scheme is determined according to the processing steps of each workpiece on each machine indicated by the solution result. Specifically in this embodiment, for example, the production line scheduling scheme is determined according to the processing steps of each workpiece on each machine that minimize the maximum completion time of all workpieces.

[0128] The disclosed embodiments construct a production line production process model of a production line instance and its flexible job shop scheduling problem, and use a taboo search algorithm to solve the flexible job shop scheduling problem. During the solution process, reinforcement learning is used to correct the parameters of the taboo search algorithm, so that the algorithm can search the solution space more widely within a limited time, effectively improving the quality of the solution, thereby making the production line scheduling plan determined according to the solution result more reasonable, and the production line scheduling plan can be quickly and dynamically formulated or adjusted.

[0129] Figure 3 A schematic diagram showing a taboo search algorithm modified by reinforcement learning in some embodiments of the present disclosure is shown. Figure 3 As shown, the Tabu Search (TS) algorithm is based on the agent action a t Make parameter adjustments, and the agent performs reinforcement learning according to the state s U Select action a t , according to action a t Calculate the reward value. If it is a positive reward value (positive feedback), increase the selected action a t If the reward value is negative (negative feedback), the probability of selecting action a is reduced.t probability.

[0130] Figure 4 A schematic diagram of a production line scheduling device according to some embodiments of the present disclosure is shown. Figure 4 As shown, the production line scheduling device 400 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410 , and the processor 420 is configured to execute the production line scheduling method in various embodiments based on the instructions stored in the memory 410 .

[0131] The production line scheduling device 400 may further include an input / output interface 430 , a network interface 440 , a storage interface 450 , etc. These interfaces 430 , 440 , 450 , the memory 410 , and the processor 420 may be connected, for example, via a bus 460 .

[0132] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0133] The processor 420 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors and other discrete hardware components.

[0134] Among them, the input and output interface 430 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networked devices. The storage interface 450 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 460 can use any bus structure among a variety of bus structures. For example, the bus structure includes but is not limited to the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0135] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more (non-transient) computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that implements its solution mainly through a computer program.

[0136] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A production line scheduling method, comprising: Construct a production line production process model for a production line instance; Constructing a flexible job shop scheduling problem for a production line production process model; The tabu search algorithm is used to solve the flexible job shop scheduling problem. During the solution process, reinforcement learning is used to modify the parameters of the tabu search algorithm. According to the solution of the flexible job shop scheduling problem, the production line scheduling plan is determined.

2. The method according to claim 1, wherein: The production process model of the production line includes: n workpieces J = {J1, J2, ..., J n } On m machines M = {M1, M2, ..., M m }Processing process, each workpiece J i Process several processes according to the set process route ij , j∈{1, 2, ..., r i }, where r i Indicates workpiece J i The total number of processes.

3. The method according to claim 2, wherein: The objective function of the flexible job shop scheduling problem is to minimize the maximum completion time of all workpieces; The constraints of the flexible job shop scheduling problem include: the completion time of each workpiece is less than or equal to the maximum completion time; Each processing operation can only be processed by one machine from the set of available machines; the completion time of the operation is not less than the sum of the start time and the processing time; and the end time of the previous operation plus the preparation time of the next operation on the same machine is not greater than the start time of the next operation.

4. The method according to claim 1, wherein: Using the taboo search algorithm, the flexible job shop scheduling problem is solved including: Generate an initial solution randomly and set the taboo table to empty; Determine whether the termination condition is met. If so, output the optimal solution and end the taboo search algorithm. If not, use the neighborhood function of the current solution to generate several neighborhood solutions and determine several candidate solutions from them. Determine whether the candidate solution meets the contempt criterion. If so, replace the current solution with the best state that meets the contempt criterion as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution, and continue the iterative search process. If not, determine the taboo status of each object corresponding to the candidate solution, select the best state of the non-taboo object in the candidate solution set as the new current solution, and replace the taboo object that first enters the taboo table with the taboo object corresponding to the new current solution; Among them, the parameters of the taboo search algorithm modified by reinforcement learning include: the length of the taboo table and the number of candidate solutions.

5. The method according to any one of claims 1 to 4, wherein: The parameters of the tabu search algorithm modified using reinforcement learning include: Initialize the parameters of the online network, the parameters of the target network, the parameters of the tabu search algorithm, the set of actions that the agent can choose, and define the state of each tabu search sub-cycle; Select a production line instance and get the initial state s0; Select action a0 based on the initial state s0, modify the parameters of the tabu search algorithm according to action a0, and run the tabu search algorithm; Determine the evolution state of the tth sub-cycle of the tabu search algorithm to obtain the state s t ; Based on the status t Select the corresponding action a t , according to action a t Corrected parameters of the tabu search algorithm; Run a tabu search sub-cycle and determine the evolution state of the tabu search algorithm's t+1th sub-cycle to obtain the state s t+1 ; Calculate action a t The reward value r at the current state t ; Storage experience(s) t , a t , r t ,s t+1 ) to the experience playback memory D; Randomly sample B experiences (s) from the experience replay memory D t′ , a t′ , r t′ ,s t′+1 ), calculate the target Q value based on the target network, use the gradient descent method to update the parameters of the online network, and use the parameters of the online network to update the parameters of the target network every C time steps.

6. The method according to claim 5, wherein: The number of actions in the action set is determined jointly based on the number of intervals within the value range of the taboo table length of the taboo search algorithm and the number of intervals within the value range of the number of candidate solutions, and each action corresponds to an interval within the value range of the taboo table length of the taboo search algorithm and an interval within the value range of the number of candidate solutions.

7. The method according to claim 5, wherein: Determine the evolution state of the tth sub-cycle of the tabu search algorithm to obtain the state s t include: The state s is represented by the evolution state of the tth sub-cycle and the evolution states of several previous sub-cycles using the tabu search algorithm. t , wherein the evolutionary state of each sub-cycle is characterized by the static feature vector and dynamic feature vector corresponding to the sub-cycle, the static feature vector is configured to include: the total number of machines, the total number of workpiece types, the total number of processes, the mean number of processes, and the standard deviation of the number of processes, and the dynamic feature vector is configured to include: the number of current taboo search sub-cycles, the optimal fitness value of the sub-cycle, the ratio of the fitness value to the initial value, the mean value of the optimal solution within the most recent preset number of times, the standard deviation, and the ratio of the mean value of the optimal solution to the initial solution.

8. The method according to claim 5, wherein: Based on the status t Select the corresponding action a t include: When selecting an action, randomly select an action with the probability of the greedy rate, and select the state s with the probability of (1-greedy rate) t The action that maximizes the expected Q value is taken, where the greedy rate gradually decreases with the increase of training times.

9. The method according to claim 5, wherein: Calculate action a t The reward value r at the current state t include: Determine the reward value r based on the difference between the minimum fitness value in the current training and the fitness value of the t-th sub-cycle in the current training t .

10. The method according to claim 5, wherein: Calculating the target Q value includes: If s t′+1 is the final state, and the reward value r t′ Determine the target Q value for the t′th sub-cycle; If s t′+1 Not the final state, determined based on the online network in state s t′+1 Take the action a that maximizes the expected Q value t′+1 , based on the target network, determine the state s t′+1 Next action a t′+1 The Q value of the t′+1th sub-cycle is calculated based on the reward value r t′ The target Q value of the t′th sub-cycle is determined by summing up the Q value of the t′+1th sub-cycle under the discount factor.

11. The method according to claim 3, wherein: Determine the production line scheduling plan including: The production line scheduling plan is determined based on the processing procedures of each workpiece on each machine so as to minimize the maximum completion time of all workpieces.

12. The method according to any one of claims 1 to 4, wherein: The production line examples include tobacco production line examples.

13. A production line scheduling device, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the production line scheduling method according to any one of claims 1 to 12 based on instructions stored in the memory.

14. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the production line scheduling method according to any one of claims 1 to 12.

15. A computer program product, comprising computer instructions, which, when executed by a processor, implement the production line scheduling method according to any one of claims 1 to 12.

Images