Hybrid flow shop scheduling problem optimization method for small-batch ordered product production

Abstract

The invention relates to a hybrid flow shop scheduling problem optimization method for small-batch customized product production, which comprises the following steps of: firstly, obtaining processing data such as processing machines and the like available for workpieces and a processing sequence according to initial data, then defining constraint conditions in a production process, and assigning an optimization target according to key factors in the production process; and according to a specified optimization target, optimization is carried out through a multi-agent reinforcement learning method, so that an optimal scheduling scheme is obtained. According to the method, the problem of hybrid flow shop scheduling of small-batch customized product production can be efficiently solved, an optimized scheduling scheme is obtained, and a method is provided for improving the production benefits of enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of scheduling in a mixed flow shop environment for the production of small batches of customized products, and in particular to a method for optimizing scheduling problems in a mixed flow shop environment for the production of small batches of customized products. Background Art

[0002] The production scheduling problem of a small batch custom product workshop is essentially a type of hybrid flow shop scheduling problem (Hybrid Flow-Shop Scheduling Problem, HFSP), which is a classic NP-hard problem. In a general hybrid flow shop, each workpiece has the same process, and each process has a drop-drying station (at least one process has multiple machines). A certain process of the workpiece can be processed by any machine corresponding to the process. In a hybrid flow shop of small batch custom products, although the products customized by different customers have the same process, these products may have large differences in appearance size and specifications. There must be multiple processing machines for one process. Therefore, some products can only choose some machines of the process in a certain process, and there must be different sets of processing machines for the same process of two workpieces. Since the same process of the same workpiece is processed on different machines, the processing time and energy consumed may be different, so a suitable scheduling plan is needed to make the production process meet the needs of maximizing enterprise benefits. Solving the above hybrid flow shop scheduling problem for the production of small batch custom products is to obtain a relatively optimal scheduling plan. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method for optimizing the scheduling problem of a mixed flow shop for the production of small batch customized products, so that the production process can meet the needs of maximizing the benefits of the enterprise.

[0004] To achieve the above object, the present invention adopts the following technical solution: a method for optimizing the scheduling problem of a mixed flow shop for the production of small batches of customized products, comprising the following steps:

[0005] Step 1: Obtain processing data for the mixed flow shop scheduling problem of small batch customized products;

[0006] Step 2: Define the constraints of the mixed flow shop scheduling problem for small batch customized products;

[0007] Step 3: Specify optimization goals based on key factors of the production process;

[0008] Step 4: Setting the elements of the multi-agent confrontation reinforcement learning Q-Learning algorithm and expressing the scheduling scheme;

[0009] Step 5: Implementation of the optimization process of the Q-Learning algorithm for multi-agent confrontation reinforcement learning;

[0010] Step 6: Get the best scheduling solution.

[0011] In a preferred embodiment, the specific implementation process of step 1 is as follows:

[0012] Assume that there are n different workpieces to be processed, each workpiece has l processes, each process has one or more processing machines, and there must be a process with multiple processing machines; the sets of processing machines available for the same process of different workpieces may be different, and there must be different sets of processing machines for the same process of two workpieces; the processing time of the same process of a workpiece on different machines is generally different; it is necessary to arrange the processing machines of these n workpieces and the processing order of the workpieces on the machines; according to the above description, the following processing data is obtained:

[0013] 1) Workpiece set P = {p1, p2, ..., p i , ..., p n}, where p i represents the i-th workpiece;

[0014] 2) Process sequence set O = {o1, o2, ..., o i , ..., o n},o i ={o i1 , o i2 , ..., o ij , ..., o il}, where o i Represents workpiece p i The process sequence, o ij Represents workpiece p i The jth process of

[0015] 3) Machine set M = {M1, M2, ..., M j , ..., M l}, where M j represents the set of machines for the jth process; M j ={m j1 , m j2 , ..., m jk′ , ..., m jm′}, where m jk′ represents the k′th machine in the jth process, m′ is the total number of machines in the jth process, and there must be a process with a total number of machines greater than 1; sort and number all the machines in the jth process into a set m = {m1, m2, ..., m k , ..., m m},mk represents the kth machine among all machines, and m is the total number of machines;

[0016] 4) Workpiece p i A collection of processing machines available for all processes OM i ={om i1 ,om i2 ,...,om ij ,...,om il}, where om ij Represents workpiece p i The jth process o ij A collection of processing machines available; That is, process o ij The available processing machines are the machine set M of the jth process j One or more of them; there is a workpiece p i and artifact p i′ , i≠i′,om ij ≠om i′j , that is, there must be two workpieces with different sets of available processing machines for the same process; ij ={om ij1 ,om ij2 ,...,om ijk″ ,...,om ijm″},om ijk″ Represents workpiece p i The k″th available processing machine for the jth process, m″ is the workpiece p i The total number of available machines for the j-th process;

[0017] 5) The processing time set T corresponds one-to-one to the elements of the set of optional processing machines OM for the workpiece, t ijk″ ∈T represents the workpiece p i The processing time of the k″th processing machine is selected for the jth process;

[0018] 6) Machine load power PL = {pl1, pl2, ..., pl k , ..., pl m}, where pl k represents the load energy consumption of the kth machine;

[0019] 7) Machine load power PUL = {pul1, pul2, ..., pul k , ..., pul m}, where pul k represents the no-load energy consumption of the kth machine.

[0020] In a preferred embodiment, the specific implementation process of step 4 is as follows:

[0021] The multi-agent confrontation reinforcement learning Q-Learning algorithm is used to generate a better scheduling solution; there are N, N>1, a set of competing agents C = {X1, X2, ..., X I , …, X N}, I∈{1, 2, …, N}, each agent can continuously improve the selection of processing sequences by executing several steps of actions, and finally arrange the processing order of all workpieces to obtain a complete scheduling plan; in the agent training process, each execution of an action is regarded as a step, and each time a complete scheduling plan is obtained, a round of actions is completed. After completing a round of actions, a reward is obtained, which is used to calculate the reward for each step in this round of training and update the Q table.

[0022] In a preferred embodiment, the elements of the agent in the reinforcement learning Q-Learning algorithm are as follows: the agent is a virtual entity that selects the next process according to the current process, and the environment is all elements that can interact with the agent, including the processing sequence OP of all agents and other agents; wherein the processing sequence OP of the agent = {OP1, OP2, ..., OP I , …, OP N}, OP I is the current processing sequence of the Ith agent, OP I It includes two parts: the process sequence of the workpiece and the machine sequence. It records the processes of the currently scheduled workpieces and the corresponding processing machines. Each intelligent action will change the current processing sequence. When the processes of all workpieces are arranged, it is considered to have obtained a complete scheduling plan.

[0023] In a preferred embodiment, the state, action and reward of the agent are specifically as follows:

[0024] 1) Status: Current processing status op ijk″ , op ijk″ Represents the current sequence OP I The latest process is workpiece p i For the jth process, the k″th available processing machine is selected, that is, the state includes the information of the currently processed workpiece, process and processing machine;

[0025] 2) Action: Select from the currently available processing steps The processing steps that the agent can choose are affected by the current processing sequence OP in the environment. I The impact of

[0026] 3) Rewards: All agents receive a complete scheduling plan. That is, after completing a round of actions, the ranking and whether the record is broken are calculated according to the scheduling plan to obtain scores, which are evenly distributed to each step in this round of training as a reward for each action executed.

[0027] In a preferred embodiment, the specific implementation process of step 5 is as follows:

[0028] The optimization process of the Q-Leaming algorithm for multi-agent confrontation reinforcement learning is as follows:

[0029] Step 51: Initialize the Q tables of N agents to 0. Each agent’s Q table is independent. Initialize the ranking reward table R′={r′1, r′2, …, r′ I ,…,r′ N}, in the ranking reward, there is at least one positive number and one negative number. The higher the ranking, the higher the reward. A scheduling scheme is randomly generated to calculate the optimization objective function value as the initial record value of all agents. The record breakthrough reward R″={0, r″} is initialized, and the solution set is initialized to an empty set;

[0030] Step 52: Initialize the states of all agents in turn, where the state of the th agent is initialized to the first process op of random workpiece a a1b , a∈{1,2,…,n}, b∈{1,2,…,m″}, b is the serial number of the available processing machine, and the initial process of different agents can be different; initialize the current processing sequence OP of the corresponding agent I =[a, b], used to record the current processing sequence;

[0031] Step 53: All agents follow the current state op in turn ijk″ , and the ∈ greedy strategy executes the action, that is, randomly select one from the processable processes with probability ∈, and select the process with the largest Q value among the processable processes according to the Q table with probability 1-∈. When there are multiple actions with equal Q values, randomly select one from these actions;

[0032] Step 54: The agent selects a new process After that, the current state becomes At the same time, the current processing sequence is updated, and the workpiece number and the selected available processing machine number are added to the end of the corresponding position of the processing sequence of the agent; all agents and corresponding sequences are updated in sequence, and then it is determined whether all agents have obtained a complete scheduling plan. If a complete scheduling plan is obtained, go to step 55, otherwise go back to step 53;

[0033] Step 55: Calculate the optimization objective function values ​​corresponding to the scheduling schemes obtained by all agents, sort them from best to worst, and obtain the scores in the ranking reward table R′ in order according to the ranking, as reward 1; determine whether the optimization objective function value of the current scheme is better than the record value, if it is better than the record value, obtain the record breakthrough reward R″ as r″, and use it as reward 2, and update the current record value to the latest record value; otherwise, the record breakthrough reward R″ is 0, and used as reward 2; each agent independently calculates the record value and the record breakthrough reward without interfering with each other; add the two parts of the reward obtained by the agent in turn as the total reward for a round of actions; in addition, in the comparison of the above agents, for single-objective problems, the target value is directly compared, and if it is a multi-objective problem, the dominance relationship is compared;

[0034] Step 56: Update the Q table and divide the total reward of each agent by the number of actions performed in one round of training to get the reward R for each step t , and update the Q table according to the following formula;

[0035] Q t+1 (s t , a t )=Q t (s t , a t )+α[R t +γmax Q t (s t+1 , a t+1 )-Q t (s t , a t )]

[0036] Step 57: Add the scheduling solution obtained by the agent currently ranked first to the solution set; update the solution set, retain the solution with the highest objective value for the single-objective problem, and retain all non-dominated solutions for the multi-objective problem;

[0037] Step 58: Determine whether the iteration termination condition, that is, the maximum number of training rounds, is reached. If the termination condition is not reached, jump to step 52; if the termination condition is reached, output the current solution set.

[0038] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a method for optimizing the scheduling problem of a small-batch customized product assembly line workshop, which can quickly and effectively obtain a better scheduling solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0043] The present invention provides an optimization method for the hybrid flow shop scheduling problem of small batch customized product production, referring to Figure 1 , including the following steps:

[0044] Step 1: Obtain processing data for the mixed flow shop scheduling problem of small batch customized products;

[0045] Step 2: Define the constraints of the mixed flow shop scheduling problem for small batch customized products;

[0046] Step 3: Specify optimization goals based on key factors of the production process;

[0047] Step 4: Setting the elements of the multi-agent confrontation reinforcement learning Q-Learning algorithm and expressing the scheduling scheme;

[0048] Step 5: Implementation of the optimization process of the Q-Learning algorithm for multi-agent confrontation reinforcement learning;

[0049] Step 6: Get the best scheduling solution.

[0050] The following is a specific implementation process of the present invention. The method of the present invention includes six steps:

[0051] 1. Obtain processing data for the mixed flow shop scheduling problem of small batch customized product production;

[0052] Assume that there are n different workpieces to be processed, each workpiece has l processes, each process has one or more processing machines, and there must be a process with multiple processing machines. The sets of processing machines available for the same process of different workpieces may be different, and there must be different sets of processing machines for the same process of two workpieces. The processing time of the same process of a workpiece on different machines is generally different. It is necessary to arrange the processing machines for these n workpieces and the processing order of the workpieces on the machines. Based on the above description, the following processing data can be obtained.

[0053] 1) Workpiece set P = {p1, p2, ..., p i , ..., p n}, where p i represents the i-th workpiece.

[0054] 2) Process sequence set O = {o1, o2, ..., o i , ..., o n},o i ={o i1 , o i2 , ..., o ij , ..., o il}, where o i Represents workpiece p i The process sequence, o ij Represents workpiece p i The j-th process.

[0055] 3) Machine set M = {M1, M2, ..., M j , ..., M l}, where M j represents the set of machines for the jth process; M j ={m j1 , m j2 , ..., m jk′ , ..., m jm′}, where m jk′ represents the k′th machine in the jth process, m′ is the total number of machines in the jth process, and there must be a process with a total number of machines greater than 1. Sort and number all the machines in the jth process into a set m = {m1, m2, ..., m k , ..., m m},m k represents the kth machine among all machines, and m is the total number of machines.

[0056] 4) Workpiece p i A collection of processing machines available for all processes OM i ={om i1 ,om i2 ,...,om ij ,...,om il}, where om ij Represents workpiece p i The jth process o ij A collection of processing machines available; That is, process o ij The available processing machines are the machine set M of the jth process j One or more of them; there is a workpiece p i and artifact pi′ , i≠i′,om ij ≠om i′j , that is, there must be two workpieces with different sets of available processing machines for the same process; ij ={om ij1 ,om ij2 ,...,om ijk″ ,...,om ijm″},om ijk″ Represents workpiece p i The k″th available processing machine for the jth process, m″ is the workpiece p i The total number of available machines for the j-th process.

[0057] 5) The processing time set T corresponds one-to-one to the elements of the set of optional processing machines OM for the workpiece, t ijk″ ∈T represents the workpiece p i The j-th process uses the processing time of the k″th processing machine.

[0058] 6) Machine load power PL = {pl1, pl2, ..., pl k , ..., pl m}, where pl k Represents the load energy consumption of the kth machine.

[0059] 7) Machine load power PUL = {pul1, pul2, ..., pul k , ..., pul m}, where pul k represents the no-load energy consumption of the kth machine.

[0060] 2. Define processing constraints

[0061] In the workshop, the processing of workpieces is subject to various constraints. It is necessary to define the constraints in the processing process of the mixed flow shop scheduling problem of small batch customized product production.

[0062] 1) Each operation of the workpiece is processed only once and can only be processed on one available machine.

[0063]

[0064] Among them, p ijk″ Represents workpiece p i Whether the j-th process is processed by the k″th machine.

[0065]

[0066] 2) The first process of the workpiece is allowed to be processed at time 0.

[0067] JS ij ≥0, i∈{1, 2,…,n}, j=1 (3)

[0069] Among them, JS ij Represents workpiece p i The start time of the j-th processing step.

[0070] 3) The workpiece can only start the next process after the previous process is completed.

[0071] JS ij ≥ JE i(j-1) ,i∈{1,2,…,n}, j∈{2,3,…,l} (4)

[0073] Among them, JE i(j-1) Represents workpiece p i The completion time of the previous operation.

[0074] 4) The machine can only start processing the next workpiece after it has finished processing a certain workpiece.

[0075] Ms ijk ≥ME i′j′k ,i∈{1,2,…,n},j∈{2,3,…,l},k∈{1,2,…,m} (5)

[0077] Among them, MS ijk Represents workpiece p i The jth process is on machine m k The start time of the upper processing, ME i′j′k Represents machine m k The completion time of the last workpiece.

[0078] 5) The processing time of any process of all workpieces is greater than 0.

[0079] t ijk″ ≥0,i∈{1,2,…,n},j∈{2,3,…,l},k∈{1,2,…,m} (6)

[0081] Among them, t ijk″ Represents workpiece p i The processing time of the j-th process on the k″th available processing machine.

[0082] 3. Set optimization goals

[0083] Common optimization objectives in hybrid flow shop scheduling problems include completion time, energy consumption, processing cost, processing quality, delivery delay time, and machine load balance. The following lists the calculation methods for minimizing the maximum completion time, minimizing processing energy consumption, and minimizing machine load imbalance.

[0084] 1) Minimize the maximum completion time.

[0085] C max =min(max{C i |i=1,2,...,n}) (7)

[0087] Among them, C i For workpiece p i The completion time of the last process.

[0088] 2) Minimize the total energy consumption of the machine

[0089]

[0090] Among them, p ijk Represents workpiece p i Is the jth process on machine m k On processing,

[0091]

[0092] 3) Minimize machine load imbalance

[0093]

[0094] Among them, CM jk represents the load time of the k″th machine in the jth process, CM j(mean) Denotes the machine set M of the j-th process j The average load time of the machines in the

[0095] 4. Reinforcement Learning Elements Setting

[0096] This design uses the multi-agent confrontation reinforcement learning Q-Learning algorithm to generate a better scheduling solution. There are N (N>1) agents in a competitive relationship C = {X1, X2, ..., X I , …, X N}, I∈{1, 2, …, N}, each agent can continuously improve the selection of processing sequence by executing several steps of actions, and finally arrange the processing sequence of all workpieces to obtain a complete scheduling plan. During the agent training process, each execution of an action is regarded as a step, and each time a complete scheduling plan is obtained, a round of actions is completed. After completing a round of actions, a reward is obtained, which is used to calculate the reward for each step in this round of training and update the Q table.

[0097] The elements of the agent in the reinforcement learning Q-Learning algorithm are as follows: In this design, the agent is a virtual entity that selects the next process based on the current process, and the environment is all the elements that can interact with the agent, including the processing sequence OP of all agents, other agents, and other factors. Among them, the agent's processing sequence OP = {OP1, OP2, ..., OP I , …, OP N}, OP I is the current processing sequence of the th agent, OP I It is divided into two parts: the process sequence of the workpiece and the machine sequence. It records the process sequence of the currently scheduled workpiece and the corresponding processing machine. Each action of the intelligent agent will change the current processing sequence. When the processes of all workpieces are scheduled, it is considered to have obtained a complete scheduling plan. The following introduces the elements such as the state, action and reward of the intelligent agent.

[0098] 1) Status: Current processing status op ijk″ , op ijk″ Represents the current sequence OP I The latest process is workpiece p i For the j-th process, the k″th available processing machine is selected, that is, the state includes the information of the currently processed workpiece, process and processing machine.

[0099] 2) Action: Select from the currently available processing steps The processing steps that the agent can choose are affected by the current processing sequence OP in the environment. I impact.

[0100] 3) Rewards: All agents receive a complete scheduling plan. That is, after completing a round of actions, the ranking and whether the record is broken are calculated according to the scheduling plan to obtain scores, which are evenly distributed to each step in this round of training as a reward for each action executed.

[0101] For example: In a scheduling problem of size n*r=3*2, the current sequence of the agent is OP I =[21132,21221]. The first number "2" in the first layer represents the first process of workpiece 2. 21 , the first number "2" corresponding to the second layer represents om212 , that is, the process selects the second available processing machine. The fifth number "2" on the first layer represents the second process of workpiece 2. 22 , corresponding to the fifth number "1" on the second layer, represents om 221 , that is, the process selects the first available processing machine. At this time, the corresponding current processing process (state) is represented by op 221 The available actions in the current state are That is, you can select the second process of workpiece 3 and select an available processing machine for this process. 32 c, after selection, the sequence is updated to OP I =[211323, 21221c], at this time all the workpieces have been processed, a complete scheduling plan is obtained, the agent completes a round of actions, and when all agents complete a round of actions, the Q table is updated and a round of training is completed.

[0102] 5. Algorithm optimization iteration process

[0103] The optimization process of the multi-agent confrontation reinforcement learning Q-Learning algorithm is as follows:

[0104] Step 1: Initialize the Q tables of N agents to 0. Each agent’s Q table is independent. Initialize the ranking reward table R′={r′1, r′2, …, r′ I ,…,r′ N}, in the ranking reward, there is at least one positive number and one negative number. The higher the ranking, the higher the reward. A scheduling plan is randomly generated to calculate the optimization objective function value as the initial record value of all agents, and the record breakthrough reward R″={0, r″} is initialized. The solution set is initialized to an empty set.

[0105] Step 2: Initialize the states of all agents in turn, where the state of the th agent is initialized to the first process op of random workpiece a a1b , a∈{1,2,…,n}, b∈{1,2,…,m″}, b is the serial number of the available processing machine, and the initial process of different agents can be different. Initialize the current processing sequence OP of the corresponding agent I =[a, b], used to record the current processing sequence.

[0106] Step 3: All agents take turns according to the current state op ijk″ , and the ∈ greedy strategy executes the action, that is, randomly select one from the processable processes with probability ∈, and select the process with the largest Q value among the processable processes according to the Q table with probability 1-∈. When there are multiple actions with equal Q values, randomly select one from these actions.

[0107] Step 4: The agent selects a new process After that, the current state becomes At the same time, the current processing sequence is updated, and the workpiece number and the selected available processing machine number are added to the end of the corresponding position of the processing sequence of the agent in the manner of the example given in claim 3. All agents and corresponding sequences are updated in sequence, and then it is determined whether all agents have obtained a complete scheduling solution. If a complete scheduling solution is obtained, go to step 5, otherwise go back to step 3.

[0108] Step 5: Calculate the optimization objective function values ​​corresponding to the scheduling schemes obtained by all agents, sort them from best to worst, and obtain the scores in the ranking reward table R′ in order of ranking as reward 1. Determine whether the optimization objective function value of the current scheme is better than the record value. If it is better than the record value, obtain the record breakthrough reward R″ as r″ as reward 2, and update the current record value to the latest record value; otherwise, the record breakthrough reward R″ is 0, and it is used as reward 2. Each agent independently calculates the record value and the record breakthrough reward without interfering with each other. The two parts of the reward obtained by the agent are added in turn as the total reward for a round of actions. In addition, in the comparison of the above agents, for single-objective problems, the target value is directly compared. If it is a multi-objective problem, the dominance relationship is compared.

[0109] Step 6: Update the Q table and divide the total reward of each agent by the number of actions performed in one round of training to get the reward R for each step t , and update the Q table according to the following formula.

[0110] Q t+1 (s t , a t )=Q t (s t , a t )+α[R t +γmax Q t (s t+1 , a t+1 )-Q t (s t , a t )]

[0111] Step 7: Add the scheduling solution obtained by the current top-ranked agent to the solution set. Update the solution set, retain the solution with the highest objective value for single-objective problems, and retain all non-dominated solutions for multi-objective problems.

[0112] Step 8: Determine whether the iteration termination condition, that is, the maximum number of training rounds, is reached. If the termination condition is not reached, jump to step 2; if the termination condition is reached, output the current solution set.

[0113] 6. Optimal scheduling scheme and fitness value

[0114] 1) Single-target problem

[0115] In a single-objective problem, the quality of the solution is consistent with the target value of the optimization goal. The individual with the best target value represents the best scheduling solution.

[0116] 2) Multi-objective problem

[0117] In the solution set obtained for the multi-objective problem, the quality of the solution is calculated based on the dominance relationship and the crowding distance. The higher the dominance level of the solution, the better the solution; when the dominance levels are the same, the solution with a larger crowding distance is better.

[0118] The optimal scheduling scheme is determined by assigning different weights to different optimization objectives as needed. The H target values ​​of all individuals in the population are finally calculated, and the weighted target value f is obtained according to formula (12) t , f t The best individual represents the best scheduling solution.

[0119]

[0120] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A hybrid flow shop scheduling optimization method for small batch custom product production, characterized in that: The steps include: Step 1: Obtain processing data for the mixed flow shop scheduling problem of small batch customized products; Step 2: Define the constraints of the mixed flow shop scheduling problem for small batch customized products; Step 3: Specify optimization goals based on key factors of the production process; Step 4: Setting the elements of the multi-agent confrontation reinforcement learning Q-Learning algorithm and expressing the scheduling scheme; Step 5: Implementation of the optimization process of the Q-Learning algorithm for multi-agent confrontation reinforcement learning; Step 6: Get the best scheduling solution.

2. The method for optimizing the scheduling problem of a hybrid flow shop for the production of small batches of customized products according to claim 1 is characterized in that: The specific implementation process of step 1 is as follows: Assume that there are n different workpieces to be processed, each workpiece has l processes, each process has one or more processing machines, and there must be a process with multiple processing machines; the sets of processing machines available for the same process of different workpieces may be different, and there must be different sets of processing machines for the same process of two workpieces; the processing time of the same process of a workpiece on different machines is generally different; it is necessary to arrange the processing machines of these n workpieces and the processing order of the workpieces on the machines; according to the above description, the following processing data is obtained: 1) Workpiece set P = {p1, p2, ..., p i , ..., p n }, where p i represents the i-th workpiece; 2) Process sequence set O = {o1, o2, ..., o i , ..., o n },o i ={o i1 , o i2 , ..., o ij , ..., o il }, where o i Represents workpiece p i The process sequence, o ij Represents workpiece p i The jth process of 3) Machine set M = {M1, M2, ..., M j , ..., M l }, where M j represents the set of machines for the jth process; M j ={m j1 , m j2 , ..., m jk′ , ..., m jm′ }, where m jk′ represents the k′th machine in the jth process, m′ is the total number of machines in the jth process, and there must be a process with a total number of machines greater than 1; sort and number all the machines in the jth process into a set m = {m1, m2, ..., m k , ..., m m },m k represents the kth machine among all machines, and m is the total number of machines; 4) Workpiece p i A collection of processing machines available for all processes OM i ={om i1 ,om i2 ,...,om ij ,...,om il }, where om ij Represents workpiece p i The jth process o ij A collection of processing machines available; That is, process o ij The available processing machines are the machine set M of the jth process j One or more of them; there is a workpiece p i and artifact p i′ , i≠i′,om ij ≠om i′j , that is, there must be two workpieces with different sets of available processing machines for the same process; ij ={om ij1 ,om ij2 ,...,om ijk″ ,...,om ijm″ },om ijk″ Represents workpiece p i The k″th available processing machine for the jth process, m″ is the workpiece p i The total number of available machines for the j-th process; 5) The processing time set T corresponds one-to-one to the elements of the set of optional processing machines OM for the workpiece, t ijk″ ∈T represents the workpiece p i The processing time of the k″th processing machine is selected for the jth process; 6) Machine load power PL = {pl1, pl2, ..., pl k , ..., pl m }, where pl k represents the load energy consumption of the kth machine; 7) Machine load power PUL = {pul1, pul2, ..., pul k , ..., pul m }, where pul k represents the no-load energy consumption of the kth machine.

3. The method for optimizing the scheduling problem of a hybrid flow shop for the production of small batches of customized products according to claim 1 is characterized in that: The specific implementation process of step 4 is as follows: The multi-agent confrontation reinforcement learning Q-Learning algorithm is used to generate a better scheduling solution; there are N, N>1, a set of competing agents C = {X1, X2, ..., X I , …, X N }, I∈{1, 2, …, N}, each agent can continuously improve the selection of processing sequences by executing several steps of actions, and finally arrange the processing order of all workpieces to obtain a complete scheduling plan; in the agent training process, each execution of an action is regarded as a step, and each time a complete scheduling plan is obtained, a round of actions is completed. After completing a round of actions, a reward is obtained, which is used to calculate the reward for each step in this round of training and update the Q table.

4. The method for optimizing the scheduling problem of a hybrid flow shop for the production of small batches of customized products according to claim 3 is characterized in that: The elements of the agent in the reinforcement learning Q-Learning algorithm are as follows: the agent is a virtual entity that selects the next process based on the current process, and the environment is all the elements that can interact with the agent, including the processing sequence OP of all agents and the rest of the agents; where the processing sequence OP of the agent = {OP1, OP2, ..., OP I , …, OP N }, OP I is the current processing sequence of the th agent, OP I It includes two parts: the process sequence of the workpiece and the machine sequence. It records the processes of the currently scheduled workpieces and the corresponding processing machines. Each intelligent action will change the current processing sequence. When the processes of all workpieces are arranged, it is considered to have obtained a complete scheduling plan.

5. The method for optimizing the scheduling problem of a hybrid flow shop for the production of small batches of customized products according to claim 4 is characterized in that: The state, action and reward of the agent are as follows: 1) Status: Current processing status op ijk″ , op ijk″ Represents the current sequence OP I The latest process is workpiece p i For the jth process, the k″th available processing machine is selected, that is, the state includes the information of the currently processed workpiece, process and processing machine; 2) Action: Select from the currently available processing steps The processing steps that the agent can choose are affected by the current processing sequence OP in the environment. I The impact of 3) Rewards: All agents receive a complete scheduling plan. That is, after completing a round of actions, the ranking and whether the record is broken are calculated according to the scheduling plan to obtain scores, which are evenly distributed to each step in this round of training as a reward for each action executed.

6. The method for optimizing the scheduling problem of a hybrid flow shop for the production of small batches of customized products according to claim 1 is characterized in that: The specific implementation process of step 5 is as follows: The optimization process of the multi-agent confrontation reinforcement learning Q-Learning algorithm is as follows: Step 51: Initialize the Q tables of N agents to 0. Each agent’s Q table is independent. Initialize the ranking reward table R′={r′1, r′2, …, r′ I ,…,r′ N }, in the ranking reward, there is at least one positive number and one negative number. The higher the ranking, the higher the reward. A scheduling scheme is randomly generated to calculate the optimization objective function value as the initial record value of all agents. The record breakthrough reward R″={0, r″} is initialized, and the solution set is initialized to an empty set; Step 52: Initialize the states of all agents in turn, where the state of the th agent is initialized to the first process op of random workpiece a a1b , a∈{1,2,…,n}, b∈{1,2,…,m″}, b is the serial number of the available processing machine, and the initial process of different agents can be different; initialize the current processing sequence OP of the corresponding agent I =[a, b], used to record the current processing sequence; Step 53: All agents follow the current state op in turn ijk″ , and the ∈ greedy strategy executes the action, that is, randomly select one from the processable processes with probability ∈, and select the process with the largest Q value among the processable processes according to the Q table with probability 1-∈. When there are multiple actions with equal Q values, randomly select one from these actions; Step 54: The agent selects a new process After that, the current state becomes At the same time, the current processing sequence is updated, and the workpiece number and the selected available processing machine number are added to the end of the corresponding position of the processing sequence of the agent; all agents and corresponding sequences are updated in sequence, and then it is determined whether all agents have obtained a complete scheduling plan. If a complete scheduling plan is obtained, go to step 55, otherwise go back to step 53; Step 55: Calculate the optimization objective function values ​​corresponding to the scheduling schemes obtained by all agents, sort them from best to worst, and obtain the scores in the ranking reward table R′ in order according to the ranking, as reward 1; determine whether the optimization objective function value of the current scheme is better than the record value, if it is better than the record value, obtain the record breakthrough reward R″ as r″, and use it as reward 2, and update the current record value to the latest record value; otherwise, the record breakthrough reward R″ is 0, and used as reward 2; each agent independently calculates the record value and the record breakthrough reward without interfering with each other; add the two parts of the reward obtained by the agent in turn as the total reward for a round of actions; in addition, in the comparison of the above agents, for single-objective problems, the target value is directly compared, and if it is a multi-objective problem, the dominance relationship is compared; Step 56: Update the Q table and divide the total reward of each agent by the number of actions performed in one round of training to get the reward R for each step t , and update the Q table according to the following formula; Q t+1 (s t ,a t )=Q t (s t ,a t )+α[R t +γmax Q t (s t+1 ,a t+1 )-Q t (s t ,a t )] Step 57: Add the scheduling solution obtained by the agent currently ranked first to the solution set; update the solution set, retain the solution with the highest objective value for the single-objective problem, and retain all non-dominated solutions for the multi-objective problem; Step 58: Determine whether the iteration termination condition, i.e., the maximum number of training rounds, is reached. If the termination condition is not reached, jump to step 52; If the termination condition is reached, output the current solution set.

Images