Method for Generating Workshop Tool Scheduling Rules Based on Q-Learning

Abstract

The present invention discloses a Q-Learning-based method for generating shop tool scheduling rules, belonging to the field of data processing technology. The method comprises: constructing a scheduling model for minimizing the weighted sum of the maximum completion time and the number of tool changes; obtaining the priority of each machine according to a machine priority heuristic rule; assigning tools to each machine according to a heuristic rule for tool numbering of the initial tool assignment and a heuristic rule for tool numbering of the tool change; outputting a tool assignment combination for each machine's processing task set; establishing state characteristics; dividing the state characteristics into multiple state spaces; generating a Q-Learning model through the state space, and outputting a shop tool scheduling solution through the Q-Learning model. The Q-Learning-based method for generating shop tool scheduling rules solves the problem that existing generation algorithms for tool configuration optimization are difficult to apply in complex production scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for generating workshop tool scheduling rules based on Q-Learning. Background Art

[0002] Customized equipment companies mainly design and produce customized equipment for specific industries or customer needs. Their products usually contain a large number of special parts, such as special medical equipment, special industrial machinery equipment, etc. The processing workpieces and processes of such products are diverse, and the processing process involves a large number of various tools. There is a great demand for the scheduling and optimization of tool resources. Optimizing tool configuration can not only effectively improve machine utilization and production efficiency and reduce idling energy consumption during the production process, but also reduce the number of tools required for the processing process and improve tool utilization, thereby increasing cost-effectiveness.

[0003] Although there are now generation algorithms for tool configuration optimization, they take relatively few factors into consideration, which makes it difficult for the algorithm to solve complex production scenarios with many factors. Summary of the Invention

[0004] In order to overcome the defects of the prior art, the present invention provides a method for generating workshop tool scheduling rules based on Q-Learning to solve the above problems.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for generating workshop tool scheduling rules based on Q-Learning, comprising the following steps:

[0006] S1: Using the set of workpieces J in the machine m , the process set of the workpiece set Workpiece j m Process Processing time The number of operations w, the machine number m, the tool number k, the number of tool sub-tools h and the tool life life kd ,construct a scheduling model about minimizing the maximum completion time and the weighted sum of minimizing the number of tool changes;

[0007] S2: Obtain the priority U of each machine according to the machine priority heuristic rule m ;

[0008] S3: Allocate tools to each machine according to the heuristic rule of the tool number of the initial tool matching and the heuristic rule of the tool number of the tool change;

[0009] S4: Output tool allocation combination of each machine processing task set;

[0010] S5: Establish state characteristics based on the number of machines v, the number of workpieces n, the number of single-workpiece processes k, the number of single-product sub-tools h, and the ratio of the total processing time of a single process to the available tool life r; divide the state characteristics into multiple state spaces s;

[0011] S6: Generate a Q-Learning model through the state space s and use the Q-Learning algorithm to output the workshop tool scheduling plan through the Q-Learning model.

[0012] It is worth noting that when executing step S2 to obtain the priority U of each machine m Before the step, the following steps are performed: Initialize the time step t on each machine m , get the set of workpieces J on each machine m , process set Get the machine number m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd .

[0013] Preferably, step S2 includes the following sub-steps:

[0014] S21: Get the set of workpieces J on each machine m , process set and processing time Initialize the machine number m=1;

[0015] S22: Determine whether the machine number m is equal to the maximum number of machines v. If so, jump to step S26;

[0016] S23: Execution P m is the total working hours of the machine, where j m Workpiece number, n m For the last workpiece, is the process number, For the last process;

[0017] S24: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S25; if no, update the process And jump to step S23;

[0018] S25: Determine the current workpiece j mIs this the last workpiece n? m If yes, jump to step S22; if no, update workpiece j m =j m +1 and update process And jump to step S23;

[0019] S26: Total working hours P of each machine m Value, assign machine priority U according to the machine priority heuristic rule m The machine priority heuristic rules include the PLPT rule that gives high priority to machines with the longest total working hours and the PSPT rule that gives high priority to machines with the shortest total working hours.

[0020] S27: Output the machine priority U of each machine under different machine priority heuristic rules m value.

[0021] Optionally, in step S3, allocating tools to each machine according to the heuristic rule of the tool numbers of the initial tool matching includes the following sub-steps:

[0022] S31: Get the job set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S32.

[0023] S32: Execution like Update Then execute step S33; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time;

[0024] S33: Get the current time step t through the heuristic rule of the tool number of the initial tool m Workpiece j on machine m m Process Tool d distribution combination;

[0025] S34: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S35, if no, update the process And jump to step S33;

[0026] S35: Determine the current workpiece j mIs this the last workpiece n? m If yes, go to step S36, if no, update workpiece j m =j m +1 and update process And jump to step S33;

[0027] S36: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

[0028] Specifically, in step S3, allocating tools to each machine according to the heuristic rule of tool numbers for tool change includes the following sub-steps:

[0029] S37: Get the workpiece set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S38.

[0030] S38: Execution like Update life kd =0, and t m =t m +life kd , then execute step S39; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time, The machine numbered m is at the current time step t m Workpiece j m Process Remaining processing time;

[0031] S39: Get the current time step t through the heuristic rule of tool number of tool change m Workpiece j on machine m m Process Remaining process hours The tool is assigned with d' and the process goes to step S310;

[0032] S310: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S311, if no, update the process And jump to step S39;

[0033] S311: Determine the current workpiece j m Is this the last workpiece n? m If yes, go to step S312, if no, update workpiece j m =j m +1 and update process And jump to step S39;

[0034] S312: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

[0035] It is worth noting that step S33 includes the following sub-steps:

[0036] S331: Get the machine m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd ;

[0037] S332: The rule KTLL of giving priority to the lowest remaining tool life, the rule KTHL of giving priority to the highest remaining tool life, and the rule KTLEPT of giving priority to the tool life equal to the process duration are combined to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. Tool d is assigned to the current time step t according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. m Workpiece j on machine m m Process And output the tool d allocation combination during initial tool allocation under different tool scheduling rules.

[0038] Preferably, the step S39 includes the following sub-steps:

[0039] S391: Get the machine m at the current time step t m Workpiece j m Process Remaining processing time Secondary tool number set D and tool life life kd ;

[0040] S392: Move workpiece j on machine m m Process Divide into processed operations o jm,1 and unprocessed process jm,2 ;

[0041] S393: The KTLL rule of giving priority to the lowest remaining tool life, the KTHL rule of giving priority to the highest remaining tool life, and the KTLEPT rule of giving priority to the tool equal process time are combined to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule, and the tool d' is allocated to the unprocessed process o according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. jm,2 ; And output the tool d' allocation combination when changing tools under different tool scheduling rules.

[0042] Optionally, step S5 includes the following sub-steps:

[0043] S51: The state characteristics include a workpiece flow characteristic f1, a tool flow characteristic f2, and a connection characteristic f3 between the workpiece flow and the tool flow; the state characteristics are divided into the workpiece flow characteristic f1, the tool flow characteristic f2, and the connection characteristic f3 between the workpiece flow and the tool flow according to the number of machines v, the number of workpieces n, the number of single-workpiece processes k, the number of single-variety sub-tools h, and the ratio r of the total processing time of a single process to the available tool life, wherein the workpiece flow characteristic f1 is the sum of the normalized representations of the number of machines v, the number of workpieces n, and the number of single-workpiece processes k, the tool flow characteristic f2 is the normalized representation of the number of single-variety sub-tools h, and the connection characteristic f3 between the workpiece flow and the tool flow is the normalized representation of the ratio r of the total processing time of a single process to the available tool life;

[0044] S52: Divide the workpiece flow feature f1 into 5 state spaces, divide the tool flow feature f2 into 4 state spaces, and divide the connection feature f3 between the workpiece flow and the tool flow into 4 state spaces.

[0045] Specifically, step S6 includes the following sub-steps:

[0046] S61: Initialize various learning parameters and enter the production environment learning within the current learning cycle;

[0047] S62: Enter state space s - And calculate the current state space s - and action a - The maximum completion time C max and the number of tool changes E; wherein each action is a tool allocation combination of a machining task set output in step S4;

[0048] S63: Calculate the reward value according to the reward function, the reward function is reward = ω1 × reward1 + ω2 × reward2, Where ω1 and ω2 are the reward weights of the reward function, and E (s,a) are the maximum completion time C obtained by the agent selecting action a in the state space s max and tool change times E, and are the minimum C obtained by all actions in state s max Value and E value;

[0049] Then update the Q-Learning algorithm Where A is the set of all actions, s - is the current state space, a - is the current action, s is the next state space, a is the next action, α ‘ is the learning rate, reward(s|s - ,a - ) is the current state using a - The reward value corresponding to the action, γ is the discount coefficient of the reward;

[0050] S64: According to Select an action a from the action set A so that when δ<ε, the action a corresponding to the maximum Q value of the state space s is selected, and when δ≥ε, the Q value vector [Q(s,a 1 ),Q(s,a 2 ),...,Q(s,a 32 )], the action probability P(s,a) corresponding to each Q value is obtained, and then the roulette wheel is used to select the action a according to the action probability distribution; where δ represents a random decimal from 0 to 1, ε is the parameter in the ε-greedy action selection strategy, and the action probability e is Euler's constant, g is the element number in the Q value vector;

[0051] S65: Updates - =s,a - =a;

[0052] S66: Determine whether the learning of all environments is completed. If so, jump to step S67; otherwise, jump to step S61;

[0053] S67: Determine whether all learning cycles are completed. If yes, jump to step S68; otherwise, jump to step S61;

[0054] S68: After learning is completed, the Q-Learning model is output, and the workshop tool scheduling plan is output through the Q-Learning model.

[0055] The beneficial effects of the present invention are as follows: in the Q-Learning-based workshop tool scheduling rule generation method, comprehensive consideration is given to tool life and quantity, tool size, and machine tool magazine capacity. Using the combined rule algorithm library as a reinforcement learning action set is equivalent to allowing the agent to learn the optimal rule combination under various production environments. This trains a rule combination model that can adapt to complex production environments. This trained model inherits the rapid solution characteristics of the rule algorithm and can immediately make optimal action decisions under different production environments and obtain new scheduling solutions based on the optimal action. Using the combined rule algorithm library as the reinforcement learning action set is also conducive to subsequent action improvement and upgrading. Workshop tool schedulers can continuously update and improve the rule algorithm library to improve the quality of action selection without changing the overall reinforcement learning algorithm framework, thereby reducing the design and maintenance costs of the algorithm. The Q-learning model obtained by the Q-learning algorithm can find the optimal solution in all factor combinations, improving adaptability and stability when handling complex scenario problems. In Q-learning algorithms with higher reward weights, it can achieve better results than traditional algorithms and solve problems much faster than other algorithms. At the same time, the Q-learning algorithm can focus on the scene optimization goal by changing the values of the weights of the two reward functions, making the algorithm more flexible in complex scene problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of a method for generating shop tool scheduling rules based on Q-Learning in one embodiment of the present invention;

[0057] Figure 2 A schematic diagram of tool scheduling in an intelligent workshop according to an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of tool allocation in one embodiment of the present invention

[0059] Figure 4 Schematic diagram of tool change allocation in one embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is a further description of specific embodiments of the present invention in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is intended to facilitate understanding of the present invention and does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0061] The following relationships exist during the modeling process:

[0062] 1) Set J of workpieces for each machine m There is a definite close relationship between each workpiece;

[0063] 2) There is a definite precedence and follow-up relationship between the processes within each workpiece;

[0064] 3) One tool can only process one process;

[0065] 4) The same knife cannot appear on more than one machine at the same time;

[0066] 5) There is no difference between the tool magazines of the machines, and all can accommodate the same type of knives, but each type of knife has only one slot available;

[0067] 6) The number of auxiliary tools of a certain type in all machine tool magazines cannot exceed h at any time;

[0068] 7) Unless the tool life becomes zero during the machining process, the tool can be adjusted only after the current process is completed;

[0069] 8) The process can only be started when the tool life of the current process in the machine is not zero;

[0070] 9) Assume that all required tools are in the machine tool magazine at the beginning;

[0071] 10) Ignore the tool transportation time and tool change time between the tool holder and the machine tool magazine.

[0072] A method for generating shop tool scheduling rules based on Q-Learning includes the following steps:

[0073] S1: This solution can be understood as a tool parallel machine scheduling optimization problem considering multiple machines, tool life and quantity, uneven tool size, and limited machine tool magazine capacity, such as Figure 2 As shown, using the workpiece set J in the machine m , the process set of the workpiece set Workpiece j m Process Processing time The number of operations w, the machine number m, the tool number k, the number of tool sub-tools h and the tool life life kd , build a scheduling model that minimizes the maximum completion time and the weighted sum of the number of tool changes; the set of workpieces J in each machine number m has been determined in advance, and the set of processes O of each workpiece set is known. This batch of processing tasks contains a total of w processes, and each workpiece j m Each process Processing time The number of tools in the workshop may be the same or different, depending on the actual order. At the same time, there are m machines and k types of tools in the workshop (where k≤w). One tool represents a type of tool size. Each tool contains h sub-tools. The life of each tool is life kd May be the same or different, depending on the actual tool library information; the optimization goal of the problem is to minimize the maximum completion time C max The tool configuration corresponding to each process of each machine is obtained by minimizing the weighted sum of the number of tool changes E.

[0074] The parameters used in the scheduling model constructed in this solution are as follows:

[0075] m: machine number index;

[0076] j m : Workpiece number index;

[0077] The process number index of the workpiece;

[0078] k, k ‘ : Tool type number index, using k and k ‘ To represent two different types of processes, and so on;

[0079] d, d': tool number index of a single type of auxiliary tool. d and d' are used to represent two different types of auxiliary tools, and so on;

[0080] t, t ‘ , t": time step number index, using t, t ‘ and t” to represent three different time steps, and so on;

[0081] Tool life index at the current time step;

[0082] Processing time index;

[0083] The processing timer on each machine is a 0-1 variable;

[0084] α: is a weight factor between 0 and 1, used to adjust the trade-off between the two objectives;

[0085] C max : Maximum completion time;

[0086] E: number of tool changes;

[0087] Machine number set: M = {m|1,2,...,v}, where v is the maximum number of machines;

[0088] Workpiece number collection for each machine: J m ={j m |1,2,...,n m}, n m For the last artifact;

[0089] The process number set for each workpiece: is the last process, in which w is the maximum number of process types;

[0090] Process type (tool type) number set for machining tasks: K = {k|1,2,...,w}; w is the maximum number of process types;

[0091] Tool number set of a single type of auxiliary tool: D = {d|1,2,...,h}; h is the maximum number of auxiliary tools;

[0092] Time step number set: T = {t|1,2,...,L}; L is the maximum number of time steps;

[0093] The decision variables of the scheduling model constructed in this solution are as follows:

[0094]

[0095]

[0096] The objective function of the scheduling model constructed in this solution is as follows:

[0097] min f=α*C max +(1-α)*E; represents the weighted sum of minimizing the maximum completion time and the number of tool changes, where α is a weight factor between 0 and 1, used to adjust the weight between the two objectives;

[0098] The constraints of the scheduling model constructed in this solution are as follows:

[0099]

[0100] Indicates the calculation method of the maximum completion time;

[0101] Indicates the calculation method of tool change times;

[0102]

[0103] It indicates that the workpieces of each machine processing task set have a close sequence relationship; Indicates that the processing steps of each workpiece have a sequence relationship between them. These two constraints represent the constraints of known workpiece process sequence information.

[0104]

[0105] It means that one tool can only process one process;

[0106] This means that the same knife cannot appear on more than one machine at the same time;

[0107] It means that there is no more than one auxiliary tool of the same type on a machine;

[0108] It means that the number of tool types on a machine is no more than the tool slot capacity a;

[0109] It means that the number of auxiliary tools of various types on all machines at the same time is no more than the total number of auxiliary tools of the same type in the workshop;

[0110]

[0111] It is equivalent to a virtual timer, which means that the processing time of each machine's workpiece set is counted until the last process of the machine is completed;

[0112] Indicates that the process can only be started when the tool life of the current process in the machine is not zero;

[0113] Indicates that the tool change due to the end of tool life occurs at the end of tool life kd When it is 0 or after 0, that is, once the tool is moved out of the central tool magazine, it needs to wait until the end of its life or the processing tasks of all machines are completed before it can be returned to the central tool magazine;

[0114]

[0115] Indicates the record of tool changes that occur other than the end of tool life (tool borrowing between machines);

[0116] Represents the guaranteed time t ‘ Life of new blades replaced due to end of life Not 0;

[0117]

[0118] Indicates the update calculation method of tool life at each time step;

[0119]

[0120] Indicates that unless the tool life becomes zero during the machining process, the tool cannot be adjusted until the current process is completed, that is, the process processing is not interrupted.

[0121] It represents the upper bound constraint of the time step, that is, the maximum completion time of all processing tasks in the workshop;

[0122]

[0123] represents the non-negativity constraint on the decision variables.

[0124] Regarding workpiece process characteristics, the products of custom equipment companies contain a large number of special parts, and the processing tasks are characterized by a wide variety of workpieces and processes. The main problem with processing workpieces and processes is that each workpiece has a process with different working times (which may also be the same, depending on the actual workpiece processing information). Each workpiece has a different number of processes (which may also be the same), and the processing time of each process can also vary. This unequal number of processes and processing time between workpieces requires that when scheduling tools, the impact of processing time on the scheduling process must be considered from both the processing time of the machine workpiece set and the processing time of each process.

[0125] Regarding tool characteristics, this solution sets constraints on the number of tool sizes, the number of tools, tool life, and the machine tool magazine capacity. Furthermore, the life of each tool, which serves as the input information, is limited and may vary. These characteristics may cause the following phenomena and problems during production:

[0126] 1. Due to the limited nature of tool resources, the larger the workpiece and process scale, the greater the probability of tool resource conflicts when multiple machines simultaneously request the same tool, which can lead to tool allocation decisions. 2. Due to the limited lifespan of tools, tool replacement decisions triggered by the end of tool life also need to be considered.

[0127] Based on the above phenomena, we can analyze the following three key points for solving optimization problems:

[0128] 1. Reasonably handle the conflict of tool resources; 2. Reduce the probability of tool resource conflicts; 3. Make good decisions on tool changes caused by the end of tool life, which are the key to the optimization problem in this paper.

[0129] S2: Obtain the priority U of each machine according to the machine priority heuristic rule m ;

[0130] Among them, when obtaining the priority U of each machine m Before, the following steps are performed: Initialize the time step t on each machine m , get the set of workpieces J on each machine m , process set Get the machine number m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd ;

[0131] The step S2 includes the following sub-steps:

[0132] S21: Get the set of workpieces J on each machine m , process set and processing time Initialize the machine number m=1;

[0133] S22: Determine whether the machine number m is equal to the maximum number of machines v. If so, jump to step S26;

[0134] S23: Execution P m is the total working hours of the machine, where j m Workpiece number, n m For the last workpiece, is the process number, For the last process;

[0135] S24: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S25; if no, update the process And jump to step S23;

[0136] S25: Determine the current workpiece j m Is this the last workpiece n? m If yes, jump to step S22; if no, update workpiece j m =j m +1 and update process And jump to step S23;

[0137] S26: Total working hours P of each machine m Value, assign machine priority U according to the machine priority heuristic rule mThe machine priority heuristic rules include the PLPT rule that gives high priority to machines with the longest total working hours and the PSPT rule that gives high priority to machines with the shortest total working hours.

[0138] S27: Output the machine priority U of each machine under different machine priority heuristic rules m value;

[0139] S3: Allocate tools to each machine according to the heuristic rule of the tool number of the initial tool matching and the heuristic rule of the tool number of the tool change;

[0140] The heuristic rule of assigning tools to each machine based on the tool number of the initial tool assignment includes the following sub-steps:

[0141] S31: Get the job set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S32.

[0142] S32: Execution like Update Then execute step S33; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time;

[0143] S33: Get the current time step t through the heuristic rule of the tool number of the initial tool m Workpiece j on machine m m Process Tool d distribution combination;

[0144] S34: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S35, if no, update the process And jump to step S33;

[0145] S35: Determine the current workpiece j m Is this the last workpiece n? m If yes, go to step S36, if no, update workpiece j m =j m +1 and update process And jump to step S33;

[0146] S36: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

[0147] The heuristic rule of assigning tools to each machine based on the tool number of the tool change includes the following substeps:

[0148] S37: Get the workpiece set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S38.

[0149] S38: Execution like Update life kd =0, and t m =t m +life kd , then execute step S39; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time, The machine numbered m is at the current time step t m Workpiece j m Process Remaining processing time;

[0150] S39: Get the current time step t through the heuristic rule of tool number of tool change m Workpiece j on machine m m Process Remaining process hours The tool is assigned with d' and the process goes to step S310;

[0151] S310: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S311, if no, update the process And jump to step S39;

[0152] S311: Determine the current workpiece j m Is this the last workpiece n? m If yes, go to step S312, if no, update workpiece j m =j m+1 and update process And jump to step S39;

[0153] S312: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

[0154] The step S33 includes the following sub-steps:

[0155] S331: Get the machine m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd ;

[0156] S332: The rule KTLL of giving priority to the lowest remaining tool life, the rule KTHL of giving priority to the highest remaining tool life, and the rule KTLEPT of giving priority to the tool life equal to the process duration are combined to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. Tool d is assigned to the current time step t according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. m Workpiece j on machine m m Process And output the tool d allocation combination when initial tool allocation under different tool scheduling rules;

[0157] like Figure 3 As shown, the time step t of each machine m Tool life under and its process hours There is a close connection: from a local perspective, find a The tool used for machining can effectively reduce the number of tool changes E, but it may cause other processes that require this tool to wait too long, and the probability of subsequent processes falling into tool resource conflicts may also increase; find a Using different tools for machining can make the machining more flexible, but it may increase the number of tool changes E; from a global perspective, allocating suitable tools to all processes can effectively reduce the probability of tool resource conflicts, thereby reducing C max Value and tool change times E. and Based on the relationship between them, the following three scheduling rules are obtained:

[0158] 1. Keep tool lowest lifetime (KTLL) first;

[0159] 2. Keep tool highest lifetime (KTHL) first;

[0160] 3. Keep tool lifetime equal to processing time (KTLEPT), that is, the current time step t m Appear When the tool d is allocated to the process first

[0161] It is worth noting that the KTLEPT tool scheduling rule occurs only when the tool life Equal to process hours KTLEPT tool scheduling rule can only be used when KTLEPT is used. Then KTLEPT tool scheduling rule will be used as an additional rule in conjunction with KTLL or KTHL tool scheduling rule. The symbol & is used as a connector to connect it to KTLL or KTHL tool scheduling rule as a representation, such as KTHL&KTLEPT or KTLL&KTLEPT. The logic of using KTLEPT tool scheduling rule as an additional rule is: when it is determined to adopt KTHL&KTLEPT or KTLL&KTLEPT tool scheduling rule, it should first be judged whether the tool to be allocated by decision meets the life requirement. Equal to process duration If the requirements are met, the tool will be directly used as the decision object for tool allocation. If the requirements are not met, the KTHL tool scheduling rule or KTLL tool scheduling rule will be enabled to perform tool decision allocation operations.

[0162] The step S39 includes the following sub-steps:

[0163] S391: Get the machine m at the current time step t m Workpiece j m Process Remaining processing time Secondary tool number set D and tool life life kd ;

[0164] S392: Move workpiece j on machine m m Process Divide into processed operations o jm,1 and unprocessed process jm,2 ;

[0165] S393: The KTLL rule of giving priority to the lowest remaining tool life, the KTHL rule of giving priority to the highest remaining tool life, and the KTLEPT rule of giving priority to the tool equal process time are combined to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule, and the tool d' is allocated to the unprocessed process o according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. jm,2 ; And output the tool d' allocation combination when changing the tool under different tool scheduling rules;

[0166] like Figure 4 As shown in the figure, after the initial tool life is completed, the nature of the problem faced is also a decision-making allocation for the tool. Therefore, the relationship between tool life and process time is also used as a basis to form the following three rules: the lowest remaining tool life priority KTLL rule, the highest remaining tool life priority KTHL rule and the tool equal to process time priority KTLEPT rule;

[0167] S4: Output the tool allocation combination for each machine processing task set.

[0168] In this way, the tool allocation combinations for each machine processing task set are obtained by combining the machine priority rules (PLPT and PSPT), the tool scheduling rules corresponding to the tool d allocation combination during initial tool allocation (KTLL, KTHL, KTHL&KTLEPT, and KTLL&KTLEPT), and the tool scheduling rules corresponding to the tool d' allocation combination during tool change (KTLL, KTHL, KTHL&KTLEPT, and KTLL&KTLEPT). 32 algorithms can be formed as follows:

[0169] PLPT-KTHL-KTHL (Algorithm 1), PLPT-KTHL-KTHL&KTLEPT (Algorithm 2), PLPT-KTHL-KTLL (Algorithm 3), PLPT-KTHL-KTLL&KTLEPT (Algorithm 4), PLPT-KTHL&KTLEPT-KTHL (Algorithm 5), PLPT-KTHL&KTLEPT-KTHL&KTLEPT (Algorithm 6), PLPT-KTHL&KTLEPT-KTLL (Algorithm 7), PLPT-KTHL&KTLEPT-KTLL&KTLEPT (Algorithm 8), PLPT- KTLL-KTHL (Algorithm 9), PLPT-KTLL-KTHL&KTLEPT (Algorithm 10), PLPT-KTHL-KTHL (Algorithm 11), PLPT-KTLL-KTLL&KTLEPT (Algorithm 12), PLPT-KTLL&KTLEPT-KTHL (Algorithm 13), PLPT-KTLL&KTLEPT-KTHL&KTLEPT (Algorithm 14), PLPT-KTLL&KTLEPT-KTLL (Algorithm 15), PLPT-KTLL&KTLEPT-KTLL&KTLEPT (Algorithm 16), PSPT -KTHL-KTHL (Algorithm 17), PSPT-KTHL-KTHL&KTLEPT (Algorithm 18), PSPT-KTHL-KTLL (Algorithm 19), PSPT-KTHL-KTLL&KTLEPT (Algorithm 20), PSPT-KTHL&KTLEPT-KTHL (Algorithm 21), PSPT-KTHL&KTLEPT-KTHL&KTLEPT (Algorithm 22), PSPT-KTHL&KTLEPT-KTLL (Algorithm 23), PSPT-KTHL&KTLEPT-KTLL&KTLEPT (Algorithm 24), PSPT PT-KTLL-KTHL (Algorithm 25), PSPT-KTLL-KTHL&KTLEPT (Algorithm 26), PSPT-KTHL-KTHL (Algorithm 27), PSPT-KTLL-KTLL&KTLEPT (Algorithm 28), PSPT-KTLL&KTLEPT-KTHL (Algorithm 29), PSPT-KTLL&KTLEPT-KTHL&KTLEPT (Algorithm 30), PSPT-KTLL&KTLEPT-KTLL (Algorithm 31), PSPT-KTLL&KTLEPT-KTLL&KTLEPT (Algorithm 32).

[0170] S5: Establish state characteristics based on the number of machines v, the number of workpieces n, the number of single-workpiece processes k, the number of single-product sub-tools h, and the ratio of the total processing time of a single process to the available tool life r; divide the state characteristics into multiple state spaces s;

[0171] The step S5 comprises:

[0172] S51: The state characteristics include workpiece flow characteristics f1, tool flow characteristics f2 and the connection characteristics f3 between the workpiece flow and the tool flow; according to the number of machines v, the number of workpieces n, the number of single workpiece processes k, the number of single-variety sub-tools h, and the ratio r of the total processing time of a single process to the available tool life, the workpiece flow characteristics f1, the tool flow characteristics f2 and the connection characteristics f3 between the workpiece flow and the tool flow, wherein the workpiece flow characteristics f1 is the sum of the normalized representations of the number of machines v, the number of workpieces n and the number of single workpiece processes k, the tool flow characteristics f2 is the normalized representation of the number of single-variety sub-tools h, and the connection characteristics f3 between the workpiece flow and the tool flow is the normalized representation of the ratio r of the total processing time of a single process to the available tool life; wherein the workpiece flow characteristics Toolflow Features Characteristics of the connection between workpiece flow and tool flow

[0173] S52: Divide the workpiece flow feature f1 into 5 state spaces (respectively [0, 0.2), [0.2, 0.4), [0.4, 0.6), [0.6, 0.8), [0.8, 1]), divide the tool flow feature f2 into 4 state spaces (respectively [0, 0.25), [0.25, 0.5), [0.5, 0.75), [0.75, 1]), and divide the connection feature f3 between the workpiece flow and the tool flow into 4 state spaces (respectively [0, 0.25), [0.25, 0.5), [0.5, 0.75), [0.75, 1]). The number of state space divisions depends on the system's requirements for solution accuracy and solution time cost. The higher the accuracy requirement, the longer the solution time is usually. According to the number of actions and the number of state space divisions in this scheme, the Q value table is an 80×32 table.

[0174] S6: Generate a Q-Learning model using the state space s and the Q-Learning algorithm:

[0175] S60: Obtain offline learning data for all production environment simulations. The Q-Learning algorithm has two forms: online learning and offline learning. Online learning refers to the real-time interaction between the agent and the environment, updating the Q-value function based on the reward signal at each time step. It is suitable for problem scenarios that require real-time decision-making. Offline learning refers to the agent learning using pre-collected data sets. It is suitable for situations where data acquisition is difficult or costly. Given the high cost of trial and error data collection for each algorithm in each production environment in actual production environments, this solution uses offline learning combined with simulation technology.

[0176] S61: Initialize various learning parameters and enter the production environment learning within the current learning cycle;

[0177] S62: Enter state space s - And calculate the current state space s - and action a - The maximum completion time C max and the number of tool changes E; each action is a tool allocation combination of a machining task set output in step S4; in this solution, since there are 32 tool allocation combinations of machining task sets, 32 actions will be obtained, and each action corresponds to a maximum completion time C max and tool change times E;

[0178] S63: Calculate the reward value according to the reward function, the reward function is reward = ω1 × reward1 + ω2 × reward2, Where ω1 and ω2 are the reward weights of the reward function, and E (s,a) are the maximum completion time C obtained by the agent selecting action a in the state space s max and tool change times E, and are the minimum C obtained by all actions in state s max Value and E value;

[0179] Then update the Q-Learning algorithm Where A is the set of all actions, s - is the current state space, a - is the current action, s is the next state space, a is the next action, α ‘ is the learning rate, reward(s|s - ,a - ) is the current state using a - The reward value corresponding to the action, γ is the discount coefficient of the reward; the Q(s - ,a - ) The value update formula is to ensure that the reward value of the reward function can only be negative or zero;

[0180] S64: According to Select an action a from the action set A so that when δ<ε, the action a corresponding to the maximum Q value of the state space s is selected, and when δ≥ε, the Q value vector [Q(s,a 1 ),Q(s,a 2 ),...,Q(s,a32 )], the action probability P(s,a) corresponding to each Q value is obtained, and then the roulette wheel is used to select the action a according to the action probability distribution; where δ represents a random decimal from 0 to 1, ε is the parameter in the ε-greedy action selection strategy, and the action probability e is the Euler constant, g is the element number in the Q value vector; traditional Q-Learning algorithms mostly use the ε-greedy strategy for action selection. In order to increase the convergence speed and improve the stability of the algorithm while ensuring that the algorithm has good exploratory properties, this scheme improves the ε-greedy action selection strategy and obtains This makes it easier for actions with high probability values to be selected, improving the convergence speed and stability of the algorithm. At the same time, the roulette action selection operation also retains the exploratory nature of the algorithm.

[0181] S65: Updates - =s,a - =a;

[0182] S66: Determine whether the learning of all environments is completed. If so, jump to step S67; otherwise, jump to step S61;

[0183] S67: Determine whether all learning cycles are completed. If yes, jump to step S68; otherwise, jump to step S61;

[0184] S68: After learning is completed, the Q-Learning model is output, and the workshop tool scheduling plan is output through the Q-Learning model.

[0185] The basic idea of Q-learning is to approximate the optimal Q function by continuously updating the Q value, so that the agent can choose the optimal action in the environment. In each step, the agent takes the action according to the current state s. - Select action a in action set A - , and observe the reward reward and next state s given by the environment, by continuously updating Q(s - ,a - ), which is the Q value, and the process is repeated until the Q value converges, at which point the action decision tends to be stable.

[0186] In the Q-Learning-based method for generating shop tool scheduling rules, using the combined rule algorithm library as the reinforcement learning action set is equivalent to having the agent learn the optimal rule combinations under various production environments. This results in a trained rule combination model that can adapt to complex production environments. This trained model inherits the rapid solution characteristics of the rule algorithm and can also immediately make optimal action decisions under different production environments and obtain new scheduling solutions based on the optimal actions. Using the combined rule algorithm library as the reinforcement learning action set also facilitates subsequent action improvements and upgrades. Shop tool schedulers can continuously update and improve the rule algorithm library to improve the quality of action selection without changing the overall reinforcement learning algorithm framework, thereby reducing the design and maintenance costs of the algorithm.

[0187] The Q-learning algorithm in this solution was able to find the optimal solution for all factor combinations, demonstrating its adaptability and stability in complex scenarios. In cases with high reward weights, the Q-learning algorithm achieved better results than the genetic algorithm and solved the problem much faster than other algorithms. Furthermore, the Q-learning algorithm can prioritize the optimization objective of a scenario by varying the weights of the two reward functions, making the algorithm more flexible in complex scenarios.

[0188] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A Q-Learning-based method for generating shop tool scheduling rules, characterized by: The following steps are involved: S1: Using the set of workpieces J in the machine m , the process set of the workpiece set Workpiece j m Process Processing time The number of operations w, the machine number m, the tool number k, the number of tool sub-tools h and the tool life life kd ,construct a scheduling model about minimizing the maximum completion time and the weighted sum of minimizing the number of tool changes; S2: Obtain the priority U of each machine according to the machine priority heuristic rule m ; S3: Allocate tools to each machine according to the heuristic rule of the tool number of the initial tool matching and the heuristic rule of the tool number of the tool change; S4: Output tool allocation combination of each machine processing task set; S5: Establish state characteristics based on the number of machines v, the number of workpieces n, the number of single workpiece processes k, the number of single-product sub-tools h, and the ratio of the total processing time of a single process to the available tool life r; The state characteristics are divided into multiple state spaces s, which The following steps are included: S51: The state characteristics include a workpiece flow characteristic f1, a tool flow characteristic f2, and a connection characteristic f3 between the workpiece flow and the tool flow; the state characteristics are divided into the workpiece flow characteristic f1, the tool flow characteristic f2, and the connection characteristic f3 between the workpiece flow and the tool flow according to the number of machines v, the number of workpieces n, the number of single-workpiece processes k, the number of single-variety sub-tools h, and the ratio r of the total processing time of a single process to the available tool life, wherein the workpiece flow characteristic f1 is the sum of the normalized representations of the number of machines v, the number of workpieces n, and the number of single-workpiece processes k, the tool flow characteristic f2 is the normalized representation of the number of single-variety sub-tools h, and the connection characteristic f3 between the workpiece flow and the tool flow is the normalized representation of the ratio r of the total processing time of a single process to the available tool life; S52: Divide the workpiece flow feature f1 into 5 state spaces, divide the tool flow feature f2 into 4 state spaces, and divide the connection feature f3 between the workpiece flow and the tool flow into 4 state spaces; S6: Generate a Q-Learning model using the state space s and the Q-Learning algorithm, and output a shop tool scheduling plan through the Q-Learning model, which includes the following steps: S61: Initialize various learning parameters and enter the production environment learning within the current learning cycle; S62: Enter state space s - And calculate the current state space s - and action a - The maximum completion time C max and the number of tool changes E; wherein each action is a tool allocation combination of a machining task set output in step S4; S63: Calculate the reward value according to the reward function, the reward function is reward = ω1 × reward1 + ω2 × reward2, Where ω1 and ω2 are the reward weights of the reward function, and E (s,a) are the maximum completion time C obtained by the agent selecting action a in the state space s max and tool change times E, and are the minimum C obtained by all actions in state s max Value and E value; Then update the Q-Learning algorithm Where A is the set of all actions, s - is the current state space, a - is the current action, s is the next state space, a is the next action, α ‘ is the learning rate, reward(s|s - ,a - ) is the current state using a - The reward value corresponding to the action, γ is the discount coefficient of the reward; S64: According to Select an action a from the action set A so that when δ<ε, the action a corresponding to the maximum Q value of the state space s is selected, and when δ≥ε, the Q value vector [Q(s,a 1 ),Q(s,a 2 ),...,Q(s,a 32 )], the action probability P(s,a) corresponding to each Q value is obtained, and then the roulette wheel is used to select the action a according to the action probability distribution; where δ represents a random decimal from 0 to 1, ε is the parameter in the ε-greedy action selection strategy, and the action probability e is Euler's constant, g is the element number in the Q value vector; S65: Updates - =s,a - =a; S66: Determine whether the learning of all environments is completed. If so, jump to step S67; otherwise, jump to step S61; S67: Determine whether all learning cycles are completed. If yes, jump to step S68; otherwise, jump to step S61; S68: After learning is completed, the Q-Learning model is output, and the workshop tool scheduling plan is output through the Q-Learning model.

2. The method for generating shop tool scheduling rules based on Q-Learning according to claim 1, characterized in that: When executing step S2, the priority U of each machine is obtained. m Before the step, the following steps are performed: Initialize the time step t on each machine m , get the set of workpieces J on each machine m , process set Get the machine number m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd .

3. The method for generating workshop tool scheduling rules based on Q-Learning according to claim 2, characterized in that: The step S2 includes the following sub-steps: S21: Get the set of workpieces J on each machine m , process set and processing time Initialize the machine number m=1; S22: Determine whether the machine number m is equal to the maximum number of machines v. If so, jump to step S26; S23: Execution P m is the total working hours of the machine, where j m Workpiece number, n m For the last workpiece, is the process number, For the last process; S24: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S25; if no, update the process And jump to step S23; S25: Determine the current workpiece j m Is this the last workpiece n? m If yes, jump to step S22; if no, update workpiece j m =j m +1 and update process And jump to step S23; S26: Total working hours P of each machine m Value, assign machine priority U according to the machine priority heuristic rule m The machine priority heuristic rules include the PLPT rule that gives high priority to machines with the longest total working hours and the PSPT rule that gives high priority to machines with the shortest total working hours. S27: Output the machine priority U of each machine under different machine priority heuristic rules m value.

4. The method for generating workshop tool scheduling rules based on Q-Learning according to claim 3, characterized in that: In step S3, allocating tools to each machine according to the heuristic rule of the tool number of the initial tool allocation includes the following sub-steps: S31: Get the job set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S32. S32: Execution like Update Then execute step S33; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time; S33: Get the current time step t through the heuristic rule of the tool number of the initial tool m Workpiece j on machine m m Process Tool d distribution combination; S34: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S35, if no, update the process And jump to step S33; S35: Determine the current workpiece j m Is this the last workpiece n? m If yes, go to step S36, if no, update workpiece j m =j m +1 and update process And jump to step S33; S36: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

5. The method for generating workshop tool scheduling rules based on Q-Learning according to claim 4, characterized in that: In step S3, allocating tools to each machine according to the heuristic rule of tool numbers for tool change includes the following sub-steps: S37: Get the workpiece set J of the current priority machine m m and process set Determine whether the current machine is the lowest priority machine. If yes, go to step S4. If not, go to step S38. S38: Execution like Update life kd =0, and t m =t m +life kd , then execute step S39; wherein life kd is the tool life, The machine numbered m is at the current time step t m Workpiece j m Process processing time, The machine numbered m is at the current time step t m Workpiece j m Process Remaining processing time; S39: Get the current time step t through the heuristic rule of tool number of tool change m Workpiece j on machine m m Process Remaining process hours The knife with d ′ Assign a combination and jump to step S310; S310: Determine the current process Is it the current workpiece j? m The last step If yes, go to step S311, if no, update the process And jump to step S39; S311: Determine the current workpiece j m Is this the last workpiece n? m If yes, go to step S312, if no, update workpiece j m =j m +1 and update process And jump to step S39; S312: Determine whether the machine with the current machine number m is the machine with the lowest priority. If so, jump to step S4. If not, update j. m = 1 and advance to the next priority machine.

6. The method for generating workshop tool scheduling rules based on Q-Learning according to claim 5, characterized in that: The step S33 includes the following sub-steps: S331: Get the machine m at the current time step t m Workpiece j m Process Processing time Secondary tool number set D and tool life life kd ; S332: The rule KTLL of giving priority to the lowest remaining tool life, the rule KTHL of giving priority to the highest remaining tool life, and the rule KTLEPT of giving priority to the tool life equal to the process duration are combined to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. Tool d is assigned to the current time step t according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. m Workpiece j on machine m m Process And output the tool d allocation combination during initial tool allocation under different tool scheduling rules.

7. The method for generating workshop tool scheduling rules based on Q-Learning according to claim 6, characterized in that: The step S39 includes the following sub-steps: S391: Get the machine m at the current time step t m Workpiece j m Process Remaining processing time Secondary tool number set D and tool life life kd ; S392: Move workpiece j on machine m m Process Divide into processed operations o jm,1 and unprocessed process jm,2 ; S393: Combine the KTLL rule of giving priority to the lowest remaining tool life, the KTHL rule of giving priority to the highest remaining tool life, and the KTLEPT rule of giving priority to the tool equal process time to obtain the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule, and assign tool d according to the KTLL tool scheduling rule, the KTHL tool scheduling rule, the KTHL&KTLEPT tool scheduling rule, and the KTLL&KTLEPT tool scheduling rule. ′ To the unprocessed process o jm,2 ; and output the tool d when changing the tool under different tool scheduling rules ′ Allocation combination.

Images