Reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution

By applying dynamic scheduling methods with reinforcement learning and rule evolution in reconfigurable workshops, and using DQN algorithm and genetic planning algorithm to optimize workpiece processing plans, the efficiency and optimization problems of traditional scheduling methods under dynamic events are solved, and more efficient Production scheduling and resource utilization.

CN118963272AActive Publication Date: 2024-11-15HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202411024659.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

When facing dynamic events, traditional reconfigurable workshop scheduling methods are difficult to adjust the production schedule in real time, resulting in reduced efficiency of the production and manufacturing process and the real-time and optimization quality cannot be guaranteed.

Method used

Using a dynamic scheduling method based on reinforcement learning methods and rule evolution, the decision-making agent of the DQN algorithm is constructed by initializing the task pool and the device pool, and the overall drag period of artifact processing is optimized using neural networks. The action space is compounded by the evolution of the genetic planning algorithm. Scheduling rules, combining the hierarchical strategies and statistical feature information of topological units, dynamically adjust the processing plan of the workpiece.

Benefits of technology

It improves the real-time and optimized quality of reconfigurable workshop scheduling, reduces enterprise management costs, ensures the stable operation of the production and manufacturing system, can quickly respond to new production conditions and adapt to complex scenarios, and reduces the total delay period of workpiece processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963272A_ABST
    Figure CN118963272A_ABST
Patent Text Reader

Abstract

The invention discloses a reconfigurable workshop dynamic scheduling method based on a reinforcement learning method and rule evolution, which belongs to the field of workshop scheduling, and comprises the following steps of: selecting and combining heuristic scheduling rules by using a genetic programming algorithm by taking the minimum total tardiness cost of workpieces as a target to obtain a composite scheduling rule for guiding the selection of processing workpieces; a hierarchical strategy of efficiency-first topological unit allocation and recombination is constructed, Double DQN, dueling DQN and a priority experience playback mechanism are fused to improve the optimization ability of a DQN algorithm, an intelligent agent neural network model fusing a recurrent neural network and a multi-layer perceptron is constructed, time sequence information of workshop processing task allocation is extracted in the training process, and the optimization ability of the DQN algorithm is improved. Meanwhile, a dynamic event response mechanism oriented to arrival of a new workpiece is established. According to the method, the real-time performance and optimization quality of reconfigurable workshop scheduling can be improved, and the utilization rate of manufacturing resources is improved, so that the production cost of an enterprise is reduced, and stable operation of a manufacturing system is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of workshop scheduling, and more specifically, relates to a reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution. Background Art

[0002] With the rapid development of the global manufacturing industry and the ever-changing trend of science and technology, the importance of intelligent manufacturing has become increasingly prominent. As a product of the deep integration of the new generation of information technology and manufacturing, the core of intelligent manufacturing lies in realizing the automation, informatization, networking and flexibility of the manufacturing process through intelligent means. Among the many fields of intelligent manufacturing, the workshop scheduling problem has always attracted much attention from academia and industry. As an important means to optimize the production process and improve production efficiency, workshop scheduling can not only improve production efficiency and reduce production costs, but also effectively respond to the rapid changes in market demand and promote the sustainable development of enterprises. Its research value and application prospects are becoming increasingly prominent.

[0003] In the actual manufacturing process, the reconfigurable workshop is a novel and efficient production model that can meet diverse manufacturing needs and is widely used in multi-type mixed-line assembly workshops. The reconfigurable workshop can efficiently reorganize the workshop equipment resources, thereby improving the utilization rate of manufacturing resources. Compared with the traditional job workshop, the scheduling of the reconfigurable workshop has a more complex problem space. On the basis of considering the flexibility of the processing process, the reconfigurable workshop also needs to formulate a reorganization strategy for equipment resources. The introduction of the reconstruction problem, while improving the production capacity of the manufacturing workshop, also brings greater uncertainty to the scheduling. Therefore, it is of great significance to study the scheduling problem of the reconfigurable workshop.

[0004] For the scheduling of reconfigurable workshops, it is ultimately necessary to formulate a corresponding production scheduling plan, that is, to determine the processing start time, processing end time and allocated resources for each process of each workpiece. The traditional reconfigurable workshop scheduling method is a multi-dimensional static scheduling method, which directly determines the production scheduling plan based on the process information of the workpieces to be processed and the resource information in the reconfigurable workshop. However, in the actual production process of the reconfigurable workshop, dynamic events such as the arrival of new workpieces occur frequently, making the original production scheduling plan no longer feasible. After the dynamic event occurs, the traditional static scheduling method needs to spend extra time to re-solve the scheduling plan, which reduces the efficiency of the production process, and the real-time and optimization quality cannot be guaranteed. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution, which aims to improve the real-time and optimization quality of reconfigurable workshop scheduling, thereby reducing the management cost of the enterprise and ensuring the stable operation of the production and manufacturing system.

[0006] To achieve the above object, according to one aspect of the present invention, a method for dynamic scheduling of a reconfigurable workshop based on reinforcement learning method and rule evolution is provided, comprising: initializing a task pool and an equipment pool, which are respectively used to store all workpieces to be processed and idle equipment units in the reconfigurable workshop in real time, and constructing a decision-making agent based on the DQN algorithm; the optimization goal of the decision-making agent is to minimize the total delay of workpiece processing, and the action space is composed of multiple composite scheduling rules evolved from a preset heuristic rule for selecting workpieces through a genetic programming algorithm;

[0007] At each decision point, dynamic scheduling is performed according to the following steps:

[0008] S1: Feedback the current state feature information of the reconfigurable workshop and the reward value after executing the action obtained in the previous decision to the decision agent, train the neural network in the decision agent and output the probability of each composite scheduling rule being selected in the action space by the neural network; the smaller the total delay of workpiece processing, the greater the reward value;

[0009] S2: Select the composite scheduling rule with the highest probability as the action obtained by the decision agent at the current decision point, select the workpiece to be processed from the task pool according to the selected composite scheduling rule, and allocate the topological unit required by the current process to it;

[0010] After all the workpieces to be processed are processed, the production schedule of the reconfigurable workshop is output; the production schedule includes the processing start time, processing end time and the allocated topological unit of each process of each workpiece;

[0011] The decision point is the start time, or the time when the topological unit releases the workpiece after completing the processing of the workpiece process and there are unfinished workpieces with no assigned processing tasks in the reconfigurable workshop.

[0012] Furthermore, the composite scheduling rules are represented by a binary tree structure;

[0013] The internal nodes of a binary tree are operation symbols, which are used to represent the operation relationship between child nodes;

[0014] The leaf nodes of the binary tree are the heuristic rules for selecting artifacts;

[0015] The operation symbols include: "+", "-", "×", "÷", "Max", and "Min". When the divisor is 0, the "÷" operation result returns the preset value; "Max" means taking the maximum value, and "Min" means taking the minimum value.

[0016] The heuristics for selecting artifacts include:

[0017] a) SPT: Select the workpiece with the shortest processing time;

[0018] b) LPT: Select the workpiece with the longest processing time;

[0019] c) SSO: Select the workpiece with the shortest processing time in the subsequent process;

[0020] d) LSO: select the workpiece with the longest processing time in the subsequent process;

[0021] e) SRPT: select the workpiece with the shortest remaining processing time;

[0022] f) LRPT: select the workpiece with the longest remaining processing time;

[0023] g) EDD: Select the workpiece with the earliest delivery date;

[0024] h)FIFO: First-in-first-out rule;

[0025] i) LNRO: select the workpiece with the largest number of remaining processes;

[0026] j)SNRO: Select the workpiece with the minimum number of remaining operations.

[0027] Furthermore, the reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution provided by the present invention further includes: for each topological unit, maintaining a buffer for storing workpiece processes that have been assigned to the corresponding topological unit but cannot be processed at present;

[0028] Furthermore, in step S2, when allocating the topological units required for the current process to the selected workpiece to be processed, a hierarchical strategy is adopted; the hierarchical strategy includes:

[0029] T1, taking the current process of the selected workpiece to be processed as the target processing object, determining the topological units that can be used to process the target processing object and the processing time of the target processing object on each topological unit according to the process information of the workpiece, and selecting the topological unit with the shortest processing time as the first target topological unit;

[0030] T2. If the equipment units in the equipment pool can be combined into the first target topological unit, the corresponding equipment units are allocated from the equipment pool to form the first target topological unit, and then allocated to the target processing object, and the allocation ends; otherwise, go to T3;

[0031] T3, if the equipment units in the equipment pool can be combined into other topological units for processing the target processing object, then the topological unit with the shortest processing time is selected as the second target topological unit, and the corresponding equipment units are allocated from the equipment pool to form the second target topological unit, and then allocated to the target processing object, and the allocation ends; otherwise, go to T4;

[0032] T4. Determine the topological units that are executing processing tasks and can be used to process the target processing object. The corresponding completion time of each topological unit after completing the processing tasks including the target processing object is determined. The topological unit with the earliest completion time is selected as the third target topological unit. The target processing object is added to the buffer of the third target topological unit, and the allocation is completed.

[0033] Furthermore, the tiered strategy also includes:

[0034] If the topological unit does not execute any processing task and its buffer is empty, the topological unit is disassembled into equipment units, and the disassembled equipment units are placed in the equipment pool.

[0035] Furthermore, the state characteristic information of the reconfigurable workshop includes: statistical characteristics oriented to workpieces and statistical characteristics oriented to topological units;

[0036] Artifact-oriented statistical features include:

[0037] (a) The average completion rate characteristics of all workpieces;

[0038] (b) The standard deviation characteristics of the completion rate of all workpieces;

[0039] (c) The average value characteristics of the remaining processing time of all workpieces;

[0040] (d) The standard deviation characteristics of the remaining processing time of all workpieces;

[0041] Statistical features for topological units include:

[0042] (e) Average characteristics of the utilization of all topological units;

[0043] (f) Standard deviation characteristics of the utilization of all topological units.

[0044] Furthermore, the reward function used to calculate the reward value is:

[0045] r(t)=(TWT(t)-TWT(t+1))+(UT(t)-UT(t-1))

[0046] Where t represents the current decision point, t-1 and t+1 represent the previous decision point and the next decision point of the current decision point respectively; r(t) represents the reward value of the current decision point, TWT(t) and TWT(t+1) represent the total delay of workpiece processing at the current decision point and the next decision point respectively, and UT(t) and UT(t-1) represent the topological resource utilization of the current decision point and the previous decision point respectively.

[0047] Furthermore, the neural network in the decision-making agent is a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multi-layer perceptron.

[0048] Furthermore, the reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution provided by the present invention also includes: constructing an evaluation agent based on the DQN algorithm; the neural network in the evaluation agent is used to evaluate the Q value calculated by the decision agent.

[0049] Furthermore, the reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution provided by the present invention further includes: initializing a sample pool;

[0050] Moreover, after each evaluation by the decision agent, the state feature information s, action a, reward value r of the current decision point and the state feature information s′ of the next decision point are added to the sample pool as samples [s, a, r, s′];

[0051] Moreover, after a fixed number of trainings, a preset number of samples are extracted from the sample pool in descending order of importance to train the neural network in the decision-making agent.

[0052] Furthermore, the decision agent and the evaluation agent are specifically constructed based on the dueling DQN algorithm.

[0053] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the above-mentioned reconfigurable workshop scheduling method based on reinforcement learning method and rule evolution provided by the present invention.

[0054] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned reconfigurable workshop scheduling method based on reinforcement learning method and rule evolution provided by the present invention.

[0055] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0056] (1) The present invention applies reinforcement learning methods to the scheduling problem of reconfigurable workshops, wherein the continuously trained decision-making agent can respond efficiently to new production situations, effectively improving the real-time performance of reconfigurable visible scheduling, and designs an action space based on composite scheduling rules based on genetic programming algorithms, wherein the composite scheduling rules are formed by calculating preset heuristic rules for selecting workpieces. Compared with directly using workpiece processes as the action space, the action space involved in the present invention is greatly reduced, which enables the decision-making agent to make decisions quickly while improving the optimization quality. In addition, the decision-making agent takes minimizing the total delay of workpiece processing as the optimization goal, which can ensure the overall quality of the production scheduling plan. In general, the present invention is based on reinforcement learning methods and rule evolution, which can effectively improve the real-time performance and optimization quality of reconfigurable workshop scheduling, thereby reducing the management cost of the enterprise and ensuring the stable operation of the production and manufacturing system.

[0057] (2) In the preferred embodiment of the present invention, ten single heuristic rules are designed. These rules are simple and closely related to the optimization goal of the decision-making agent, namely, minimizing the total delay of workpiece processing. On this basis, the genetic programming algorithm can be used to evolve composite rules that can adapt to more complex scenarios, further improving the optimization quality of the reconfigurable workshop scheduling plan.

[0058] (3) In the preferred embodiment of the present invention, when allocating topological units to the current process of the selected object to be processed, a hierarchical strategy of topological unit allocation and reorganization is adopted. Specifically, first, at the allocation layer, the workpiece is allocated to the topological unit with the shortest completion time; if the topological unit that can execute the processing task is occupied, the allocation plan is adjusted at the reorganization layer, and a processing topological unit with a shorter completion time is reselected, or the workpiece is added to the buffer pool of the topological unit that is executing the processing task. Based on this hierarchical strategy, it can be ensured that each process of the workpiece is completed as soon as possible, which is conducive to minimizing the total delay of the workpiece processing.

[0059] (4) In the preferred embodiment of the present invention, idle topological units, i.e., topological units that have not executed any processing tasks and whose buffers are empty, will be disassembled into equipment units and placed in an equipment pool, thereby improving the resource utilization of the reconfigurable workshop.

[0060] (5) In the preferred embodiment of the present invention, at each decision point, the state feature information of the reconfigurable workshop based on which the decision-making agent makes decisions is statistical features oriented towards the workpiece and statistical features oriented towards the topological unit, thereby being able to comprehensively consider the shortest delay and the highest machine utilization rate, and using statistical features to extract important information from the original state features, namely, the mean value feature and the standard deviation feature, thereby reducing the dimension of the state feature representation, extracting the key properties of the state feature, and improving the generalization ability of the agent model, while being able to adapt to the noise and unnecessary changes in the state feature space.

[0061] (6) In the optimization scheme of the present invention, the reward function used to calculate the reward value adopts a reward mechanism based on the difference in total estimated delays between adjacent decision points, that is, the greater the total delay between adjacent decision points, the smaller the reward value will be, thereby being able to train the intelligent agent to make decisions that are conducive to minimizing the total delay; the reward function also introduces the difference in utilization of topological units at adjacent decision points. In the early decision-making process, since the estimated total delay of the workpiece is zero, it is difficult for the decision-making intelligent agent to evaluate the value of the action taken, and thus it is impossible to effectively train and update the network of the decision-making intelligent agent. By introducing the difference in utilization of topological units at adjacent decision points, the problem of difficulty in early learning and training of the decision-making intelligent agent can be effectively solved.

[0062] (7) In the preferred embodiment of the present invention, the neural network in the decision-making agent is a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multi-layer perceptron. Under the design of this neural network, the recurrent neural network layer enables the deep reinforcement learning algorithm to optimize short-term benefits while taking into account long-term returns, which is crucial to improving the production efficiency and task completion rate of the entire workshop; and the multi-layer perceptron layer can further learn and map complex nonlinear relationships from the features extracted by the recurrent neural network layer, enhance the expression ability of the model, and thus better approximate the optimal strategy function; this network architecture provides the flexibility to handle complex scheduling problems. In the face of uncertainty and interference, the algorithm can also make robust decisions and optimize long-term production efficiency.

[0063] (8) In the preferred embodiment of the present invention, a decision agent based on the DQN algorithm is constructed, in which a neural network selects actions and updates them in each iteration. At the same time, an evaluation agent based on the DQN algorithm is constructed, in which a neural network is used to evaluate the Q value calculated by the decision agent and updates it at intervals. Thus, a double DQN mechanism is integrated into the DQN algorithm. This mechanism separates the calculation of the predicted value and the target value in the evaluation process, which can weaken the negative impact of the bad training set on the training and accelerate the convergence of the algorithm.

[0064] (9) In the preferred embodiment of the present invention, the experience replay mechanism is integrated into the DQN algorithm. Specifically, when the decision agent is evaluating, the state feature information s, action a, reward value r of the current decision point and the state feature information s of the next decision point are replayed. ′ As the sample [s, a, r, s′], it is added to the sample pool, and after a fixed number of trainings, a certain amount of samples are extracted from the sample pool in descending order of importance to train the samples in the decision-making agent. This can improve the learning ability of the neural network in the decision-making agent and make the scheduling method more efficient.

[0065] (10) In the preferred embodiment of the present invention, the decision agent and the evaluation agent are specifically constructed based on the dueling DQN algorithm, thereby integrating the dueling DQN mechanism into the DQN algorithm. Based on the dueling DQN mechanism, the output of the neural network will be vectorized. Specifically, the output of the neural network will be designed to be in the form of the sum of a state value function and an advantage function. The former is used to evaluate the state value of the input neural network, and the latter is used to evaluate each action that can be taken under this state feature. At the same time, the mean value of the advantage function is limited to 0, ensuring that the state value function is adaptively adjusted each time it is updated, so as to ensure that it can always play a role in the process of evaluating the state value of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of a reconfigurable workshop dynamic scheduling method based on a dual DQN algorithm with a confrontation mechanism and rule evolution provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a composite rule for workpiece selection based on a genetic programming algorithm provided by an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of an interaction strategy between an intelligent agent and a workshop environment provided by an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of an agent training network and an improved DQN algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0071] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0072] In order to improve the scientificity and real-time response capability of reconfigurable workshop production scheduling, this paper applies reinforcement learning methods to the scheduling problem of reconfigurable workshops, and proposes a reconfigurable workshop dynamic scheduling method based on reinforcement learning methods and rule evolution. The application of reinforcement learning methods can enable the continuously trained intelligent agents to respond efficiently to new production situations, so that they can select the most appropriate action strategy to ensure the maximization of cumulative benefits, thereby reducing the management costs of enterprises and ensuring the stable operation of the production and manufacturing system.

[0073] In order to effectively apply deep reinforcement learning methods to the scheduling problem of reconfigurable workshops, the present invention establishes a dynamic scheduling architecture for reconfigurable workshops, including a task pool, an equipment pool, a topological unit, a decision-making agent, and an interaction strategy between the agent and the workshop environment. Among them, the task pool stores all the workpieces to be processed in the production workshop at the current moment, and the workpieces to be processed represent unfinished workpieces that have not been scheduled for processing; the equipment pool stores idle equipment units in the reconfigurable workshop. It is easy to understand that the equipment unit is an auxiliary processing resource, including a console, a power supply, a water tank, etc.; the topological unit is formed by combining multiple equipment units in the reconfigurable workshop, and can complete the processing of a certain workpiece. The present invention maintains a buffer for each topological unit to store the workpieces to be processed that have been assigned to the topological unit. It is easy to understand that the topological unit is a machine used to complete the workpiece process processing, such as a reconfigurable machine tool, a multi-functional reconfigurable workstation, etc. There are multiple topological units to choose from for the processing of each workpiece and each process, and the time for different types of topological units to process the same process is not the same; the decision-making agent is used to decide the processing sequence of the workpieces to be processed, as well as the allocation and reorganization of the topological units. Its optimization goal is to minimize the total delay of the workpiece processing, and the present invention designs a high-quality action space based on a composite scheduling rule for it, wherein the composite scheduling rule is evolved from a single workpiece selection heuristic rule through a genetic programming algorithm, and can adapt to more complex scenarios.

[0074] In order to facilitate the decision-making agent to make decisions at the right time and complete the scheduling rules that are compatible with the current environmental state of the reconfigurable workshop, the present invention formulates a decision point based on the processing characteristics of the reconfigurable workshop, that is, the starting time, or the time when the topological unit completes the processing of the workpiece process and releases it and there are unfinished workpieces with no assigned processing tasks in the reconfigurable workshop.

[0075] In order to prevent dynamic disturbance events such as the arrival of new workpieces from interfering with the current decision-making process, when a new workpiece arrives, the present invention will give priority to determining whether the decision-making agent has entered the decision point. If so, the new workpiece information will be placed in the task pool and the state feature information of the workshop will be updated; if not, the new workpiece information will be placed in the processing environment and wait for the decision-making agent to enter the decision point. When the decision-making agent enters the decision point, the new workpiece information will be placed in the task pool from the processing environment and the state feature information of the workshop will be updated again.

[0076] In order to further optimize the scheduling results, in some preferred embodiments, the present invention further designs multiple statistical state characteristics of the reconfigurable workshop, while comprehensively considering the shortest delay and the highest machine utilization, reduces the dimension of state feature representation, extracts the key properties of the state characteristics, and improves the generalization ability of the intelligent model; proposes a hierarchical strategy of topological unit utilization allocation and reorganization to ensure that each process of the workpiece is completed as soon as possible, which is conducive to minimizing the total delay of workpiece processing; in line with the optimization goal, the present invention designs a reward function based on the difference in total delay costs of adjacent decision points.

[0077] In some preferred embodiments, this embodiment improves the DQN (Deep Q Network) algorithm on the basis of the algorithm. Specifically, Double DQN, Dueling DQN and a priority experience replay mechanism are incorporated, wherein DoubleDQN is to establish an additional evaluation network on top of the training network, Dueling DQN is to vectorize the output of the neural network, and the priority experience replay mechanism is to extract a certain number of samples from the experience pool for learning every fixed number of trainings based on the experience priority; based on the above improvements, the DQN algorithm used in the relevant embodiments is actually a double DQN algorithm with an adversarial mechanism. The adoption of the double DQN algorithm with an adversarial mechanism improves the learning ability of the model, is conducive to efficient decision-making based on learning in production scenarios, and outputs a production scheduling plan with better effect in a shorter time.

[0078] In order to further optimize the scheduling results, in a preferred embodiment, the neural network architecture in the decision-making agent is also optimized.

[0079] The following is a detailed explanation.

[0080] In one embodiment of the present invention, namely, embodiment 1, a reconfigurable workshop scheduling method based on reinforcement learning method and rule evolution is provided, such as Figure 1 As shown, in the initialization phase, this embodiment performs the following operations:

[0081] Genetic programming algorithm is used to evolve multiple composite scheduling rules based on the preset workpiece selection heuristic rules to form the action space of the decision-making agent;

[0082] Initialize the task pool and equipment pool, which are used to store all the workpieces to be processed and the idle equipment units in the reconfigurable workshop in real time;

[0083] A decision-making agent based on the DQN algorithm is constructed. The optimization goal of the decision-making agent is to minimize the total delay of workpiece processing. The action space is composed of multiple composite scheduling rules evolved from the preset heuristic rules for selecting workpieces through the genetic programming algorithm.

[0084] It is easy to understand that at the initial moment, the state characteristic information of the reconfigurable workshop is also initialized.

[0085] In this embodiment, the composite scheduling rule is represented by a binary tree structure, whose internal nodes (non-leaf nodes) are operation symbols used to represent the operation relationship between child nodes, and the leaf nodes are heuristic rules for selecting workpieces;

[0086] The operation symbols include: "+", "-", "×", "÷", "Max", and "Min". When the divisor is 0, the "÷" operation result returns the preset value; "Max" means taking the maximum value, and "Min" means taking the minimum value.

[0087] The heuristics for selecting artifacts include:

[0088] a) SPT (Job with the Shortest Processing Time): Select the job with the shortest processing time;

[0089] b)LPT (Job with The Longest Processing Time): Select the job with the longest processing time;

[0090] c) SSO (Job with the Shortest Processing Time of Successor Operation): Select the job with the shortest processing time of the subsequent operation;

[0091] d) LSO (Job with the Shortest Processing Time of Successor Operation): Select the job with the longest processing time of the subsequent operation;

[0092] e)SRPT (Job with The Shortest Remaining Processing Time): Select the job with the shortest remaining processing time;

[0093] f) LRPT (Job with The Longest Remaining Processing Time): Select the job with the longest remaining processing time;

[0094] g) EDD (Job with The Earliest Due Date): Select the job with the earliest due date;

[0095] h)FIFO (First In, First Out): First In, First Out rule;

[0096] i) LNRO (Job with the Largest Number of Remaining Operations): Select the job with the largest number of remaining operations;

[0097] j) SNRO (Job with The Smallest Number of Remaining Operations): Select the job with the smallest number of remaining operations.

[0098] Since the dimensions of attributes in different rules are different, the heuristic rules for selecting artifacts cannot be directly and simply combined. It is necessary to normalize the artifact attributes and determine the priority of the artifact under each heuristic rule. Taking rule a) as an example, the priority weight of artifact i in the task pool is:

[0099]

[0100] Among them, pt ij It represents the average processing time of process j for workpiece i.

[0101] In this embodiment, the operators of the composite scheduling rules are managed as functions through a function set, and the single artifact selection heuristic rule is managed as a terminator through a terminator set. Optionally, in this embodiment, the binary tree used to represent the composite scheduling rules is generated using a hybrid method, 50% of the individuals are generated by the complete method, and 50% of the individuals are generated by the growing method. In the complete method, all terminators in the generated individuals are located at the maximum depth of the tree; in the growing method, all nodes of the tree can be arbitrarily selected from the operator set and the heuristic rule set.

[0102] The algorithm flow of the genetic programming algorithm is as follows Figure 2 As shown in the figure. The fitness function in the genetic programming algorithm is formulated according to the total delay, and the fitness function value is negatively correlated with the performance of the individual. The genetic programming algorithm receives the fitness value, updates the population, and outputs a better rule. Compared with the scheduling rules calculated manually or summarized by experience, the scheduling rules generated by genetic programming can obtain more scientific results in a shorter time, which is conducive to adapting to the production environment with frequent state switching.

[0103] Figure 2 The composite scheduling rule example shown can be expressed as:

[0104] (LPT*SSO)+SPT

[0105] Based on the composite scheduling rule, the priority of each workpiece in the task pool is calculated, the workpieces are sorted according to the priority, and the topological units are allocated to the workpieces in order.

[0106] In this embodiment, the state feature information of the reconfigurable workshop is multiple statistical state feature information, including workpiece-oriented statistical features and topological unit-oriented statistical features;

[0107] Artifact-oriented statistical features include:

[0108] (a) The average completion rate characteristics of all workpieces;

[0109] (b) The standard deviation characteristic of the completion rate of all workpieces;

[0110] (c) The average value characteristics of the remaining processing time of all workpieces;

[0111] (d) The standard deviation characteristics of the remaining processing time of all workpieces;

[0112] Statistical features for topological units include:

[0113] (e) Average characteristics of the utilization of all topological units;

[0114] (f) Standard deviation characteristics of the utilization of all topological units.

[0115] The above-mentioned state feature information is a real-time feature, which will change in real time with the processing of the workpiece and the arrival of new workpieces. It comprehensively considers the shortest delay and the highest machine utilization rate, uses statistical features to extract important information from the original state, reduces the dimension of state feature representation, extracts the key properties of state features, improves the generalization ability of the intelligent model, and can adapt to the noise and unnecessary changes in the state feature space.

[0116] In this embodiment, the optimization goal of the decision-making agent is to minimize the total delay of workpiece processing. At any time, the total delay of the workpiece can be expressed as:

[0117]

[0118] Where n represents the number of existing workpieces in the production workshop, T 总 The total delay of all workpieces, ET i represents the completion time of the last process of workpiece i, DT i represents the delivery date of job i.

[0119] Based on the above calculation method, when the completion time is earlier than the delivery date, the delay is 0 instead of a negative value. This design can prevent the intelligent agent from pursuing premature delivery of individual processes, which is not conducive to shortening the total delay.

[0120] After initialization, the decision agent can interact with the environment of the reconfigurable workshop to determine the start and end processing time of each workpiece and each process, as well as the corresponding topological units. Figure 1 and Figure 3 As shown in Figure 2, the interaction strategies between the decision-making agent and the reconfigurable workshop include:

[0121] For the topological units that are processing workpieces, if there is a topological unit that completes the processing of the workpiece process at this moment and puts the released workpiece back into the task pool, the decision agent enters the decision point at this moment. After entering the decision point, the decision agent makes decisions according to the following steps to realize the dynamic scheduling of the decision point:

[0122] S1: Feedback the current state feature information of the reconfigurable workshop and the reward value after executing the action obtained in the previous decision to the decision agent, train the neural network in the decision agent and output the probability of each composite scheduling rule being selected in the action space by the neural network; the smaller the total delay of workpiece processing, the greater the reward value;

[0123] S2: Select the composite scheduling rule with the highest probability as the action obtained by the decision-making agent at the current decision point, select the workpiece to be processed from the task pool according to the selected composite scheduling rule, and allocate the topological unit required for the current process to it.

[0124] In step S1 of this embodiment, a reward function based on the difference in total estimated delay costs between adjacent decision points is specifically designed to calculate the reward value. At the current decision point t, the total estimated delay of the workpiece is:

[0125]

[0126] Where m represents the number of available topological units; at the current decision point t, CT k (t) represents the earliest completion time of the topological unit numbered k, ET i (t) represents the completion time of the last scheduled process of job i;

[0127] Based on the above calculation expression of the estimated total delay of the workpiece, the reward function of the agent at the current decision point t can be expressed as:

[0128] r(t)=(TWT(t)-TWT(t+1))+(UT(t)-UT(t-1))

[0129] Among them, r(t) represents the reward value of the current decision point, TWT(t) and TWT(t+1) represent the total delay of workpiece processing at the current decision point and the next decision point, respectively, and UT(t) and UT(t-1) represent the topological resource utilization of the current decision point and the previous decision point, respectively.

[0130] Since the optimization goal of the decision-making agent is to minimize the total delay of the workpiece, the setting of this reward function can set a gradient of the estimated delay to ensure that the objective function value can be continuously reduced during the training process; in the early decision-making process, since the estimated total delay of the workpiece is zero, it is difficult for the agent to evaluate the value of the action taken, and thus it is impossible to train and update the network of the agent; this embodiment can effectively solve the problem of the difficulty of early learning and training of the decision-making agent by introducing the difference in the utilization rate of the topological unit at adjacent decision points in the reward function.

[0131] In this embodiment, the neural network of the decision-making agent is specifically designed as a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multi-layer perceptron. The recurrent neural network is used for training and learning to obtain feature parameters, and the last layer of the network is fully connected; the multi-layer perceptron is applied to the fully connected layer of the recurrent neural network, and the feature parameters output by the recurrent neural network are finally classified and calculated to obtain the action value function value predicted for each sample.

[0132] Under this neural network design method, the recurrent neural network layer enables the deep reinforcement learning algorithm to optimize short-term benefits while taking into account long-term returns, which is crucial to improving the production efficiency and task completion rate of the entire workshop; and the multi-layer perceptron layer can further learn and map complex nonlinear relationships from the features extracted by the recurrent neural network layer, enhance the expressive power of the model, and thus better approximate the optimal strategy function; this combined network architecture provides the flexibility to handle complex scheduling problems; in the face of uncertainty and interference, the algorithm can also make robust decisions and optimize long-term production efficiency.

[0133] In step S2 of this embodiment, when allocating the topological units required for the current process to the selected workpiece to be processed, a hierarchical strategy is adopted; the hierarchical strategy includes:

[0134] T1, taking the current process of the selected workpiece to be processed as the target processing object, determining the topological units that can be used to process the target processing object and the processing time of the target processing object on each topological unit according to the process information of the workpiece, and selecting the topological unit with the shortest processing time as the first target topological unit; the specific expression is as follows:

[0135]

[0136] Among them, M s The topological unit representation of the allocation, t ijk is the jth process of workpiece i in topological unit M k The processing time on M ij is the set of optional topological units for the j-th process of workpiece i;

[0137] T2. If the equipment units in the equipment pool can be combined into the first target topological unit, the corresponding equipment units are allocated from the equipment pool to form the first target topological unit, and then allocated to the target processing object, and the allocation ends; otherwise, go to T3;

[0138] T3, if the equipment units in the equipment pool can be combined into other topological units for processing the target processing object, then the topological unit with the shortest processing time is selected as the second target topological unit, and the corresponding equipment units are allocated from the equipment pool to form the second target topological unit, and then allocated to the target processing object, and the allocation ends; otherwise, go to T4;

[0139] T4, determine the completion time corresponding to each topological unit after completing the processing task including the target processing object among the topological units that are executing the processing task and can be used to process the target processing object, select the topological unit with the earliest completion time as the third target topological unit, add the target processing object to the buffer of the third target topological unit, and the allocation ends; the specific expression is as follows:

[0140]

[0141] Among them, t′ i′j′k is the j′th process of workpiece i′ in topological unit M k The remaining processing time on i″j″k It is the topological unit M k The processing time of the jth operation of workpiece i″ in the buffer;

[0142] If the buffer of a topological unit is empty and there is no workpiece being processed on the topological unit at this moment, the topological unit is in an idle state; the idle topological unit can be disassembled into equipment units and put back into the equipment pool for subsequent topological unit reconstruction to improve the utilization efficiency of reconfigurable workshop resources.

[0143] In the above-mentioned layering strategy, T1~T2 constitute the allocation layer, and T3~T4 constitute the reorganization layer. Such topological unit allocation rules can enable the workpiece process to be completed as soon as possible, improve the processing efficiency of the reconfigurable workshop, and shorten the total delay.

[0144] By making decisions and scheduling in the above manner, after all the workpieces to be processed are completed, the production schedule of the reconfigurable workshop can be output, including the processing start time, processing end time and the allocated topological unit of each process of each workpiece.

[0145] like Figure 4 As shown, in this embodiment, the improvement of the DQN algorithm is specifically as follows:

[0146] (1) Integrate the double DQN mechanism to solve the overestimation problem in DQN. The specific implementation method is to introduce two neural networks, one network is used for training, selects actions and updates in each iteration, and the other network is used for evaluation, which is updated at intervals. This mechanism separates the calculation of the predicted value and the target value in the evaluation process, which can weaken the negative impact of the bad training set on the training and accelerate the convergence of the algorithm. The mathematical expression of the Q value calculation under this mechanism is as follows:

[0147] Q(s,a)=Q(s,a)+α(r+yQ target (s′, a)-Q(s, a))

[0148] Where Q(s, a) represents the predicted Q value calculated by the training network; Q target (s′, a) represents the target Q value calculated by the neural network of the evaluation agent; s′ represents the state characteristics at the next moment, and the state characteristics in the target network are used to calculate the Q value during evaluation; α and γ represent discount coefficients; r represents the reward value.

[0149] (2) Integrate the dueling DQN mechanism and design the output of the neural network as the sum of the state value function and the advantage function. The former is used to evaluate the state feature value of the input neural network, and the latter is used to evaluate each action that can be taken under this state feature. At the same time, the mean value of the advantage function is limited to 0, ensuring that the state value function will be adaptively adjusted at each update to ensure that it can always play a role in the process of evaluating the state value. This structural design can make DuelingDQN more efficient when processing state features that have little effect on actions. The mathematical expression for Q value calculation under this mechanism is as follows:

[0150]

[0151] Among them, V(s; θ, β) is the state value function, which outputs a scalar whose value is only related to the state feature value s and the training parameters θ and β; A(s, a; θ, δ) is the advantage function, which outputs a vector whose length is equal to the size of the action space. The value is related to the state feature value s and the action value a as well as the parameters θ and δ; θ refers to the parameters of the network convolution layer; δ and β are the parameters of the fully connected layers of the two branches respectively; Make sure the advantage function has a mean of 0.

[0152] (3) Fusion priority experience replay mechanism. Specifically, the sample pool is initialized at the initial moment. After each evaluation by the decision agent, the state feature information s, action a, reward value r of the current decision point and the state feature information s′ of the next decision point are taken as samples [s, a, r, s′] and added to the sample pool. After a fixed number of trainings, a preset number of samples are extracted from the sample pool in descending order of importance to train the neural network in the decision agent. For the sample [s, a, r, s′], the difference between its current Q value and target Q value in the time series difference is calculated as its importance. The calculation expression is as follows:

[0153]

[0154] To make training | TD error The expectation of | is as small as possible, and a priority queue is maintained using a greedy mechanism to update |TD first. error |Large sample.

[0155] The improved DQN algorithm proposed in this embodiment considers the separation of training and evaluation networks and the hierarchical calculation of Q values ​​on the basis of DQN, and improves the experience replay mechanism of randomly selected samples, so that the algorithm has higher efficiency.

[0156] In general, this embodiment provides a reconfigurable workshop agent system with a dynamic disturbance event response mechanism. With the goal of minimizing the total delay of workpieces, multiple real-time features combining workpiece processing information and workshop resource information are designed. The agent makes decisions based on the real-time state feature information of the workshop, thereby realizing adaptive scheduling when dynamic disturbance events occur in resource collaborative assembly workshops, and improving the production efficiency of manufacturing workshops. A composite rule for workpiece selection based on genetic programming evolution is provided. In a composite rule, multiple factors affecting decision-making are comprehensively considered, and the composite rule is optimized based on the relationship between factors, so that the decision-making process makes reference to different degrees according to the importance of each factor, greatly improving the ability of the model to search for the optimal solution. The designed training network uses a recurrent neural network combined with a multi-layer perceptron model, which can more fully consider the influence of time series on the basis of allocating the probability of action selection, improve the agent's targeted processing ability for continuous processes, and enhance the adaptability and robustness of the model. The designed reward mechanism based on the difference in total estimated delays between adjacent decision points trains the agent to make decisions that are conducive to reducing total delays. The cumulative reward obtained is the negative value after the estimated delay gradient is summed. The training goal is to minimize the total delay of the workpiece and obtain a better solution by maximizing the cumulative reward. The provided deep reinforcement learning algorithm considers three improvements on the basis of the DQN algorithm, adds the processing modes of "separation of training and evaluation" and "vectorized representation of Q value" to the evaluation network of the neural network, and optimizes the experience playback mechanism, which improves the learning ability of the model, is conducive to efficient decision-making based on learning in production scenarios, and outputs a better scheduling plan in a shorter time.

[0157] Embodiment 2:

[0158] A computer program product includes a computer program. When the computer program is executed by a processor, the reconfigurable workshop scheduling method based on reinforcement learning method and rule evolution provided in the above-mentioned embodiment 1 is implemented.

[0159] Embodiment 3:

[0160] A computer-readable storage medium includes a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the reconfigurable workshop scheduling method based on reinforcement learning method and rule evolution provided in the above-mentioned embodiment 1.

[0161] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution, characterized in that: include: Initialize a task pool and an equipment pool, which are used to store all workpieces to be processed and idle equipment units in the reconfigurable workshop in real time, and build a decision-making intelligent agent based on the DQN algorithm; The optimization goal of the decision-making agent is to minimize the total delay of workpiece processing, and the action space is composed of multiple composite scheduling rules evolved from the preset workpiece selection heuristic rules through the genetic programming algorithm; At each decision point, dynamic scheduling is performed according to the following steps: S1: Feedback the current state feature information of the reconfigurable workshop and the reward value after executing the action obtained in the previous decision to the decision agent, train the neural network in the decision agent and output the probability of each composite scheduling rule being selected in the action space by the neural network; the smaller the total delay of workpiece processing, the greater the reward value; S2: Selecting the composite scheduling rule with the highest probability as the action obtained by the decision agent at the current decision point, and selecting the workpiece to be processed from the task pool according to the selected composite scheduling rule, and allocating the topological unit required for the current process to it; After all the workpieces to be processed are processed, the production schedule of the reconfigurable workshop is output; the production schedule includes the processing start time, processing end time and the allocated topological unit of each process of each workpiece; The decision point is the start time, or the time when the topological unit releases the workpiece after completing the processing of the workpiece process and there are unfinished workpieces with no processing tasks assigned in the reconfigurable workshop.

2. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 1, characterized in that: The composite scheduling rule is represented by a binary tree structure; The internal nodes of the binary tree are operation symbols, which are used to represent the operation relationship between child nodes; The leaf nodes of the binary tree are heuristic rules for selecting workpieces; The operation symbols include: "+", "-", "×", "÷", "Max", and "Min". When the divisor is 0, the "÷" operation result returns the preset value. "Max" means taking the maximum value, and "Min" means taking the minimum value. The heuristic rules for selecting artifacts include: a) SPT: Select the workpiece with the shortest processing time; b) LPT: Select the workpiece with the longest processing time; c) SSO: Select the workpiece with the shortest processing time in the subsequent process; d) LSO: select the workpiece with the longest processing time in the subsequent process; e) SRPT: select the workpiece with the shortest remaining processing time; f) LRPT: select the workpiece with the longest remaining processing time; g) EDD: Select the workpiece with the earliest delivery date; h)FIFO: First-in-first-out rule; i) LNRO: select the workpiece with the largest number of remaining processes; j)SNRO: Select the workpiece with the minimum number of remaining operations.

3. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 1, characterized in that: For each topological unit, a buffer is maintained to store the workpiece processes that have been assigned to the corresponding topological unit but cannot be processed at present; Furthermore, in the step S2, when allocating the topological units required for the current process to the selected workpiece to be processed, a hierarchical strategy is adopted; The tiered strategy includes: T1, taking the current process of the selected workpiece to be processed as the target processing object, determining the topological units that can be used to process the target processing object and the processing time of the target processing object on each topological unit according to the process information of the workpiece, and selecting the topological unit with the shortest processing time as the first target topological unit; T2, if the equipment units in the equipment pool can be combined into the first target topological unit, then allocate the corresponding equipment units from the equipment pool to form the first target topological unit, and then allocate them to the target processing object, and the allocation ends; otherwise, go to T3; T3, if the equipment units in the equipment pool can be combined into other topological units for processing the target processing object, then select the topological unit with the shortest processing time as the second target topological unit, and allocate the corresponding equipment units from the equipment pool to form the second target topological unit, and then allocate it to the target processing object, and the allocation ends; otherwise, go to T4; T4. Determine the topological units that are executing processing tasks and can be used to process the target processing object, the completion time corresponding to each topological unit after completing the processing tasks including the target processing object, select the topological unit with the earliest completion time as the third target topological unit, add the target processing object to the buffer of the third target topological unit, and the allocation ends.

4. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 3, characterized in that: The tiered strategy also includes: If the topological unit does not execute a processing task and its buffer is empty, the topological unit is disassembled into equipment units, and the disassembled equipment units are placed into the equipment pool.

5. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 1, characterized in that: The state characteristic information of the reconfigurable workshop includes: statistical characteristics oriented to workpieces and statistical characteristics oriented to topological units; The artifact-oriented statistical features include: (a) The average completion rate characteristics of all workpieces; (b) The standard deviation characteristics of the completion rate of all workpieces; (c) The average value characteristics of the remaining processing time of all workpieces; (d) The standard deviation characteristics of the remaining processing time of all workpieces; The statistical features of the topological unit include: (e) Average characteristics of the utilization of all topological units; (f) Standard deviation characteristics of the utilization of all topological units.

6. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 1, characterized in that: The reward function used to calculate the reward value is: r(t)=(TWT(t)-TWT(t+1))+(UT(t)-UT(t-1)) Where t represents the current decision point, t-1 and t+1 represent the previous decision point and the next decision point of the current decision point respectively; r(t) represents the reward value of the current decision point, TWT(t) and TWT(t+1) represent the total delay of workpiece processing at the current decision point and the next decision point respectively, and UT(t) and UT(t-1) represent the topological resource utilization of the current decision point and the previous decision point respectively.

7. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 1, characterized in that: The neural network in the decision-making agent is a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multi-layer perceptron.

8. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution according to any one of claims 1 to 7, characterized in that: Also includes: An evaluation agent based on the DQN algorithm is constructed; the neural network in the evaluation agent is used to evaluate the Q value calculated by the decision agent.

9. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 8, characterized in that: Also includes: Initialize the sample pool; Moreover, each time the decision agent makes an evaluation, the state feature information s, action a, reward value r of the current decision point and the state feature information s′ of the next decision point are added to the sample pool as samples [s, a, r, s′]; Furthermore, after each fixed number of trainings, a preset number of samples are extracted from the sample pool in descending order of importance to train the neural network in the decision-making agent.

10. The reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution as claimed in claim 9, characterized in that: The decision-making agent and the evaluation agent are specifically constructed based on the dueling DQN algorithm.

Citation Information

Patent Citations

  • Cluster resource scheduling method and system based on multi-agent deep reinforcement learning

    CN111694656A

  • Unrelated parallel machine dynamic hybrid flow shop scheduling method based on a deep Q network

    CN113406939A

  • Low-orbit satellite hopping beam optimization method based on migration deep reinforcement learning

    CN114362810A

  • Prefabricated part production scheduling optimization method and system based on reinforcement learning

    CN115204497A

  • Smart factory-oriented real-time scheduling method based on multi-agent architecture

    CN115759552A

Cited By

  • Additive manufacturing workshop scheduling method based on safety reinforcement learning

    CN119849842A