A reconfigurable workshop dynamic scheduling method based on reinforcement learning method and rule evolution

Abstract

This invention discloses a reconfigurable shop floor dynamic scheduling method based on reinforcement learning and rule evolution, belonging to the field of shop floor scheduling. It includes: using a genetic programming algorithm to select and combine heuristic scheduling rules with the goal of minimizing total delay, resulting in composite scheduling rules guiding the selection of workpieces; constructing a hierarchical strategy for efficiency-first topological unit allocation and reorganization; integrating Double DQN, Dueling DQN, and a priority experience replay mechanism to improve the optimization capability of the DQN algorithm; and building an intelligent agent neural network model integrating recurrent neural networks and multilayer perceptrons. During training, the temporal information of shop floor processing task allocation is extracted, and a dynamic event response mechanism for the arrival of new workpieces is established. This invention can improve the real-time performance and optimization quality of reconfigurable shop floor scheduling, increase manufacturing resource utilization, thereby reducing enterprise production costs and ensuring the stable operation of the manufacturing system.

Description

Technical Field

[0001] This invention belongs to the field of shop floor scheduling, and more specifically, relates to a reconfigurable shop floor dynamic scheduling method based on reinforcement learning and rule evolution. Background Technology

[0002] With the rapid development of global manufacturing and the ever-evolving trends in technology, the importance of intelligent manufacturing is becoming increasingly prominent. As a product of the deep integration of next-generation information technology and manufacturing, intelligent manufacturing's core lies in achieving automation, informatization, networking, and flexibility in the manufacturing process through intelligent means. Among the many areas of intelligent manufacturing, shop floor scheduling has consistently attracted significant attention from both academia and industry. As a crucial means of optimizing production processes and improving efficiency, shop floor scheduling not only enhances production efficiency and reduces production costs but also effectively addresses rapid changes in market demand, promoting sustainable development for enterprises. Its research value and application prospects are increasingly evident.

[0003] In actual manufacturing processes, reconfigurable workshops represent a novel and efficient production model capable of meeting diverse manufacturing needs and are widely used in fields such as aviation, aerospace, and marine engineering. Reconfigurable workshops enable efficient reorganization of workshop equipment resources, thereby improving the utilization rate of manufacturing resources. Compared to traditional workshops, the scheduling of reconfigurable workshops presents a more complex problem space. In addition to considering the flexibility of the processing flow, reconfigurable workshops also require the development of equipment resource reorganization strategies. The introduction of reconfiguration problems, while improving the production capacity of the manufacturing workshop, also brings greater uncertainty to scheduling. Therefore, research on the scheduling problem of reconfigurable workshops is of great significance.

[0004] For reconfigurable shop floor scheduling, the final step is to formulate a corresponding production scheduling plan, which involves determining the start and end times of each process for each workpiece, as well as the allocated resources. Traditional reconfigurable shop floor scheduling methods are mostly static scheduling methods, which directly determine the production scheduling plan based on the process information of the workpieces to be processed and the resource information within the reconfigurable shop floor. However, in actual reconfigurable shop floor production, dynamic events such as the arrival of new workpieces occur frequently, rendering the original production scheduling plan infeasible. After a dynamic event occurs, traditional static scheduling methods require additional time to resolve the scheduling plan, which reduces the efficiency of the manufacturing process and fails to guarantee real-time performance and optimization quality. Summary of the Invention

[0005] To address the shortcomings and improvement needs of existing technologies, this invention provides a dynamic scheduling method for reconfigurable workshops based on reinforcement learning and rule evolution. The aim is to improve the real-time performance and optimization quality of reconfigurable workshop scheduling, thereby reducing enterprise management costs and ensuring the stable operation of the production and manufacturing system.

[0006] To achieve the above objectives, according to one aspect of the present invention, a dynamic scheduling method for a reconfigurable workshop based on reinforcement learning and rule evolution is provided, comprising: initializing a task pool and a device pool, which are used to store all workpieces to be processed and idle devices in the reconfigurable workshop in real time, respectively, and constructing a decision agent based on the DQN algorithm; the optimization objective of the decision agent is to minimize the total delay, and the action space is composed of multiple composite scheduling rules obtained by the evolution of a preset heuristic rule for selecting workpieces through a genetic programming algorithm; At each decision point, dynamic scheduling is performed according to the following steps: S1: Feed back the current state characteristics of the reconfigurable workshop and the reward value after executing the action obtained from the previous decision to the decision agent, train the neural network in the decision agent, and output the probability of each composite scheduling rule in the action space being selected by the neural network; the smaller the estimated total delay, the larger the reward value. S2: Select the composite scheduling rule with the highest probability as the action of the decision agent at the current decision point, and select the workpiece to be processed from the task pool according to the selected composite scheduling rule, and allocate the topological unit required for the current process to it; where the task pool is responsible for storing the workpieces to be processed in the workshop, and the workpiece to be processed refers to the unfinished workpiece that is waiting to be processed. After all workpieces to be processed are completed, the production schedule of the reconfigurable workshop is output; the production schedule includes the start time, end time and assigned topology unit for each process of each workpiece. The decision point includes two scenarios: one is the initial moment, and the other is the moment when the task pool is not empty and a certain topological unit has just completed its process. A topology unit is a processing unit formed by combining several devices. One combination of devices corresponds to one type of topology unit, which can process a specified type of operation. If a topology unit is idle at a certain time, it will be dismantled and all its devices will be added to the device pool to achieve flexibility and agility in the production process.

[0007] For each topology unit, maintain a buffer to store workpiece operations that have been assigned to the corresponding topology unit but cannot be processed at present; Furthermore, in step S2, when allocating the topology units required for the current process to the selected workpiece to be processed, a hierarchical strategy is adopted; including: T1. Take the current process of the selected workpiece to be processed as the target processing object. Based on the process information of the workpiece, determine the topological unit that can be used to process the target processing object, and the processing time of the target processing object on each topological unit. Select the topological unit with the shortest processing time as the first target topological unit. T2. If the devices in the device pool can be combined to form the first target topology unit, then the corresponding devices are allocated from the device pool to form the first target topology unit and then allocated to the target processing object, and the allocation ends; otherwise, proceed to T3. T3. If the devices in the device pool can be combined into other topological units for processing the target processing object, then the topological unit with the shortest processing time is selected as the second target topological unit. After the corresponding devices are allocated from the device pool to form the second target topological unit, it is allocated to the target processing object, and the allocation ends; otherwise, proceed to T4. T4. Determine all topological units that are currently executing processing tasks and can be used to process the target processing object. Then calculate the completion time of each topological unit for completing the processing task, including the target processing object. Finally, select the topological unit with the smallest completion time as the third target topological unit, add the target processing object to the buffer of the third target topological unit, and the allocation ends. The layered strategy also includes: If a topology unit does not perform a processing task and its buffer is empty, the topology unit is disassembled into a device and the disassembled device is placed into the device pool. The reward function used to calculate the reward value is:

[0008] in, t Indicates the current decision point. t -1 and t +1 represents the previous and next decision points of the current decision point, respectively; This represents the reward value at the current decision point. and These represent the estimated total delays at the current decision point and the next decision point, respectively. and These represent the topology unit utilization rates of the current decision point and the previous decision point, respectively.

[0009] Furthermore, the composite scheduling rules are represented using a binary tree structure; The function set of a binary tree is a series of operators used to represent various operational relationships; The terminal set of the binary tree serves as the heuristic rule for selecting the workpiece; The operators include: "+", "-", "×", "÷", "Max", and "Min"; when the divisor is 0, the result of the "÷" operation returns a preset value; "Max" indicates the maximum value, and "Min" indicates the minimum value. The heuristic rules for selecting workpieces include: a) SPT: Select the workpiece with the shortest processing time; b) LPT: Select the workpiece with the longest processing time; c) SSO: Select the workpiece with the shortest processing time in subsequent processes; d) LSO: Select the workpiece with the longest processing time in the subsequent process; e) SRPT: Select the workpiece with the shortest remaining processing time; f) LRPT: Select the workpiece with the longest remaining processing time; g) EDD: Select the workpiece with the earliest delivery date; h) FIFO: First-In-First-Out rule; i) LNRO: Select the workpiece with the largest number of remaining operations; j) SNRO: Select the workpiece with the fewest remaining operations.

[0010] Furthermore, the state characteristic information of the reconfigurable workshop includes: statistical characteristics oriented towards workpieces and statistical characteristics oriented towards topological units; Statistical characteristics oriented towards the workpiece include: (a) The average completion rate of all workpieces; (b) The standard deviation of the completion rate for all workpieces; (c) The average remaining processing time for all workpieces; (d) The standard deviation of the remaining processing time for all workpieces; Statistical characteristics oriented towards topological units include: (e) The average utilization rate of all topological units; (f) Standard deviation of utilization of all topological units.

[0011] Furthermore, the neural network in the decision-making agent is a recurrent neural network, and the fully connected layers of the recurrent neural network are implemented by a multilayer perceptron.

[0012] Furthermore, the reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution provided by the present invention also includes: constructing an evaluation agent based on the DQN algorithm; the neural network in the evaluation agent is used to evaluate the Q value calculated by the decision agent.

[0013] Furthermore, the reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution provided by the present invention also includes: initializing the sample pool; Furthermore, after each evaluation by the decision-making agent, the state characteristic information of the current decision point is recorded. ,action Reward Value and the state characteristic information of the next decision point As a sample Add to the sample pool; Furthermore, after a fixed number of training sessions, a predetermined number of samples are drawn from the sample pool in descending order of importance to train the neural network in the decision-making agent.

[0014] Furthermore, the algorithm for constructing the decision agent and the evaluation agent is the dueling DQN algorithm.

[0015] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described reconfigurable shop scheduling method based on reinforcement learning and rule evolution provided by the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program, which, when executed by a processor, controls the device in which the computer-readable storage medium is located to execute the above-described reconfigurable shop floor scheduling method based on reinforcement learning and rule evolution provided by the present invention.

[0017] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention applies reinforcement learning to the scheduling problem of reconfigurable workshops. The continuously trained decision agent can respond efficiently to new production situations, effectively improving the real-time performance of reconfigurable scheduling. Furthermore, based on the genetic programming algorithm, an action space based on composite scheduling rules is designed. The composite scheduling rules are formed by computation of preset heuristic rules for selecting workpieces. Compared to directly using workpiece processes as the action space, the action space designed in this invention is greatly reduced, making it easier for the decision agent to learn reasonable "state-action" mapping relationships and improving optimization quality. In addition, the decision agent takes minimizing the total delay as the optimization objective, which can ensure the overall quality of the production scheduling plan. In summary, this invention, based on reinforcement learning and rule evolution, can effectively improve the real-time performance and optimization quality of reconfigurable workshop scheduling, thereby reducing enterprise management costs and ensuring the stable operation of the production and manufacturing system.

[0018] (2) In the preferred embodiment of the present invention, ten single heuristic rules are designed. These rules are simple and closely related to the optimization objective of the decision agent, namely minimizing the total delay. Based on this, the genetic programming algorithm can be used to evolve composite rules that can adapt to more complex scenarios, thereby further improving the optimization quality of the reconfigurable workshop scheduling scheme.

[0019] (3) In the preferred embodiment of the present invention, when allocating topology units for the current process of the selected workpiece, a hierarchical strategy of topology unit allocation and reorganization is adopted. Specifically, firstly, in the allocation layer, the workpiece is allocated to the topology unit with the shortest completion time; if the topology unit that can perform the processing task is occupied, the allocation scheme is adjusted in the reorganization layer, and a topology unit with a shorter completion time is selected again, or the workpiece is added to the buffer of the topology unit that is performing the processing task. Based on this hierarchical strategy, it can be ensured that each process of the workpiece is completed as early as possible, which is conducive to minimizing the total delay.

[0020] (4) In a preferred embodiment of the present invention, idle topology units, i.e. topology units that have not performed processing tasks and whose buffers are empty, are disassembled into devices and placed into the device pool, thereby improving the resource utilization of the reconfigurable workshop.

[0021] (5) In a preferred embodiment of the present invention, at each decision point, when the decision-making agent makes a decision, the state feature information of the reconfigurable workshop on which it relies includes: statistical features oriented towards workpieces and statistical features oriented towards topological units. This allows for a comprehensive consideration of minimizing delays and maximizing machine utilization. Statistical features are used to extract important information from the original state features, namely the mean and standard deviation, reducing the dimensionality of the state feature representation, extracting the key properties of the state features, improving the generalization ability of the agent model, and simultaneously adapting to noise and unnecessary changes in the state feature space.

[0022] (6) In the preferred embodiment of the present invention, the reward function used to calculate the reward value adopts the reward mechanism of the difference in total estimated delay between adjacent decision points. That is, the larger the estimated total delay between adjacent decision points, the smaller the reward value will be, thereby enabling the agent to make a decision that is conducive to minimizing the total delay. The reward function also introduces the difference in the utilization rate of topology units at adjacent decision points. In the early decision-making process, since the estimated total delay of the workpiece is zero, the decision agent has difficulty evaluating the value of the actions taken, and thus cannot effectively train and update the network of the decision agent. By introducing the difference in the utilization rate of topology units at adjacent decision points, the problem of the difficulty in early learning and training of the decision agent can be effectively solved.

[0023] (7) In a preferred embodiment of the present invention, the neural network in the decision agent is a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multilayer perceptron. Under this design of the neural network, the recurrent neural network layer enables the deep reinforcement learning algorithm to optimize short-term benefits while taking into account long-term returns, which is crucial for improving the overall production efficiency and task completion rate of the workshop. The multilayer perceptron layer can further learn and map complex nonlinear relationships from the features extracted by the recurrent neural network layer, enhance the expressive power of the model, and thus better approximate the optimal policy function. This network architecture has the flexibility to handle complex scheduling problems, enabling the algorithm to make robust decisions in the face of uncertainty and interference, and improve long-term production efficiency.

[0024] (8) In a preferred embodiment of the present invention, a decision agent based on the DQN algorithm is constructed, wherein the neural network selects and updates an action in each iteration. At the same time, an evaluation agent based on the DQN algorithm is constructed, wherein the neural network is used to evaluate the Q value calculated by the decision agent and updates it at intervals. Thus, the doubleDQN mechanism is integrated into the DQN algorithm. This mechanism separates the calculation of the predicted value and the target value in the evaluation process, which can weaken the negative impact of poor training set on training and accelerate the convergence of the algorithm.

[0025] (9) In a preferred embodiment of the present invention, an experience replay mechanism is incorporated into the DQN algorithm. Specifically, when the decision agent performs evaluation, the state feature information of the current decision point is used. ,action Reward Value and the state characteristic information of the next decision point As a sample The samples are added to the sample pool, and after a fixed number of training sessions, a certain number of samples are drawn from the sample pool in descending order of importance to train the neural network in the decision agent. This can improve the learning ability of the neural network in the decision agent and make the scheduling method more efficient.

[0026] (10) In a preferred embodiment of the present invention, the construction algorithm of the decision agent and the evaluation agent is the duelingDQN algorithm, thereby integrating the duelingDQN mechanism into the DQN algorithm. Based on the duelingDQN mechanism, the output of the neural network will be vectorized. Specifically, the output of the neural network will be designed as the sum of the state value function and the advantage function. The former is used to evaluate the state value of the input neural network, and the latter is used to evaluate each action that can be taken under this state feature. At the same time, the mean value of the advantage function is limited to 0 to ensure that the state value function will be adaptively adjusted each time it is updated, so as to ensure that it can always play a role in the process of evaluating the state value of the neural network. Attached Figure Description

[0027] Figure 1 A flowchart of a reconfigurable workshop dynamic scheduling method based on adversarial mechanism dual DQN algorithm and rule evolution provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a composite rule for workpiece selection based on a genetic programming algorithm, provided in an embodiment of the present invention. Figure 3 A schematic diagram illustrating the interaction strategy between the intelligent agent and the workshop environment provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the agent training network and the improved DQN algorithm provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] To improve the scientific nature and real-time responsiveness of reconfigurable workshop scheduling, this invention applies reinforcement learning to the scheduling problem of reconfigurable workshops, proposing a dynamic scheduling method for reconfigurable workshops based on reinforcement learning and rule evolution. The application of reinforcement learning enables continuously trained agents to respond efficiently to new production situations, selecting the most appropriate action strategy to maximize cumulative profits, thereby reducing enterprise management costs and ensuring the stable operation of the production system.

[0031] To effectively apply deep reinforcement learning methods to the scheduling problem of reconfigurable workshops, this invention establishes a dynamic scheduling architecture for reconfigurable workshops, including a task pool, a device pool, topology units, decision agents, and interaction strategies between agents and the workshop environment. The system comprises several components: a task pool stores all workpieces awaiting processing in the production workshop at the current moment, representing unfinished workpieces that have not yet been scheduled for processing; an equipment pool stores idle equipment in the reconfigurable workshop, which, as the name suggests, are auxiliary processing resources, including control consoles, power supplies, water tanks, etc.; a topology unit is formed by combining multiple devices in the reconfigurable workshop and can complete the processing of a specific workpiece; each topology unit maintains a buffer to store workpieces assigned to that topology unit, which can be understood as a machine used to complete workpiece processing steps, such as a reconfigurable machine tool or a multi-functional reconfigurable workstation; multiple topology units are available for each workpiece and each step of the process, and different types of topology units have different processing times for the same step; and a decision-making agent is used to decide the processing order of workpieces and the allocation and reorganization of topology units, with the optimization objective being to minimize the total delay. Furthermore, this invention designs a high-quality action space based on composite scheduling rules, where the composite scheduling rules are evolved from a single workpiece selection heuristic rule through a genetic programming algorithm, enabling adaptation to more complex scenarios.

[0032] To facilitate decision-making agents in making decisions at appropriate times and selecting composite scheduling rules that are adapted to the current environmental state of the reconfigurable workshop, this invention formulates decision points based on the processing characteristics of the reconfigurable workshop. Decision points include two scenarios: one is the initial moment, and the other is the moment when the task pool is not empty and a certain topological unit has just completed its process.

[0033] To avoid dynamic disturbances such as the arrival of new workpieces interfering with the current decision-making process, when a new workpiece arrives, this invention first determines whether the decision-making agent has entered the decision point. If so, the new workpiece information is placed in the task pool, and the workshop's state characteristic information is updated. If not, the new workpiece information is placed in the processing environment, and the system waits for the decision-making agent to enter the decision point. When the decision-making agent enters the decision point, the new workpiece information is then placed from the processing environment into the task pool, and the workshop's state characteristic information is updated again.

[0034] To further optimize the scheduling results, in some preferred embodiments, this invention further designs multiple statistical state features of the reconfigurable workshop. While comprehensively considering the shortest delay and the highest machine utilization, it reduces the dimensionality of the state feature representation, extracts the key properties of the state features, and improves the generalization ability of the agent model. A hierarchical strategy for topological unit allocation and reorganization is proposed to ensure that each process of the workpiece is completed as early as possible, which is conducive to minimizing the total delay. In line with the optimization objective, this invention designs a reward function based on the estimated difference in total delay between adjacent decision points.

[0035] In some preferred embodiments, this embodiment improves upon the DQN (Deep Q Network) algorithm by incorporating Double DQN, Dueling DQN, and a priority experience replay mechanism. Double DQN adds an evaluation network on top of the training network; Dueling DQN vectorizes the neural network output; and the priority experience replay mechanism extracts a certain number of samples from the experience pool for learning after a fixed number of training iterations based on experience priority. Based on these improvements, the DQN algorithm used in the relevant embodiments is actually an adversarial double DQN algorithm. The adoption of the adversarial double DQN algorithm enhances the model's learning ability, facilitating efficient learning-based decision-making in production scenarios and enabling the output of better production scheduling solutions in a shorter time.

[0036] To further optimize the scheduling results, in a preferred embodiment, the neural network architecture in the decision agent is also optimized.

[0037] The following is a detailed explanation.

[0038] In one embodiment of the present invention, namely Embodiment 1, a reconfigurable workshop scheduling method based on reinforcement learning and rule evolution is provided, such as... Figure 1 As shown, during the initialization phase, this embodiment will perform the following operations: Based on the preset job selection heuristic rules, a genetic programming algorithm is used to evolve multiple composite scheduling rules, which constitute the action space of the decision-making agent. Initialize the task pool and device pool, which are used to store all workpieces to be processed and idle devices in the reconfigurable workshop in real time, respectively. A decision agent based on the DQN algorithm is constructed. The optimization objective of the decision agent is to minimize the total delay. The action space consists of multiple composite scheduling rules evolved from the preset heuristic rules for selecting workpieces through a genetic programming algorithm.

[0039] It is easy to understand that, at the initial moment, the state characteristic information of the reconfigurable workshop will also be initialized.

[0040] In this embodiment, the composite scheduling rule is represented by a binary tree structure, its function set is a series of operators used to represent various operational relationships, and the terminal set is a heuristic rule for selecting workpieces; The operators include: "+", "-", "×", "÷", "Max", and "Min"; when the divisor is 0, the result of the "÷" operation returns a preset value; "Max" indicates the maximum value, and "Min" indicates the minimum value. The heuristic rules for selecting workpieces include: a) SPT (Job with The Shortest Processing Time): Select the job with the shortest processing time. b) LPT (Job with The Longest Processing Time): Select the workpiece with the longest processing time. c) SSO (Job with The Shortest Processing Time of Successor Operation): Select the job with the shortest processing time for the subsequent operation; d) LSO (Job with the Shortest Processing Time of Successor Operation): Select the job whose subsequent processing time is the longest; e) SRPT (Job with The Shortest Remaining Processing Time): Select the job with the shortest remaining processing time; f) LRPT (Job with The Longest Remaining Processing Time): Selects the job with the longest remaining processing time; g) EDD (Job with The Earliest Due Date): Select the job with the earliest delivery date; h) FIFO (First In, First Out): First-in, first-out rule; i) LNRO (Job with The Largest Number of Remaining Operations): Select the job with the largest number of remaining operations; j) SNRO (Job with The Smallest Number of Remaining Operations): Select the job with the smallest number of remaining operations.

[0041] Because the dimensions of attributes differ across rules, the heuristic rules for workpiece selection cannot be directly and simply combined. It is necessary to normalize the workpiece attributes to determine the priority of each workpiece under each heuristic rule. Taking rule a) as an example, the workpieces in the task pool... The priority weights are:

[0042] in, Indicates workpiece process The average processing time.

[0043] In this embodiment, the operators of the composite scheduling rules are managed as functions through a function set, and the single workpiece selection heuristic rules are managed as terminators through a terminal set. Optionally, in this embodiment, the binary tree used to represent the composite scheduling rules is generated using a hybrid method, with 50% of the individuals generated by the complete method and 50% by the growth method. In the complete method, all terminators in the generated individuals are located at the maximum depth of the tree; in the growth method, all nodes of the tree can be arbitrarily selected from the operator set and the heuristic rule set.

[0044] The algorithm flow of the genetic programming algorithm is as follows: Figure 2 As shown, the fitness function in the genetic programming algorithm is determined based on the estimated total delay. The fitness function value is negatively correlated with the performance of an individual. The genetic programming algorithm receives the fitness value, updates the population, and outputs better rules. Compared with scheduling rules calculated manually or summarized from experience, scheduling rules generated based on genetic programming can obtain more scientific results in a shorter time, which is beneficial for adapting to production environments with frequent state transitions.

[0045] Figure 2 The example of the composite scheduling rule shown can be represented as: (LPT*SSO)+SPT Based on this composite scheduling rule, the priority of each workpiece in the task pool is calculated, the workpieces are sorted according to their priority, and topology units are assigned to the workpieces in order.

[0046] In this embodiment, the state feature information of the reconfigurable workshop is multi-statistical state feature information, including statistical features oriented towards workpieces and statistical features oriented towards topological units. Statistical characteristics oriented towards the workpiece include: (a) The average completion rate of all workpieces; (b) The standard deviation of the completion rate for all workpieces; (c) The average remaining processing time for all workpieces; (d) The standard deviation of the remaining processing time for all workpieces; Statistical characteristics oriented towards topological units include: (e) The average utilization rate of all topological units; (f) Standard deviation of utilization of all topological units.

[0047] The aforementioned state feature information is a real-time feature that changes in real time as the workpiece is processed and new workpieces arrive. It comprehensively considers the shortest delay and the highest machine utilization rate, and uses statistical features to extract important information from the original state, reducing the dimensionality of the state feature representation, extracting the key properties of the state features, improving the generalization ability of the agent model, and being able to adapt to noise and unnecessary changes in the state feature space.

[0048] In this embodiment, the optimization objective of the decision-making agent is to minimize the total delay time. At any given time, the total delay time of the workpiece can be expressed as:

[0049] in, This indicates the number of workpieces currently existing in the production workshop. This indicates the total delay time for all workpieces. Indicates workpiece The completion time of the final process. Indicates workpiece Delivery date.

[0050] Based on the above calculation method, when the completion time is earlier than the delivery date, the delay is 0 instead of a negative value. This design can prevent the agent from pursuing the early delivery of individual processes, which is not conducive to shortening the total delay.

[0051] After initialization, the decision-making agent can interact with the environment of the reconfigurable workshop to determine the start and end times of each workpiece and each process, as well as the corresponding allocated topological units. For example... Figure 1 and Figure 3 As shown, the interaction strategies between the decision-making agent and the reconfigurable workshop include: For a topological unit currently processing a workpiece, if any of these units completes its processing at that moment and returns the released workpiece to the task pool, the decision-making agent enters the decision point at that instant. Upon entering the decision point, the agent makes a decision according to the following steps to achieve dynamic scheduling of that decision point: S1: Feed back the current state characteristics of the reconfigurable workshop and the reward value after executing the action obtained from the previous decision to the decision agent, train the neural network in the decision agent, and output the probability of each composite scheduling rule in the action space being selected by the neural network; the smaller the estimated total delay, the larger the reward value. S2: Select the composite scheduling rule with the highest probability as the action of the decision agent at the current decision point, and select the workpiece to be processed from the task pool according to the selected composite scheduling rule, and assign it the topological unit required for the current process.

[0052] In step S1 of this embodiment, a reward function based on the total estimated delay difference between adjacent decision points is specifically designed to calculate the reward value. At the current decision point... t The estimated total delay for the workpiece is:

[0053] in, Indicates the number of available topological units; at the current decision point t , Indicates the number is The earliest completion time of the topology unit, Indicates workpiece The completion time of the last scheduled process; Based on the above calculation expression for the estimated total delay of the workpiece, the agent makes a decision at the current decision point. t The reward function can be expressed as:

[0054] in, This represents the reward value at the current decision point. and These represent the estimated total delays at the current decision point and the next decision point, respectively. and These represent the topology unit utilization rates of the current decision point and the previous decision point, respectively.

[0055] Since the optimization objective of the decision agent is to minimize the total delay, this reward function setting can set a gradient for the estimated delay, ensuring that the objective function value can continuously decrease during training. In the early decision-making process, since the estimated total delay of the workpiece is zero, the agent has difficulty evaluating the value of the actions taken, thus making it impossible to train and update the agent's network. This embodiment can effectively solve the problem of the difficulty in early learning and training of the decision agent by introducing the difference in the utilization rate of topological units at adjacent decision points into the reward function.

[0056] In this embodiment, the neural network of the decision-making agent is specifically designed as a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multilayer perceptron. The recurrent neural network is used for training and learning to obtain feature parameters, and the last layer of the network is fully connected; the multilayer perceptron is applied to the fully connected layer of the recurrent neural network to perform the final classification calculation on the feature parameters output by the recurrent neural network, and obtain the action value function value predicted for each sample.

[0057] Under this neural network design approach, the recurrent neural network layer enables deep reinforcement learning algorithms to optimize short-term benefits while considering long-term returns, which is crucial for improving the overall production efficiency and task completion rate of the workshop. Meanwhile, the multilayer perceptron layer can further learn and map complex nonlinear relationships from the features extracted by the recurrent neural network layer, enhancing the model's expressive power and thus better approximating the optimal policy function. This combined network architecture provides flexibility in handling complex scheduling problems. In the face of uncertainty and interference, the algorithm can also make robust decisions and optimize long-term production efficiency.

[0058] In step S2 of this embodiment, when allocating the topology units required for the current process to the selected workpiece to be processed, a layered strategy is adopted; the layered strategy includes: T1. Taking the current process of the selected workpiece as the target processing object, based on the workpiece's process information, determine the topological units that can be used to process the target processing object and the processing time of the target processing object on each topological unit. Select the topological unit with the shortest processing time as the first target topological unit; specifically, it is expressed as follows:

[0059] in, Represents the topological unit of the allocation. It is a workpiece The The process in the topological unit On the processing time, It is a workpiece The A set of optional topological units for each process step; T2. If the devices in the device pool can be combined into the first target topology unit, then allocate the corresponding devices from the device pool to form the first target topology unit, and then allocate it to the target processing object. The allocation ends. Otherwise, proceed to T3. T3. If the devices in the device pool can be combined into other topological units for processing the target processing object, then select the topological unit with the shortest processing time as the second target topological unit, allocate the corresponding devices from the device pool to combine into the second target topological unit, and then allocate it to the target processing object. The allocation ends; otherwise, proceed to T4. T4. Determine the topology units currently performing processing tasks and available for use with the target processing object. Then, calculate the completion time for each topology unit after completing the processing task, including the target processing object. Finally, select the topology unit with the earliest completion time as the third target topology unit, add the target processing object to the buffer of the third target topology unit, and the allocation ends. The specific representation is as follows:

[0060] in, It is a workpiece The The process in the topological unit The remaining processing time, It is a topological unit Workpiece in buffer The The processing time for each step; If the buffer of a topology unit is empty and there are no workpieces being processed on the topology unit at this time, the topology unit is in an idle state. Idle topology units can be disassembled into equipment and put back into the equipment pool for subsequent topology unit reconstruction, so as to improve the utilization efficiency of reconfigurable workshop resources.

[0061] In the above-mentioned layered strategy, T1~T2 constitute the allocation layer, and T3~T4 constitute the reorganization layer. Such topological unit allocation rules can enable workpiece processes to be completed as soon as possible, improve the processing efficiency of the reconfigurable workshop, and shorten the total delay time.

[0062] By making decisions and scheduling in the above manner, once all workpieces to be processed are completed, a production schedule for the reconfigurable workshop can be output, including the start time, end time, and assigned topological unit for each process of each workpiece.

[0063] like Figure 4 As shown, in this embodiment, the improvement to the DQN algorithm is specifically as follows: (1) Integrating the double DQN mechanism to solve the overestimation problem in DQN. The specific implementation is to introduce two neural networks: one network is used for training, selecting actions and updating in each iteration, and the other network is used for evaluation, updating at intervals. This mechanism separates the calculation of predicted values ​​and target values ​​during the evaluation process, which can weaken the negative impact of poor training sets on training and accelerate algorithm convergence. The mathematical expression for Q-value calculation under this mechanism is as follows:

[0064] in, This represents the predicted Q-value calculated during network training; This represents the target Q-value calculated by the neural network used to evaluate the agent; This represents the state characteristics at the next moment, and the Q-value is calculated using the state characteristics in the target network during evaluation; Indicates the learning rate; Indicates the discount factor; This represents the reward value.

[0065] (2) Integrating the Dueling DQN mechanism, the output of the neural network is designed as the sum of a state value function and an advantage function. The former is used to evaluate the state feature value of the input neural network, and the latter is used to evaluate each action that can be taken under this state feature. At the same time, the mean of the advantage function value is limited to 0 to ensure that the state value function is adaptively adjusted with each update, so as to ensure that it can always play a role in the process of evaluating the state value. This structural design makes Dueling DQN more efficient in processing state features that are not greatly affected by actions. The mathematical expression for calculating the Q value under this mechanism is as follows:

[0066] in, It is a state-value function that outputs a scalar whose value depends only on the state eigenvalues. and training parameters related; It is the advantage function, which outputs a vector whose length is equal to the size of the action space. The magnitude of this value is related to the state feature value. With action value and parameters related; Refers to the parameters of the network's convolutional layers; and These are the parameters of the fully connected layers in the two branches, respectively; Ensure that the mean of the dominance function value is 0.

[0067] (3) Integration of priority experience replay mechanism. Specifically, the sample pool is initialized at the initial moment, and after each evaluation by the decision agent, the state feature information of the current decision point is used. ,action Reward Value and the state characteristic information of the next decision point As a sample The samples are added to the sample pool; after a fixed number of training iterations, a predetermined number of samples are drawn from the sample pool in descending order of importance to train the neural network in the decision-making agent. For the samples... Its importance is calculated by the difference between its current Q-value and the target Q-value in the time-series difference, as shown in the following expression:

[0068] In order to make training To minimize the expected value, a greedy mechanism is used to maintain a priority queue, prioritizing updates. Large sample size.

[0069] The improved DQN algorithm used in this embodiment considers the separation of training and evaluation networks and the hierarchical calculation of Q-values ​​based on DQN, and improves the experience replay mechanism of randomly sampled samples, making the algorithm more efficient.

[0070] In summary, this embodiment provides a reconfigurable workshop agent system with a dynamic disturbance event response mechanism. Aiming to minimize total delay, it designs multiple real-time features combining workpiece processing information and workshop resource information. The agent makes decisions based on the real-time state characteristics of the workshop, thereby achieving adaptive scheduling when dynamic disturbance events occur in the resource-coordinated assembly workshop, improving the production efficiency of the manufacturing workshop. A workpiece selection composite rule based on genetic programming evolution is provided. This composite rule comprehensively considers multiple factors influencing decision-making and optimizes the composite rule based on the relationships between these factors. This allows the decision-making process to consider different levels of importance for each factor, significantly improving the model's ability to search for optimal solutions. The designed training network uses a recurrent neural network combined with a multilayer perceptron model. This allows for a more thorough consideration of the impact of time series on the allocation of action selection probabilities, improving the agent's ability to handle continuous processes and enhancing the model's adaptability and robustness. The designed reward mechanism, based on the total estimated delay difference between adjacent decision points, trains the agent to make decisions that reduce the total delay. The cumulative reward is the negative value of the sum of the estimated delay gradients. The training objective is to minimize the total delay of the workpiece, achieving a better solution by maximizing the cumulative reward. The provided deep reinforcement learning algorithm incorporates three improvements on the DQN algorithm: a "training and evaluation separation" and "Q-value vectorization" processing mode are added to the evaluation network of the neural network; and the experience replay mechanism is optimized, improving the model's learning ability. This facilitates efficient learning-based decision-making in production scenarios, enabling the output of better production scheduling solutions in a shorter time.

[0071] Example 2: A computer program product includes a computer program that, when executed by a processor, implements the reconfigurable shop floor scheduling method based on reinforcement learning and rule evolution provided in Embodiment 1 above.

[0072] Example 3: A computer-readable storage medium includes a stored computer program that, when executed by a processor, controls the device where the computer-readable storage medium is located to execute the reconfigurable shop floor scheduling method based on reinforcement learning and rule evolution provided in Embodiment 1 above.

[0073] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic scheduling method for reconfigurable workshops based on reinforcement learning and rule evolution, characterized in that, include: Initialize the task pool and the equipment pool to store all workpieces to be processed and idle equipment in the reconfigurable workshop in real time, and construct a decision agent based on the DQN algorithm with the goal of minimizing the total delay. The optimization objective of the decision-making agent is to minimize the total delay of workpiece processing. The action space consists of multiple composite scheduling rules evolved from the preset workpiece selection heuristic rules by a genetic programming algorithm. At each decision point, dynamic scheduling is performed according to the following steps: S1: Feed back the current state feature information of the reconfigurable workshop and the reward value after executing the action obtained from the previous decision to the decision agent, train the neural network in the decision agent and output the probability of each composite scheduling rule in the action space being selected by the neural network; the smaller the estimated total delay, the larger the reward value; S2: Select the composite scheduling rule with the highest probability as the action obtained by the decision-making agent at the current decision point, and select the workpiece to be processed from the task pool according to the selected composite scheduling rule, and allocate the topology unit required for the current process to it; wherein, the task pool is responsible for storing the workpieces to be processed in the workshop, and the workpiece to be processed refers to the unfinished workpiece that is waiting to be processed. After all workpieces to be processed are completed, the production schedule of the reconfigurable workshop is output; the production schedule includes the start time, end time and assigned topological unit for each process of each workpiece. The decision point includes two scenarios: one is the initial moment, and the other is the moment when the task pool is not empty and a certain topological unit has just completed its process. The topology unit refers to a processing unit formed by combining several devices. One combination of devices corresponds to one type of topology unit, which can process a specified type of operation. If a topology unit is idle at a certain time, it will be dismantled and all its devices will be added to the device pool to achieve flexibility and agility in the production process. For each topology unit, a buffer is maintained to store workpiece operations that have been assigned to the corresponding topology unit but cannot be processed at present. Furthermore, in step S2, when allocating the topology units required for the current process to the selected workpiece to be processed, a layered strategy is adopted; the layered strategy includes: T1. Take the current process of the selected workpiece to be processed as the target processing object. Based on the process information of the workpiece, determine the topological units that can be used to process the target processing object, as well as the processing time of the target processing object on each topological unit. Select the topological unit with the shortest processing time as the first target topological unit. T2. If the devices in the device pool can be combined to form the first target topology unit, then the corresponding devices are allocated from the device pool to form the first target topology unit and then allocated to the target processing object, and the allocation ends; otherwise, proceed to T3. T3. If the devices in the device pool can be combined into other topological units for processing the target processing object, then the topological unit with the shortest processing time is selected as the second target topological unit. After the corresponding devices are allocated from the device pool to form the second target topological unit, it is allocated to the target processing object, and the allocation ends; otherwise, proceed to T4. T4. Determine the topology unit that is currently performing a processing task and can be used for the target processing object. Then calculate the completion time of each topology unit for completing the processing task, including the target processing object. Finally, select the topology unit with the smallest completion time as the third target topology unit, add the target processing object to the buffer of the third target topology unit, and the allocation ends. The hierarchical strategy also includes: If a topology unit does not perform a processing task and its buffer is empty, the topology unit is disassembled into a device and the disassembled device is placed into the device pool. The reward function used to calculate the reward value is: in, t Indicates the current decision point. t -1 and t +1 represents the previous and next decision points of the current decision point, respectively; This represents the reward value at the current decision point. and These represent the estimated total delays at the current decision point and the next decision point, respectively. and These represent the topology unit utilization rates of the current decision point and the previous decision point, respectively.

2. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in claim 1, characterized in that, The composite scheduling rules are represented in a binary tree structure; The function set of the binary tree is a series of operators used to represent various operational relationships; The terminal set of the binary tree is the heuristic rule for selecting the workpiece; The operators include: "+", "-", "×", "÷", "Max", and "Min"; when the divisor is 0, the result of the "÷" operation returns a preset value; "Max" indicates the maximum value, and "Min" indicates the minimum value. The heuristic rules for selecting the workpiece include: a) SPT: Select the workpiece with the shortest processing time; b) LPT: Select the workpiece with the longest processing time; c) SSO: Select the workpiece with the shortest processing time in subsequent processes; d) LSO: Select the workpiece with the longest processing time in the subsequent process; e) SRPT: Select the workpiece with the shortest remaining processing time; f) LRPT: Select the workpiece with the longest remaining processing time; g) EDD: Select the workpiece with the earliest delivery date; h) FIFO: First-In-First-Out rule; i) LNRO: Select the workpiece with the largest number of remaining operations; j) SNRO: Select the workpiece with the fewest remaining operations.

3. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in claim 1, characterized in that, The state characteristic information of the reconfigurable workshop includes: statistical characteristics oriented towards workpieces and statistical characteristics oriented towards topological units; The statistical characteristics oriented towards the workpiece include: (a) The average completion rate of all workpieces; (b) The standard deviation of the completion rate for all workpieces; (c) The average remaining processing time for all workpieces; (d) The standard deviation of the remaining processing time for all workpieces; The statistical characteristics oriented towards topological units include: (e) The average utilization rate of all topological units; (f) Standard deviation of utilization of all topological units.

4. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in claim 1, characterized in that, The neural network in the decision-making agent is a recurrent neural network, and the fully connected layer of the recurrent neural network is implemented by a multilayer perceptron.

5. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in any one of claims 1 to 4, characterized in that, Also includes: Construct an evaluation agent based on the DQN algorithm; The neural network in the evaluation agent is used to evaluate the Q-value calculated by the decision agent.

6. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in claim 5, characterized in that, Also includes: Initialize the sample pool; Furthermore, after each evaluation by the decision-making agent, the state feature information of the current decision point is recorded. ,action Reward Value and the state characteristic information of the next decision point As a sample Add to the sample pool; Furthermore, after a fixed number of training sessions, a preset number of samples are drawn from the sample pool in descending order of importance to train the neural network in the decision-making agent.

7. The reconfigurable workshop dynamic scheduling method based on reinforcement learning and rule evolution as described in claim 6, characterized in that, The algorithm used to construct the decision-making agent and the evaluation agent is the dueling DQN algorithm.

Images