Method and system for dynamic scheduling of a machine shop based on multi-agent deep deterministic policy gradient

Abstract

This invention belongs to the field of production scheduling technology, specifically relating to a dynamic scheduling method and system for a machining workshop based on multi-agent deep deterministic policy gradient. The method includes: constructing a multi-agent system from storage, machines, and buffer zones; establishing a Markov game process encompassing machine selection and workpiece sequencing; designing state features and rule-weighted action spaces for each agent, and setting a shared reward function; after each agent perceives the observed value, it outputs rule weights through a policy network, which are then weighted and aggregated with multiple single scheduling rules and applied to the environment to generate experience tuples for network updates, iterating repeatedly until convergence. This invention employs the MADDPG algorithm to achieve joint training of multiple agents, enabling each agent to adaptively select the optimal weighted combination scheduling rule, exhibiting good versatility and scheduling performance even under machine failure environments.

Description

Technical Field

[0001] This invention belongs to the field of production scheduling technology, specifically relating to a dynamic scheduling method and system for machining workshops based on multi-agent deep deterministic strategy gradient. Background Technology

[0002] As a crucial production unit in the manufacturing industry, the machining workshop primarily processes various mechanical parts through machining processes such as turning and milling, and is widely used in fields such as automotive parts manufacturing and aerospace manufacturing. With consumers increasingly demanding personalized products, machining workshops are gradually shifting towards a flexible production model characterized by diverse product types and small batches. Against this backdrop, the manufacturing environment of the machining workshop exhibits high uncertainty, such as the arrival of random orders and sudden interruptions in manufacturing resources, necessitating effective dynamic production scheduling to ensure efficient manufacturing execution response.

[0003] Over the past few decades, numerous classic dynamic scheduling methods have been proposed, such as scheduling rules, genetic algorithms, and ant colony algorithms. Among them, scheduling rules have been widely used in enterprises due to their advantages of rapid scheduling and flexible implementation. However, research shows that under different production environment configurations, no single scheduling rule is consistently superior to others. In recent years, the rise of reinforcement learning has provided new solutions to these problems. In particular, multi-agent reinforcement learning, by effectively mitigating the curse of dimensionality caused by high-dimensional action spaces, has been widely applied in the field of dynamic scheduling in workshops.

[0004] The invention patent [Publication No.: CN117891220A, Publication Date: 2024.04.16] discloses a distributed hybrid flow shop scheduling method based on multi-agent deep reinforcement learning. This method maps each machine to an agent and trains the multi-machine agents through multi-agent reinforcement learning to optimize workpiece sorting and scheduling decisions. However, this method often neglects the joint training of machine selection scheduling decisions, thus limiting further improvement in overall scheduling performance. In addition, existing research mainly focuses on the dynamic event of new workpiece arrival, while the impact of machine fault disturbances is less explored.

[0005] Therefore, how to achieve joint training of two-stage scheduling decisions for workpiece sorting and machine selection in a machining workshop based on multi-agent reinforcement learning under machine fault disturbance environment, so as to further improve scheduling performance, has become an important issue that urgently needs in-depth research. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for dynamic scheduling of a machining workshop based on multi-agent deep deterministic policy gradient, so as to solve the problems mentioned in the background art.

[0007] The present invention achieves the above objectives through the following technical solutions: This invention proposes a dynamic scheduling method for a machining workshop based on multi-agent deep deterministic policy gradient, the method comprising: The warehouse, machines, and buffer zones are constructed as warehouse intelligent agents, machine intelligent agents, and buffer intelligent agents, respectively, and a Markov game process covering machine selection and workpiece sorting is established. Design machine state characteristics and machine selection weight action space for warehouse agents and machine agents, design workpiece state characteristics and workpiece sorting weight action space for buffer agents, and set a shared reward function for all agents. Each agent perceives the observed value at the current scheduling moment, outputs the rule weight through the policy network, and applies it to the environment after being weighted and aggregated with multiple single scheduling rules, so that the workshop can be transferred to the next state and receive a reward. Then, the observed value, weight, reward and new observed value are stored in the experience pool as experience tuples. Each agent samples the experience tuples from the experience pool, updates the network using the sampled data, and iterates until convergence, thereby achieving dynamic scheduling of the machining workshop under machine fault disturbances.

[0008] As a preferred embodiment, the types of scheduling decision points experienced by the Markov game process include: The arrival time of the initial workpiece, the arrival time of the new workpiece, the completion time of the process, and the time of machine failure; The warehouse intelligence agent and the machine intelligence agent are responsible for machine selection and scheduling decisions. The warehouse intelligence agent selects the processing machine for the first process of the workpiece, and the machine intelligence agent selects the processing machine for the subsequent processes of the workpiece. When a machine failure occurs, the machine intelligence agent corresponding to the failed machine selects an available processing machine for the affected workpiece. The buffer intelligence agent is responsible for workpiece sorting and scheduling decisions.

[0009] As a preferred embodiment, the machine status characteristics include: average machine utilization rate, the ratio of the standard deviation of machine utilization rate to the average value, the difference between the maximum and minimum values ​​of machine utilization rate, the percentage of busy machines, the ratio of the standard deviation of machine load to the average value, and the ratio of the maximum and average values ​​of machine load. The workpiece status characteristics include: the number of workpieces in the system, the proportion of delayed workpieces, the average number of compatible machines for workpieces to be processed, the average processing time of workpieces to be processed, the minimum processing time of workpieces to be processed, the average remaining processing time of workpieces to be processed, the minimum remaining processing time of workpieces to be processed, the minimum relaxation time of workpieces to be processed, the average relaxation time of workpieces to be processed, the maximum delay time of workpieces to be processed, the average delay time of workpieces to be processed, the average critical ratio of workpieces to be processed, the average interval between the two most recent arriving workpieces, and the average interval between the five most recent arriving workpieces.

[0010] As a preferred embodiment, the multiple single scheduling rules corresponding to the machine selection weight action space include: minimum machine processing time priority rule, minimum machine load priority rule, and minimum buffer processing queue priority rule; The multiple single scheduling rules corresponding to the workpiece sorting weight action space include: minimum processing time priority rule, shortest delivery time priority rule, and shortest modification delivery time priority rule.

[0011] As a preferred embodiment, the shared reward function is expressed as follows: ,in and These represent the current state of the workshop. and the next state The average latency value.

[0012] As a preferred embodiment, the policy network is composed of a multi-layer fully connected neural network, with the number of neurons in the input layer corresponding to the dimension of the agent's observation features and the number of neurons in the output layer corresponding to the dimension of the action space; wherein, the warehouse agent and the machine agent adopt the same policy network structure.

[0013] As a preferred approach, each agent incorporates exploration noise when outputting rule weights. To form new weight values ,in For the policy network of the h-th agent, Let h be the observation value of the h-th agent.

[0014] As a preferred solution, specific methods for updating the network include: Each agent samples a mini-batch of Z samples from the experience pool, where the g-th (g=1,…,Z) sample contains the observations of all N agents. Action value Shared rewards and new observations ; By minimizing the loss function Update value network parameters ,in Represents all input variables of the value network, and the target value. , Z represents the reward discount factor, and Z represents the number of samples in the mini-batch. Through policy gradient Update strategy network parameters ; And replace the target value network with the updated parameters. and target policy network The parameters.

[0015] This invention further proposes a dynamic scheduling system for machining workshops based on multi-agent deep deterministic policy gradients, used to implement the dynamic scheduling method for machining workshops as described above. The system includes: The agent construction module is used to construct the warehouse, machine and buffer as warehouse agents, machine agents and buffer agents respectively, and establish a Markov game process covering machine selection and workpiece sorting. The feature and space design module is used to design machine state features and machine selection weight action space for warehouse agents and machine agents, design workpiece state features and workpiece sorting weight action space for buffer agents, and set a shared reward function for all agents. The decision interaction module is configured in each intelligent agent to perceive the observed value at the current scheduling moment, output the rule weight through the policy network, and then apply it to the environment after weighted aggregation with multiple single scheduling rules to make the workshop move to the next state and get a reward. The current observed value, rule weight, reward and new observed value are stored as experience tuples in the experience pool. The network update module is used to sample the experience tuples from the experience pool, update the value network and policy network of each agent using the sampled data, and iterate cyclically until the policy converges, thereby realizing dynamic scheduling of the machining workshop under machine fault disturbances.

[0016] The beneficial effects of this invention are as follows: 1. This invention constructs corresponding agents for various types of manufacturing equipment (including storage, machines, and buffer zones) in a machining workshop, and clarifies the scheduling decision types responsible for each agent, covering two categories of scheduling decisions: machine selection and workpiece sequencing. Simultaneously, under machine failure disturbances, a Markov game process incorporating the above two types of decisions is constructed, thereby enabling joint training of machine selection and workpiece sequencing using multi-agent reinforcement learning.

[0017] 2. This invention uses the MADDPG algorithm to train multi-device agents in a machining workshop, enabling each agent to adaptively select the optimal machine selection rule weight or workpiece sorting rule weight at each scheduling decision moment, thereby forming a weighted combined scheduling rule, which has good versatility in machine failure environments. Attached Figure Description

[0018] Figure 1 This is a flowchart of a dynamic scheduling method for a machining workshop according to one embodiment of the present invention; Figure 2 This is another flowchart of a dynamic scheduling method for a machining workshop in one embodiment of the present invention; Figure 3 This is a schematic diagram of the layout of the experimental environment in a machining workshop according to one embodiment of the present invention; Figure 4This is a schematic diagram of a Markov game process for dynamic scheduling of a machining workshop in one embodiment of the present invention; Figure 5 This is a schematic diagram of the policy network structure of the warehouse agent and the machine agent in one embodiment of the present invention; Figure 6 This is a schematic diagram of the policy network structure of the buffer agent in one embodiment of the present invention; Figure 7 This is a system block diagram of a dynamic scheduling system for a machining workshop according to one embodiment of the present invention; Figure 8 This is a cumulative reward curve of each training segment when using MADDPG to train multiple agents in one embodiment of the present invention; Figure 9 This is a graph showing the average latency of each training segment when using MADDPG to train multiple agents in one embodiment of the present invention. Figure 10 This is a comparison chart showing the ranking percentages of the method of the present invention and nine composite scheduling rules under 27 production environment configurations in one embodiment of the present invention. Detailed Implementation

[0019] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0020] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0021] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0022] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0023] This invention proposes a dynamic scheduling method for a machining workshop based on multi-agent deep deterministic policy gradient. The method uses the multi-agent deep deterministic policy gradient (MADDPG) algorithm as the training framework. Through centralized training and distributed execution mechanism among multiple agents, it realizes the joint optimization of the two-stage scheduling strategy of machine selection and workpiece sorting, so as to cope with the frequent machine failure disturbances in the machining workshop.

[0024] MADDPG is a reinforcement learning algorithm suitable for continuous action spaces in multi-agent environments. Within this framework, each agent has an independent policy network and value network. During training, each agent's value network acquires global information (i.e., the observations and actions of all agents) for value evaluation. This allows each agent to perceive the impact of changes in other agents' policies on the joint action's utility when updating its own policy, effectively mitigating the environmental non-stationarity problem caused by the simultaneous evolution of agent policies in multi-agent environments. After training, each agent can output decision actions through its policy network based solely on its local observations during the execution phase, without relying on information from other agents or communicating with each other. This reduces communication overhead and coordination complexity during practical deployment, facilitating implementation in industrial environments.

[0025] First Embodiment Please see Figure 1 and Figure 2 This embodiment proposes a dynamic scheduling method for a machining workshop based on multi-agent deep deterministic policy gradient. This method achieves joint optimization of the two-stage scheduling strategy—machine selection and workpiece sequencing—through a multi-agent collaborative decision-making mechanism to address frequent machine failure disturbances in the machining workshop. The method includes: S1. Construct the warehouse, machines, and buffer as warehouse intelligent agents, machine intelligent agents, and buffer intelligent agents, and establish a Markov game process covering machine selection and workpiece sorting. S2. Design machine state characteristics and machine selection weight action space for warehouse intelligent agents and machine intelligent agents, design workpiece state characteristics and workpiece sorting weight action space for buffer intelligent agents, and set a shared reward function for all intelligent agents. S3. Each agent perceives the observed value at the current scheduling moment, outputs the rule weight through the policy network, and applies it to the environment after being weighted and aggregated with multiple single scheduling rules, so that the workshop can be transferred to the next state and receive a reward. Then, the observed value, weight, reward and new observed value are stored in the experience pool as experience tuples. S4. Each agent samples experience tuples from the experience pool, updates the network using the sampled data, and iterates until convergence, thereby realizing dynamic scheduling of the machining workshop under machine fault disturbances.

[0026] In this embodiment, the types of scheduling decision points experienced by the Markov game process include: The arrival time of the initial workpiece, the arrival time of the new workpiece, the completion time of the process, and the time of machine failure; The four decision points mentioned above cover all key event types in the machining workshop's production execution process, ensuring that the scheduling strategy can respond promptly to any production disturbances. The warehouse agent and the machine agent are responsible for machine selection and scheduling decisions. The warehouse agent selects the processing machine for the first process of a workpiece, while the machine agent selects the processing machine for subsequent processes. When a machine malfunctions, the machine agent corresponding to the malfunctioning machine reselects an available processing machine for the affected workpiece. The buffer agent is responsible for workpiece sequencing and scheduling decisions. Through this division of responsibilities, each type of agent has a clear division of labor in the decision space, avoiding strategy conflicts caused by a single agent undertaking too many decision-making tasks.

[0027] Specifically, this embodiment constructs a machining workshop experimental environment, such as... Figure 3 As shown, the system consists of a warehouse and six machine tools, including two lathes, two milling machines, and two engraving machines. Each machine tool is equipped with a buffer of sufficient capacity for temporary storage of workpieces to be processed. Robotic arms and AGVs handle loading / unloading and transportation tasks, respectively; their operation time is not considered in this embodiment. The warehouse, machine tools, and buffers are all encapsulated as intelligent agents: one warehouse agent, six machine tool agents, and six buffer agents. Based on these intelligent agents, Figure 4A Markov game process for two-stage scheduling decisions—machine selection and workpiece sequencing—was constructed. This Markov game process begins with the placement of a workpiece order and ends with the completion of that order. Throughout this process, the workshop continuously experiences the four types of scheduling decision points mentioned above. These four types of decision points cover all critical event types in the machining workshop's production execution process, ensuring that the scheduling strategy can respond promptly to any production disturbances.

[0028] At each scheduling decision point, each agent must observe and make corresponding scheduling decisions to drive the continuous execution of workshop production and reach the next decision point. The warehouse agent and the machine agent are responsible for machine selection and scheduling decisions. The warehouse agent selects the processing machine for the first process of a workpiece, while the machine agent selects the processing machine for subsequent processes. When a machine malfunctions, the machine agent corresponding to the malfunctioning machine selects an available processing machine for the affected workpiece. The buffer agent is responsible for workpiece sequencing and scheduling decisions. Through this division of responsibilities, each type of agent has a clear division of labor in the decision space, avoiding strategy conflicts caused by a single agent undertaking too many decision-making tasks.

[0029] In this embodiment, in order to achieve a comprehensive perception of the production status of the workshop, the machine status characteristics include: average machine utilization rate, the ratio of the standard deviation of machine utilization rate to the average value, the difference between the maximum and minimum values ​​of machine utilization rate, the percentage of busy machines, the ratio of the standard deviation of machine load to the average value, and the ratio of the maximum and average values ​​of machine load. The above six-dimensional characteristics describe the real-time operating status of each machine from the two dimensions of load balance and resource utilization rate.

[0030] The workpiece status characteristics include: the number of workpieces in the system, the proportion of delayed workpieces, the average number of compatible machines for workpieces to be processed, the average processing time for workpieces to be processed, the minimum processing time for workpieces to be processed, the average remaining processing time for workpieces to be processed, the minimum remaining processing time for workpieces to be processed, the minimum relaxation time for workpieces to be processed, the average relaxation time for workpieces to be processed, the maximum delay time for workpieces to be processed, the average delay time for workpieces to be processed, the average critical ratio for workpieces to be processed, the average interval between the two most recent arriving workpieces, and the average interval between the five most recent arriving workpieces. These 14 characteristics reflect the comprehensive status of workpieces to be processed in the buffer from three levels: workpiece load, delivery urgency, and arrival sequence, providing a sufficient information basis for the buffer agent to formulate a reasonable sorting strategy.

[0031] In this embodiment, in order to achieve flexible integration of scheduling rules, the multiple single scheduling rules corresponding to the machine selection weight action space include: minimum machine processing time priority rule, minimum machine load priority rule, and minimum buffer processing queue priority rule; the above three rules guide the machine selection decision from the three perspectives of processing efficiency, load balancing, and queue blocking mitigation, respectively.

[0032] The workpiece sorting weighted action space corresponds to several single scheduling rules, including: minimum processing time priority rule, shortest delivery time priority rule, and shortest modified delivery time priority rule. These three rules guide workpiece sorting decisions from three aspects: processing efficiency, delivery time urgency, and dynamic adjustment, respectively. By introducing the weighted action space, each agent can adaptively adjust the contribution level of each rule under different production environments, thereby overcoming the deficiency of insufficient generalization ability of a single fixed rule in variable environments.

[0033] Specifically, in this embodiment, the action space of the warehouse intelligent agent and the machine intelligent agent is defined as the weight variables corresponding to the three machine selection scheduling rules mentioned above. , and The action space of the buffer agent is defined as the weight variables corresponding to the three workpiece sorting and scheduling rules mentioned above. , and Each agent selects a set of weight values ​​from the action space through a policy network, and then performs a weighted fusion with the corresponding three rules to form a weighted combined scheduling rule.

[0034] In this embodiment, the shared reward function is represented as follows: ,in and These represent the current state of the workshop. and the next state The reward function is designed to ensure that all agents share the same optimization objective: minimizing the global average latency. This effectively guides the agents' strategies towards the cooperative optimum. The reward is positive when the workshop state improves and negative when it deteriorates. This immediate feedback mechanism clearly communicates the quality of their joint actions to each agent.

[0035] In this embodiment, the policy network consists of a multi-layer fully connected neural network. The number of neurons in the input layer corresponds to the dimension of the agent's observed features, and the number of neurons in the output layer corresponds to the dimension of the action space. The warehouse agent and the machine agent use the same policy network structure. This network structure design ensures that agents with similar decision-making logic can share the same feature extraction and policy mapping capabilities, thereby reducing training complexity and accelerating the convergence process.

[0036] Specifically, Figure 5 This demonstrates the policy network structure jointly employed by the warehouse agent and the machine agent. Figure 6 The policy network structure used by the buffer agent is shown. Both types of policy networks consist of 5 fully connected neural networks with 64 neurons in the hidden layers. This network structure design ensures that agents with similar decision-making logic can share the same feature extraction and policy mapping capabilities, thereby reducing training complexity and accelerating the convergence process.

[0037] In a preferred embodiment, each agent incorporates exploration noise when outputting rule weights. To form new weight values ,in For the policy network of the h-th agent, Let be the observation value of the h-th agent. By introducing a noise mechanism, the agent can fully explore the action space in the early stages of training, avoiding the policy from getting trapped in local optima too early. As training progresses, the influence of noise gradually weakens, and the policy gradually stabilizes.

[0038] In a preferred embodiment, the specific methods for updating the network include: Each agent samples a mini-batch of Z samples from the experience pool, where the g-th (g=1,…,Z) sample contains the observations of all N agents. Action value Shared rewards and new observations The aforementioned centralized sampling mechanism enables each agent to evaluate the effectiveness of joint actions using global information when updating its own value network, thereby effectively mitigating the training instability problem caused by environmental non-stationarity.

[0039] By minimizing the loss function Update value network parameters , where the target value ,in Represents all input variables of the value network. Z is the reward discount factor and Z is the number of mini-batch samples. This loss function gradually improves the accuracy of value assessment by minimizing the temporal difference error between the current value network output and the target value.

[0040] Through policy gradient Update strategy network parameters The policy gradient transmits the gradient information of the value network to the policy network through the chain rule, guiding the policy parameters to be updated in the direction of increasing value.

[0041] And replace the target value network with the updated parameters. and target policy network The parameters are periodically copied to the target network through soft or hard updates, ensuring the stability of the training objective and avoiding convergence difficulties caused by drastic fluctuations in network parameters.

[0042] This embodiment employs the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm as the training framework for the policy networks and value networks of each agent. Within this framework, each agent possesses an independent policy network and value network. During training, each agent's value network acquires global information (i.e., the observations and actions of all agents) for value evaluation. This allows each agent to perceive the impact of policy changes of other agents on the joint action utility when updating its own policy, effectively mitigating the environmental non-stationarity problem caused by the simultaneous evolution of policies among agents in a multi-agent environment. After training, each agent can output decision actions through its policy network during the execution phase based solely on its own local observations, without relying on information from other agents or communicating with them.

[0043] In this embodiment, the training of the MADDPG-based multi-agent policy network was conducted on a computer equipped with an Intel Core i7-8700 @ 3.2 GHz CPU and 16 GB RAM. The required development packages for the programming environment were mainly Python 3.6 and TensorFlow 2.0. The hyperparameters of the MADDPG algorithm were set as follows: 5000 training segments, 256 training batches, and a learning rate of 10. -4 The memory pool capacity is 10,000, and the reward discount factor is 0.99.

[0044] In this scheduling problem, it is assumed that some workpieces arrive at an initial time, followed by other workpieces arriving continuously at certain time intervals as new workpieces. Each workpiece is generated randomly from three types: shafts (process: turning-milling), flanges (process: turning-milling-drilling), and supports (process: milling-turning-drilling). The processing times for turning, milling, and drilling are assumed to follow a uniform distribution, with ranges of [5 min, 20 min], [10 min, 30 min], and [5 min, 15 min], respectively. Furthermore, several production environment configuration parameters are set: the total number of workpieces is 50; the average arrival interval between new workpieces is {5, 10, 15}; the urgency of workpiece delivery is {1, 2, 3}; and the machine failure rate is {3%, 5%, 7%}. By setting different values ​​for these parameters, 27 different environmental conditions can be constructed to test the versatility of the scheduling method.

[0045] Figure 8 and Figure 9This figure illustrates the cumulative reward and average delay time of each training segment when training multiple agents using MADDPG under a set of environmental configuration parameters (average arrival interval of 5, job delivery urgency of 2, and machine failure rate of 5%). The blue curve represents the original values, while the red curve represents the results after applying a moving average to the original values. The figure shows that as the number of training iterations increases, the cumulative reward gradually rises, while the average delay gradually decreases and eventually stabilizes, indicating that the agents have learned an effective scheduling strategy and the network training has converged.

[0046] The combination of the three machine selection rules and the three workpiece sorting rules designed in this embodiment can form nine composite scheduling rules, which are respectively denoted as: SMPT-SPT, SMPT-EDD, SMPT-MDD, NINQ-SPT, NINQ-EDD, NINQ-MDD, WINQ-SPT, WINQ-EDD, and WINQ-MDD. Figure 10 This paper presents the statistical ranking of the proposed MADDPG-based method and the nine scheduling rules across 27 production environment configurations. The ranking percentage refers to the ratio of the number of times the method achieved a certain ranking across the 27 configurations to the total number of times 27 is used. It can be observed that the proposed method achieved optimal performance in over 90% of the production environment configurations and ranked second in the remaining configurations. Conversely, only four of the nine scheduling rules achieved optimal performance in any one configuration, with each accounting for less than 10%. Furthermore, the rankings of these scheduling rules fluctuated significantly across different configurations. This indicates that, compared to these single scheduling rules, the present invention achieved optimal performance in over 90% of the production environment configurations and exhibits more stable performance across various production environment configurations, verifying the good versatility of the invention under machine failure environments.

[0047] Second Embodiment Please see Figure 7 This embodiment proposes a machine shop dynamic scheduling system based on multi-agent deep deterministic policy gradient, used to implement the machine shop dynamic scheduling method as described in Embodiment 1. The system includes: The agent construction module is used to construct the warehouse, machine and buffer as warehouse agents, machine agents and buffer agents respectively, and establish a Markov game process covering machine selection and workpiece sorting. The feature and space design module is used to design machine state features and machine selection weight action space for warehouse agents and machine agents, design workpiece state features and workpiece sorting weight action space for buffer agents, and set a shared reward function for all agents. The decision interaction module is configured in each intelligent agent to perceive the observed value at the current scheduling moment, output the rule weight through the policy network, and then apply it to the environment after weighted aggregation with multiple single scheduling rules to make the workshop move to the next state and get a reward. The current observed value, rule weight, reward and new observed value are stored as experience tuples in the experience pool. The network update module is used to sample experience tuples from the experience pool, update the value network and policy network of each agent using the sampled data, and iterate cyclically until the policy converges, thereby realizing dynamic scheduling of the machining workshop under machine failure disturbances.

[0048] During the operation of the above system, the agent instances output by the agent construction module are respectively transmitted to the feature and space design module and the decision interaction module. The feature and space design module configures the corresponding state feature dimensions and action space definitions for each agent according to its type, and feeds back the configuration results to the decision interaction module as the basis for setting the input and output layers of the policy network. At each scheduling decision moment, the decision interaction module receives real-time state data from the machining workshop, extracts features to form the observation values ​​of each agent, calls the policy network for forward inference to obtain rule weights, generates scheduling instructions after weighted aggregation, and sends them to the workshop execution equipment. At the same time, the experience tuples generated by the current decision are stored in the experience pool. The network update module samples batch experience tuples from the experience pool, uses the sampled data to perform backpropagation and parameter updates of the value network and policy network, and transmits the updated network parameters to the decision interaction module for use in forward inference at subsequent scheduling decision moments. The above four modules are connected in series, with the output of the previous module serving as the input of the next module, forming a data closed loop, and jointly realizing the continuous adaptive optimization of machining workshop scheduling under machine failure conditions.

[0049] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0050] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0051] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for dynamic scheduling of a machine shop based on multi-agent deep deterministic policy gradient, characterized in that the method comprises the steps of include: The warehouse, machines, and buffer zones are constructed as warehouse intelligent agents, machine intelligent agents, and buffer intelligent agents, respectively, and a Markov game process covering machine selection and workpiece sorting is established. Design machine state characteristics and machine selection weight action space for warehouse agents and machine agents, design workpiece state characteristics and workpiece sorting weight action space for buffer agents, and set a shared reward function for all agents. Each agent perceives the observed value at the current scheduling moment, outputs the rule weight through the policy network, and applies it to the environment after being weighted and aggregated with multiple single scheduling rules, so that the workshop can be transferred to the next state and receive a reward. Then, the observed value, weight, reward and new observed value are stored in the experience pool as experience tuples. Each agent samples the experience tuples from the experience pool, updates the network using the sampled data, and iterates until convergence, thereby achieving dynamic scheduling of the machining workshop under machine fault disturbances.

2. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, characterized in that, The types of scheduling decision points experienced by the Markov game process include: The arrival time of the initial workpiece, the arrival time of the new workpiece, the completion time of the process, and the time of machine failure; The warehouse intelligence agent and the machine intelligence agent are responsible for machine selection and scheduling decisions. The warehouse intelligence agent selects the processing machine for the first process of the workpiece, and the machine intelligence agent selects the processing machine for the subsequent processes of the workpiece. When a machine failure occurs, the machine intelligence agent corresponding to the failed machine selects an available processing machine for the affected workpiece. The buffer intelligence agent is responsible for workpiece sorting and scheduling decisions.

3. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, wherein, The machine status characteristics include: average machine utilization, the ratio of the standard deviation of machine utilization to the average value, the difference between the maximum and minimum values ​​of machine utilization, the percentage of busy machines, the ratio of the standard deviation of machine load to the average value, and the ratio of the maximum and average values ​​of machine load. The workpiece status characteristics include: the number of workpieces in the system, the proportion of delayed workpieces, the average number of compatible machines for workpieces to be processed, the average processing time of workpieces to be processed, the minimum processing time of workpieces to be processed, the average remaining processing time of workpieces to be processed, the minimum remaining processing time of workpieces to be processed, the minimum relaxation time of workpieces to be processed, the average relaxation time of workpieces to be processed, the maximum delay time of workpieces to be processed, the average delay time of workpieces to be processed, the average critical ratio of workpieces to be processed, the average interval between the two most recent arriving workpieces, and the average interval between the five most recent arriving workpieces.

4. The machining workshop dynamic scheduling method based on multi-agent deep deterministic policy gradient as described in claim 1, characterized in that, The multiple single scheduling rules corresponding to the machine selection weight action space include: minimum machine processing time priority rule, minimum machine load priority rule, and minimum buffer processing queue priority rule; The multiple single scheduling rules corresponding to the workpiece sorting weight action space include: minimum processing time priority rule, shortest delivery time priority rule, and shortest modification delivery time priority rule.

5. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, wherein, The shared reward function is expressed as follows: ,in and These represent the current state of the workshop. and the next state The average delay time value.

6. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, wherein, The policy network consists of a multi-layer fully connected neural network. The number of neurons in the input layer corresponds to the dimension of the agent's observation features, and the number of neurons in the output layer corresponds to the dimension of the action space. The warehouse agent and the machine agent use the same policy network structure.

7. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, wherein, Each agent adds exploration noise when outputting the rule weights to form new weight values where is the policy network of the hth agent, is the observation of the hth agent.

8. The multi-agent deep deterministic policy gradient based job shop dynamic scheduling method according to claim 1, wherein, The specific methods for updating the network include: Each agent samples a mini-batch of Z samples from the experience pool, where the g-th (g=1,…,Z) sample contains the observations of all N agents. Action value Shared rewards and new observations ; By minimizing the loss function Update value network parameters ,in Represents all input variables of the value network, and the target value. , Z represents the reward discount factor, and Z represents the number of samples in the mini-batch. By policy gradient Updating policy network parameters ; and replacing parameters of the target value network with the updated parameters and the target policy network .

9. A machine shop dynamic scheduling system based on multi-agent deep deterministic policy gradient, used to implement the machine shop dynamic scheduling method as described in any one of claims 1-8, characterized in that the system... include: The agent construction module is used to construct the warehouse, machine and buffer as warehouse agents, machine agents and buffer agents respectively, and establish a Markov game process covering machine selection and workpiece sorting. The feature and space design module is used to design machine state features and machine selection weight action space for warehouse agents and machine agents, design workpiece state features and workpiece sorting weight action space for buffer agents, and set a shared reward function for all agents. The decision interaction module is configured in each intelligent agent to perceive the observed value at the current scheduling moment, output the rule weight through the policy network, and then apply it to the environment after weighted aggregation with multiple single scheduling rules to make the workshop move to the next state and get a reward. The current observed value, rule weight, reward and new observed value are stored as experience tuples in the experience pool. The network update module is used to sample the experience tuples from the experience pool, update the value network and policy network of each agent using the sampled data, and iterate cyclically until the policy converges, thereby realizing dynamic scheduling of the machining workshop under machine fault disturbances.

Images