Multi-agent based dynamic regulation method, system and model for flexible processing unit

Abstract

The present application relates to the technical field of intelligent manufacturing and intelligent scheduling, and particularly relates to a flexible processing unit dynamic regulation method, system and model based on multiple agents. The method comprises the following steps: S1: establishing an event-driven Markov decision process model; S2: constructing corresponding task selection agents and workpiece selection agents to form a hierarchical multi-agent deep reinforcement learning architecture; S3: designing the system state features, action space and reward function of the task selection agents and the workpiece selection agents; S4: jointly training the task selection agents and the workpiece selection agents based on a multi-agent proximal policy optimization training algorithm to obtain optimal policy parameters of the task selection agents and the workpiece selection agents; and S5: the task selection agents and the workpiece selection agents output task actions and workpiece actions respectively based on the optimal policy parameters obtained through training. The present application can improve the adaptability of the system to dynamic disturbances and the real-time decision-making capability.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing and intelligent scheduling technology, specifically to a method, system, and model for dynamic control of flexible processing units based on multiple agents. Background Technology

[0002] As the manufacturing industry continues to develop towards flexibility, digitalization, and intelligence, flexible processing units, consisting of robots, CNC equipment, buffer stations, and material handling units, have been widely used in machining, electronic assembly, and precision manufacturing. These processing units typically need to coordinate a series of tasks, including workpiece loading and unloading, process processing, inter-process transfer, and finished product output, under limited equipment resources. Their scheduling level directly affects production efficiency, equipment utilization, and order delivery capabilities.

[0003] In flexible manufacturing units, production scheduling involves not only the sequencing of workpieces across multiple machines but also the processes of robot pick-up, handling, and delivery of workpieces. Particularly in scenarios involving dual-gripper robots, the robot's transport process is deeply coupled with the machine's processing process. Whether a process can start in a timely manner depends not only on the machine's state but also on factors such as the robot's position, the gripper's occupancy status, the transport path, and the loading / unloading sequence. When new workpieces arrive dynamically, the system state and resource competition relationships continuously change, making this type of scheduling problem exhibit characteristics such as high dimensionality, strong coupling, and significant dynamic disturbances.

[0004] Currently, research on the dynamic scheduling problem of flexible processing units mainly employs heuristic scheduling rules, mathematical programming methods, and metaheuristic methods such as genetic algorithms, particle swarm optimization, and ant colony optimization. While these methods are effective in static or small-scale scenarios, they still have significant shortcomings in dynamic and complex environments. Traditional scheduling rules, although computationally fast, largely rely on manual experience for design, limiting their adaptability and making it difficult to consider global optimization objectives under different operating states. Mathematical programming methods typically require significant problem simplification, and their solution complexity increases rapidly with increasing problem size or frequent disturbances, making them unsuitable for dynamic real-time scheduling. Heuristic and metaheuristic algorithms often require repeated searches or re-optimization when faced with dynamic disturbances such as the random arrival of new workpieces, resulting in limited real-time response capabilities.

[0005] Deep reinforcement learning (DRM) can autonomously learn decision-making strategies through interaction with the environment, providing a new technical approach to solving complex dynamic scheduling problems. However, the applicant found that directly applying single-agent DRM to the dynamic scheduling problem of flexible processing units still faces challenges such as a large action space, low training efficiency, and difficulty in policy convergence, especially in handling coupled decision-making processes such as transportation task selection and service job selection simultaneously. Furthermore, in the dynamic scheduling scenario of flexible processing units, the selection of transportation task types and specific service jobs exhibits clear hierarchical characteristics and a collaborative relationship. Compared to single-agent DRM, multi-agent DRM can decompose and model complex decision-making processes through the division of labor and cooperation among multiple agents, making it more suitable for handling this multi-stage, strongly coupled dynamic scheduling problem.

[0006] In summary, existing methods still suffer from insufficient dynamic adaptability, low decision-making efficiency, and limited global optimization capabilities when facing dynamic environments involving dual-gripper robots, random workpiece arrival, and deep coupling of production and logistics. Therefore, there is an urgent need to propose an efficient dynamic production scheduling method for flexible machining units to achieve collaborative optimization of the production and logistics processes and improve scheduling performance under dynamic disturbance environments. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, the technical problem this invention aims to solve is: how to provide a dynamic control method, system, and model for flexible processing units based on multi-agent systems. This involves establishing a hierarchical multi-agent deep reinforcement learning architecture composed of task selection agents and workpiece selection agents, decomposing the high-dimensional, strongly coupled joint scheduling decision into two levels: transportation task type selection and specific service workpiece selection. Furthermore, a collaborative learning mechanism is constructed by combining state feature design, action space constraints, and a reward function based on the weighted delay change of the system's total estimated delay. The scheduling strategy is then jointly trained using a multi-agent proximal policy optimization algorithm, enabling the agents to quickly generate effective scheduling decisions in a dynamic environment where new workpieces arrive randomly and resource competition relationships continuously change. This improves the system's adaptability to dynamic disturbances and its real-time decision-making capabilities.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A dynamic control method for flexible processing units based on multi-agent deep reinforcement learning includes:

[0010] S1: Establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units;

[0011] S2: Based on the production logistics collaborative decision-making of flexible processing units, corresponding task selection agents and workpiece selection agents are constructed to form a hierarchical multi-agent deep reinforcement learning architecture.

[0012] S3: Based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, design the system state features, action space and reward function of the task selection agent and the workpiece selection agent, and generate a multi-agent network.

[0013] S4: The training algorithm based on multi-agent near-end policy optimization is used to jointly train the task selection agent and the workpiece selection agent in the multi-agent network to obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a cooperative scheduling strategy for the dynamic environment of random workpiece arrival.

[0014] S5: During the execution phase, the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal policy parameters obtained from training, thereby realizing dynamic production scheduling of flexible processing units.

[0015] Preferably, in step S1, the processing steps for establishing the event-driven Markov decision process model include:

[0016] S101: Construct a dynamic scheduling environment for flexible processing units that includes flexible processing units, workpiece sets, machine sets, process routes, and dynamic arrival mechanisms for new workpieces;

[0017] S102: Set the decision event type that triggers scheduling decisions based on the dynamic scheduling environment of flexible processing units;

[0018] S103: Based on the dynamic scheduling environment of flexible processing units and the types of decision events that trigger scheduling decisions, construct an event-driven Markov decision process model for the dynamic scheduling process of flexible processing units.

[0019] The formula for the event-driven Markov decision process model is expressed as follows:

[0020] ;

[0021] In the formula: For state space, For the action space, Let be the state transition probability. For the reward function, The discount factor is used to describe the global resource state of the system, the differences between candidate tasks, and the differences between candidate artifacts at the moment of decision-making. The action space is composed of task selection actions and artifact selection actions. The state transition is achieved by advancing to the next triggering event after the current action is executed. The reward function is used to measure the impact of the current decision on the future scheduling goal.

[0022] The optimization objective of the event-driven Markov decision process model is to minimize the total weighted delay of the system.

[0023] The formula is expressed as:

[0024] ;

[0025] In the formula: Indicates the total weighted average delay; Indicate the importance of different workpieces or orders; This indicates the completion time when the workpiece is finally delivered to the finished product export area; Indicates the delivery time limit for the workpiece.

[0026] Preferably, in step S2, the processing steps for establishing a hierarchical multi-agent deep reinforcement learning architecture include:

[0027] S201: Based on the production logistics collaborative decision-making of the flexible processing unit environment, a task selection agent and a workpiece selection agent are constructed, and a hierarchical joint decision-making mechanism is established to generate joint scheduling actions; among them, the task selection agent is used to determine whether the robot should prioritize the picking task or the delivery task based on the current system state; the workpiece selection agent is used to select the specific service workpiece from the corresponding candidate workpiece set after the task selection result is given.

[0028] S202: Establish a hierarchical constraint relationship between the task selection agent and the workpiece selection agent; wherein the task selection agent is located at the upper level, and its output results are used to limit the effective decision range of the workpiece selection agent; the workpiece selection agent is located at the lower level, and is used to make decisions among the candidate workpiece set determined by the task selection results.

[0029] S203: Design a shared environment interaction mechanism between the task selection agent and the workpiece selection agent; at the decision moment, the task selection agent outputs the task action, and the workpiece selection agent outputs the workpiece action from the constrained candidate workpiece set. The two form a joint scheduling action and act on the flexible processing unit environment.

[0030] S204: Based on the joint scheduling actions between the task selection agent and the workpiece selection agent, as well as the hierarchical constraint relationship and the shared environment interaction mechanism, a hierarchical multi-agent deep reinforcement learning architecture is formed.

[0031] Preferably, in step S201, the processing steps for generating the joint scheduling action between the task selection agent and the workpiece selection agent include:

[0032] S2011: Construct the Actor network and Critic network corresponding to the task selection agent, which are used to determine the type of transportation task to be executed first based on the current environment information of the flexible processing unit. The specific processing steps include:

[0033] (1) At the decision-making moment The task selection agent network (Actor network) selects agents based on input feature information. Output task action probability distribution:

[0034] ;

[0035] In the formula: Selecting an agent for a task at time Output task actions;

[0036] (2) The task selection agent Critic network evaluates the value of the state corresponding to the current input and outputs the result. ;

[0037] S2012: Construct the Actor network and Critic network corresponding to the workpiece selection agent, which are used to determine the specific service workpiece in the corresponding candidate workpiece set under the constraints of the transportation task type. The specific processing steps include:

[0038] (1) At the decision-making moment The workpiece selection intelligent agent Actor network selects based on input feature information. And the task actions output by the task selection agent. In the corresponding candidate workpiece set Probability distribution of workpiece actions in the output:

[0039] ;

[0040] In the formula: Select an intelligent agent for the workpiece at time. Output of workpiece motion; For the purpose of task action The determined set of candidate workpieces;

[0041] (2) The workpiece selection agent Critic network evaluates the state value corresponding to the current input and outputs: ;

[0042] S2013: Establish a hierarchical joint decision-making mechanism between the task selection agent and the workpiece selection agent, and combine the task actions output by the task selection agent with the workpiece actions output by the workpiece selection agent to generate joint scheduling actions.

[0043] The formula for joint scheduling actions is expressed as follows:

[0044] ;

[0045] The corresponding joint strategy is expressed as:

[0046] ;

[0047] After applying the joint scheduling action to the flexible processing unit environment, the system state transition is represented as follows:

[0048] .

[0049] Preferably, in step S3, the steps for processing the system state characteristics include:

[0050] S301: Extracting information to characterize the decision moment Global resource status characteristics of the overall operation of the flexible processing unit Global resource status characteristics This includes the number of idle grippers on the robot, the number of idle robots, the robot's current position, workpiece completion rate statistics, and order completion rate;

[0051] The formula is expressed as:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] In the formula, This represents the number of idle grippers on the robot. Indicates the total number of robot grippers; Indicates the first A robotic gripper at any time The occupancy status; This represents the number of idle machines. For the first The machine is at all times The occupancy status; For workpiece The completion rate; For workpiece At any moment Number of completed processes; For workpiece Total number of processes; This represents the average completion rate of the workpieces. For a moment Number of workpieces reached; The standard deviation of the workpiece completion rate; For order completion rate; This represents the number of completed orders. This represents the number of orders that have arrived.

[0059] S302: Extract candidate task state features to characterize the distribution of candidate tasks under different transportation task types at the current decision time. And candidate workpiece status features used to characterize differences in delivery time, transportation, waiting time, and processing of candidate workpieces. Candidate task state characteristics and candidate workpiece state characteristics Calculate the minimum, mean, maximum, or standard deviation of each feature separately;

[0060] The formula is expressed as:

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] In the formula: The percentage of candidate pickup tasks; This is a set of candidate pickup tasks; The percentage of candidate delivery tasks; A set of candidate delivery tasks; For candidate workpieces At any moment Relaxation time; For workpiece Delivery date; For workpiece At any moment The estimated completion time; For candidate workpieces At any moment The delivery time; This is the robot's current position. For workpiece The current location; For workpiece Current process The target machine location; For loading and unloading time; For workpiece The completion time of the previous process; For workpiece The emergency weighting coefficient;

[0069] S303: Configure global resource status characteristics Candidate task state characteristics and candidate workpiece state characteristics Merge the data to generate system state characteristics. ;

[0070] The formula is expressed as:

[0071] ;

[0072] S304: System state characteristics Perform normalization processing;

[0073] The normalized system state characteristics are represented as follows:

[0074] ;

[0075] In the formula: This represents the normalized system state feature vector; The mean of the state characteristics; The operational variance represents the state characteristics; To prevent constants with a denominator of zero.

[0076] Preferably, in step S3, the steps for processing the motion space include:

[0077] S311: Designing the action space of the task selection agent The task selection agent is responsible for choosing the priority transportation task between pickup and delivery tasks.

[0078] Action space The formula is expressed as:

[0079] ;

[0080] In the formula: To prioritize the pickup task; To prioritize delivery tasks;

[0081] S312: At the Decision Moment The Actor network, which selects tasks, outputs the policy probability distribution for each task action:

[0082] );

[0083] In the formula: ; Select input features for the agent after normalization of the task;

[0084] S313: Construct task action mask vectors based on the executability of candidate tasks in the current system. It also imposes constraints on unexecutable tasks; based on the policy distribution after mask constraints, it performs sampling or greedy selection to obtain the task selection agent at time step [time value missing]. Task Actions ;

[0085] S314: The workpiece selection agent uses a shared backbone network and a dual-output head structure for workpiece selection. The shared backbone network is used to extract the common hidden representation of the current state, and then selects the corresponding output head according to different task modes to calculate the weight vector. Then, candidate workpieces are calculated based on the weight vector. Based on the score, the candidate workpiece with the highest score is ultimately selected. ;

[0086] Specifically:

[0087] (1) Common hidden representation of the output of the shared backbone network for:

[0088] ;

[0089] In the formula: It is a shared feature extraction module consisting of two fully connected layers; Select the input features of the intelligent agent for the normalized workpiece;

[0090] (2) Activate the corresponding output head according to the task mode selection result and calculate the weight vector under the current task mode. ;

[0091] The formula is expressed as:

[0092] ;

[0093] In the formula: and These represent the weight matrix and bias vector of the output layer in the pickup task mode, respectively. and These represent the weight matrix and bias vector of the output layer in the delivery task mode, respectively.

[0094] (3) For each candidate workpiece Constructing eigenvectors And based on the weight vector Calculate candidate workpieces rating ;

[0095] The formula is expressed as:

[0096] ;

[0097] ;

[0098] In the formula: The ranking feature corresponding to the Maximum Weighted Delay Rule (MWT); The ranking feature corresponding to the maximum weighted delay rule (LWT); The ranking features corresponding to the shortest processing time rule (SPT); The ranking feature corresponding to the longest waiting time rule (LWT); The ranking features corresponding to the Minimum Transit Time (MTT) rule; The sorting features are those corresponding to the Minimum Remaining Time (MRT) rule.

[0099] S315: Construct task action mask vectors based on the executability of candidate tasks in the current system. Constraining unexecutable tasks; sampling or greedy selection based on the policy distribution after mask constraints to obtain the workpiece selection agent at time t. Workpiece movement ;

[0100] S316: Combine the task action of the task selection agent with the workpiece action output by the workpiece selection agent to form a joint scheduling action, as shown below:

[0101] ;

[0102] In the formula: This indicates the task action output by the task selection agent; This indicates the workpiece action output by the selected intelligent agent.

[0103] Preferably, in step S3, the processing steps for designing the reward function include:

[0104] S321: Calculate the workpiece Delivery time ;

[0105] The formula is expressed as:

[0106] ;

[0107] ;

[0108] );

[0109] In the formula: For workpiece Total processing time; For process Processing time on the target machine; respectively workpiece Total transportation and loading / unloading time; Raw material area; Finished product area; This is the delivery time factor;

[0110] S322: Place the workpiece Estimated completion time Defined as after its final processing step is completed, it is transported and unloaded by robots to the finished product area. The completion time for each workpiece; Add a virtual terminal process Its target location is And it does not include the processing procedure;

[0111] workpiece At the decision point of Represented as:

[0112] ;

[0113] In the formula: and Representing the workpieces respectively Actual and expected delivery Completion time;

[0114] S323: Calculate any incomplete process The estimated completion time is calculated using the following formula:

[0115] ;

[0116] ;

[0117] In the formula: For process Expected arrival time at the target machine; This refers to the earliest available time for the target machine; The estimated completion time for the virtual terminal process;

[0118] S324: Based on workpiece Estimated completion time and delivery date Calculation at the decision moment Total estimated weighted delay for all workpieces:

[0119] ;

[0120] ;

[0121] In the formula: For workpiece In Delay in time estimation; Weighted average delay for total forecast; For decision-making moments The number of workpieces that have arrived in the system but have not yet been delivered to the finished goods area;

[0122] S325: Construct an immediate reward function based on the change in the total estimated weighted delay of the system at adjacent decision times, and calculate the cumulative reward using the following formula:

[0123] ;

[0124] ;

[0125] In the formula: This is the feedback value for the improvement of the current joint scheduling action on the overall estimated weighted delay of the system; This refers to the cumulative reward value. Discount factor; The total number of decision steps within a scheduling cycle;

[0126] The task selection agent and the workpiece selection agent share the reward function and jointly optimize the total weighted delay target through collaborative learning.

[0127] Preferably, the processing steps of the training algorithm based on multi-agent proximal policy optimization include:

[0128] S401: Employs a training paradigm of centralized training and distributed execution, enabling the task selection agent and the workpiece selection agent to interact in a shared environment and form joint scheduling actions. ; Upon application to a flexible manufacturing cell environment, an immediate reward is obtained. and the state at the next moment And constitute a sample of decision-making experience. ;

[0129] S402: Based on the decision-making experience sample, calculate the cumulative discount return (reward-to-go) and normalize it to obtain the normalized cumulative discount return;

[0130] The formula is expressed as:

[0131] ;

[0132] ;

[0133] In the formula: For a moment Initial cumulative discount reward; This is the normalized cumulative discount return; This is the mean of the discounted returns for all samples; The standard deviation of the discounted returns for all samples; To prevent constants with a denominator of zero;

[0134] S403: Calculate and estimate the dominance of the task selection agent and the job selection agent respectively using the generalized dominance estimation method. and Then, normalization was performed to obtain...

[0135] ;

[0136] ;

[0137] ;

[0138] In the formula: For the Critic network under the strategy at time 10:00 State value estimation; This is a termination marker; The dominant attenuation coefficient; The mean of the odds across all samples; The standard deviation of the odds for all samples;

[0139] S404: The probability ratio is obtained by subtracting the logarithmic probabilities of the current policy and the old policy on the actions already performed.

[0140] The formula is expressed as:

[0141] ;

[0142] In the formula: Select an agent for the task and the current strategy for selecting an agent for the workpiece; The old strategy of selecting agents for tasks and selecting agents for workpieces; This is the cutting factor;

[0143] S405: A training algorithm based on multi-agent proximal policy optimization is used to update the Actor network and Critic network of the task selection agent and the workpiece selection agent by combining the total loss function.

[0144] Total loss function The formula is expressed as:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] In the formula: , , These represent the agent's policy loss, value loss, and entropy regularization term, respectively. These are the weighting coefficients for the strategy loss term; This refers to the weighting coefficient for the value loss item; The weighting coefficients for the entropy regularization term; For policy entropy; This indicates the estimated value after trimming; This represents the current state value estimate of the agent output by the Critic network; This represents the state value estimate of the old Critic network; This represents a clipping function, used to clip variables. Limited to the range Inside; Indicates the cutting factor; This represents the expectation of all time-step samples in the current training batch; This represents the mean squared error loss function;

[0151] S406: Repeat steps S401 to S405 until the training reaches the preset number of iterations or the policy converges, and obtain the optimal policy parameters of the task selection agent and the workpiece selection agent.

[0152] A flexible processing unit dynamic control system based on multi-agent deep reinforcement learning includes:

[0153] The MDP model building module is used to establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units.

[0154] The agent construction module is used to construct corresponding task selection agents and workpiece selection agents based on the production logistics collaborative decision-making of flexible processing units, forming a hierarchical multi-agent deep reinforcement learning architecture.

[0155] The agent design module, based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, designs the system state characteristics, action space and reward function of the task selection agent and the workpiece selection agent.

[0156] The model training module is used to jointly train the task selection agent and the workpiece selection agent based on the training algorithm of multi-agent proximal policy optimization, obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a cooperative scheduling strategy for the dynamic environment of random workpiece arrival.

[0157] The dynamic production scheduling module is used to achieve dynamic production scheduling of flexible processing units by having the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal strategy parameters obtained from training during the execution phase.

[0158] A dynamic control model for flexible processing units based on multi-agent deep reinforcement learning includes: one or more processors;

[0159] The processor is used to store one or more programs;

[0160] When the one or more programs are executed by the one or more processors, the method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in any one of claims 1 to 8 is implemented.

[0161] Compared with existing technologies, the dynamic control method, system, and model of flexible processing units based on multi-agent systems in this invention have the following advantages:

[0162] This invention addresses the challenges of deep coupling between production and logistics processes in the context of dual-gripper robots and the continuous changes in system state caused by random workpiece arrivals. It constructs an event-driven Markov decision process model, which uses key events such as machine processing completion, robot transportation completion, and the arrival of new workpieces as decision triggers. This model more accurately represents the evolution of system state in dynamic environments, improving the adaptability of scheduling methods to complex dynamic scenarios. Simultaneously, a hierarchical multi-agent deep reinforcement learning architecture is constructed, decomposing the original high-dimensional, strongly coupled joint scheduling decision into two levels: transportation task type selection and specific service workpiece selection. Through the division of labor and cooperation between task selection agents and workpiece selection agents, the action space scale and policy learning difficulty are reduced, overcoming the shortcomings of single-agent methods in complex dynamic scheduling problems, such as large decision dimensions, low training efficiency, and convergence difficulties.

[0163] This invention constructs a state feature system from three levels: global resource state, candidate task state, and candidate workpiece state, and improves training stability through normalization processing. It designs corresponding action spaces for each of the two agents and combines candidate task and candidate workpiece constraint mechanisms to reduce invalid action searches, improve decision-making efficiency, and enhance the feasibility of scheduling results. Furthermore, it constructs a reward function based on the weighted change in the system's total estimated delay, effectively linking local transportation behavior with the global delivery time optimization objective. This enables the agents to collaboratively learn around minimizing the total weighted delay, thereby improving overall scheduling performance.

[0164] This invention employs a multi-agent proximal policy optimization algorithm to jointly train two agents, combining centralized training with distributed execution mechanisms to enhance collaborative decision-making capabilities among multiple agents while ensuring training stability. This method can rapidly generate effective scheduling strategies under dynamic perturbation environments, reducing the difficulty of solving complex dynamic scheduling problems, improving the collaborative optimization capabilities of robot transportation and machining processes, and effectively improving the total weighted delay and overall operating efficiency of flexible machining units. Attached Figure Description

[0165] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0166] Figure 1 This is a logic block diagram of a dynamic control method for flexible processing units based on multi-agent deep reinforcement learning.

[0167] Figure 2 This is a schematic diagram of a hierarchical multi-agent deep reinforcement learning architecture.

[0168] Figure 3 This is a network structure diagram of a multi-agent network. Detailed Implementation

[0169] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0170] The following detailed explanation illustrates the specific implementation methods:

[0171] Example 1:

[0172] This embodiment discloses a dynamic control method for flexible processing units based on multi-agent deep reinforcement learning.

[0173] like Figure 1 As shown, the dynamic control method for flexible processing units based on multi-agent deep reinforcement learning includes:

[0174] S1: Establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units;

[0175] S2: Based on the production logistics collaborative decision-making of flexible processing units, corresponding task selection agents and workpiece selection agents are constructed to form a hierarchical multi-agent deep reinforcement learning architecture.

[0176] S3: Based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, design the system state features, action space and reward function of the task selection agent and the workpiece selection agent, and generate a multi-agent network.

[0177] S4: The training algorithm based on multi-agent near-end policy optimization is used to jointly train the task selection agent and the workpiece selection agent in the multi-agent network to obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a cooperative scheduling strategy for the dynamic environment of random workpiece arrival.

[0178] S5: During the execution phase, the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal policy parameters obtained from training, thereby realizing dynamic production scheduling of flexible processing units.

[0179] During the scheduling and execution phase, an online decision is triggered when events such as machine completion of processing, robot completion of transportation, or arrival of a new workpiece occur, and at least one executable robot task exists in the system. At this time, the task selection agent outputs task actions based on the current environmental state, and the workpiece selection agent outputs workpiece actions under the constraints of the task actions. Together, they form a joint scheduling action and drive the flexible processing unit to execute the scheduled task. This method enables the rapid generation of effective scheduling decisions in dynamic disturbance environments, achieving collaborative optimization of the robot transportation process and the machine processing process.

[0180] This invention addresses the dynamic scheduling problem involving a dual-gripper robot, where the production and logistics processes are deeply coupled, and the random arrival of workpieces leads to continuous changes in the system state. It constructs an event-driven Markov decision process model, using key events such as machine completion, robot transport completion, and the arrival of new workpieces as decision triggers to more accurately represent the system evolution process under dynamic conditions. Based on this, a hierarchical multi-agent deep reinforcement learning architecture is built, decomposing the high-dimensional, strongly coupled joint scheduling decision into two levels: transportation task type selection and specific service workpiece selection. The collaborative decision-making between the task selection agent and the workpiece selection agent reduces the difficulty of policy learning. Simultaneously, it considers the global resource state and candidate task states... A state feature system is constructed at three levels: the current state, the state of the candidate workpiece, and the state of the current workpiece. Normalization is used to improve training stability. Combined with action space design and candidate task and workpiece constraint mechanisms, invalid searches are reduced and decision feasibility is improved. Furthermore, a reward function based on the weighted delay change of the total estimated system time is constructed to unify local transportation decisions with the global delivery time optimization objective. Finally, a multi-agent proximal policy optimization algorithm is adopted and a centralized training and decentralized execution mechanism is combined to jointly train two agents. This enables the rapid generation of effective scheduling strategies in dynamic disturbance environments, improves the collaborative optimization capability of the robot transportation process and the machine processing process, and effectively improves the total weighted delay and overall operating efficiency of the flexible processing unit.

[0181] To better illustrate the technical solution of the present invention, this embodiment will be described in more detail through the following parts.

[0182] I. Event-Driven Markov Decision Process (MDP) Model

[0183] In specific implementation, the steps for establishing an event-driven Markov decision process model include:

[0184] S101: Construct a dynamic scheduling environment for flexible processing units that includes flexible processing units, workpiece sets, machine sets, process routes, and dynamic arrival mechanisms for new workpieces;

[0185] In this embodiment, the flexible processing unit includes a raw material area, a finished product exit area, several processing machines, and a dual-gripper transport robot. The raw material area stores workpieces to be processed, the finished product exit area receives workpieces after all processes are completed, the processing machines execute the processing tasks corresponding to each workpiece, and the dual-gripper transport robot handles the handling, loading, unloading, and transfer of workpieces between the raw material area, the processing machines, and the finished product exit area. This flexible processing unit simultaneously includes the production execution process and the material transportation process, which are coupled in terms of time, space, and resource constraints to jointly constitute a dynamic production scheduling environment.

[0186] At the initial moment, the robot and a set of initial workpieces are located in the raw material area, let the set of workpieces be denoted as . The machine set is For any workpiece Its processing consists of several steps, denoted as... ,in, Indicates workpiece The total number of processes. Each process must be executed strictly according to the predetermined process sequence, and each process corresponds to only one selectable processing machine. Let the processes be... The target machine is The processing time is Once a workpiece has completed a certain process, it is temporarily stored in the output buffer area of ​​the corresponding machine, waiting for the robot to perform subsequent pick-up, transfer, and delivery operations. After the workpiece completes the last process, it is transported by the robot to the finished product export area, thus completing the entire processing.

[0187] Robotic transportation operations can be divided into two categories: pickup tasks and delivery tasks. These two types of tasks correspond to the pickup and delivery phases in the robot's service process, respectively, and together constitute a complete transportation process of the workpiece within the robot unit. Their specific meanings are as follows:

[0188] (1) The task of picking up the workpiece refers to the robot moving from its current position to the location of the target workpiece without load, and completing the gripping and loading of the workpiece. The target workpiece may be located in the raw material area or in the buffer area of ​​the machine where the previous process was completed.

[0189] (2) Delivery task refers to the robot transporting the gripped workpiece to the target machine corresponding to the current process and completing the placement and unloading of the workpiece on the target machine.

[0190] Furthermore, to characterize the dynamic disturbances in a real production environment, this embodiment further considers the dynamic arrival of new workpieces. That is, during the scheduling process, in addition to the initial workpieces, new workpiece orders randomly enter the system, and their workpiece type, arrival time, and subsequent processing requirements change dynamically over time. After the new workpieces arrive, they enter the raw material area to await robot loading and transfer, thus competing with existing workpieces in the system for machine resources and robot transport resources. Because the insertion of new workpieces changes the current candidate task set, resource occupancy relationships, and the connection status of subsequent processes, the system state of the entire flexible processing unit is a continuously changing dynamic process.

[0191] S102: Set the decision event type that triggers scheduling decisions based on the dynamic scheduling environment of flexible processing units;

[0192] In this embodiment, the decision event types include the following three categories:

[0193] (1) Machine process completion event: When a machine completes the current process, the processed workpiece enters the output buffer area of ​​the corresponding machine. The system may add a new pick-up task, and a new decision needs to be made.

[0194] (2) Robot transportation completion event: After the robot completes a pickup or delivery operation, its current position, gripper occupancy status and candidate task set will change, so it is necessary to redetermine the next service object.

[0195] (3) New workpiece arrival event: When a new workpiece arrives dynamically in the raw material area, the system may add a new pick-up task, and the resource competition relationship changes accordingly, thus triggering a new decision.

[0196] In this system, decision points are no longer determined by fixed time intervals, but are triggered by key events. Let the event sequence be... In this case, the time interval between two adjacent decision points is usually not fixed, which is consistent with the characteristics of event-driven modeling. Compared with the fixed time step method, this decision point setting can more accurately reflect the actual operation process of asynchronous event-driven processes in the robot unit and effectively reduce the computational burden caused by redundant decisions.

[0197] S103: Based on the dynamic scheduling environment of flexible processing units and the types of decision events that trigger scheduling decisions, construct an event-driven Markov decision process model for the dynamic scheduling process of flexible processing units.

[0198] The formula for the event-driven Markov decision process model is expressed as follows:

[0199] ;

[0200] In the formula: For state space, For the action space, Let be the state transition probability. For the reward function, The discount factor is used to describe the global resource state of the system, the differences between candidate tasks, and the differences between candidate artifacts at the moment of decision-making. The action space is composed of task selection actions and artifact selection actions. The state transition is achieved by advancing to the next triggering event after the current action is executed. The reward function is used to measure the impact of the current decision on the future scheduling goal.

[0201] Based on the above event-driven mechanism, at any decision moment The scheduling process of a flexible machining cell can be described as a Markov decision process consisting of states, actions, state transitions, and rewards. This process represents the coupled evolution of machining execution, material transport, and the dynamic arrival of new workpieces within the flexible machining cell. The MDP (Multi-Device Process) triggers decisions only when a critical change in the system state occurs and the robot's service objects need to be reassigned.

[0202] In this embodiment, the optimization objective of the event-driven Markov decision process model is to minimize the total weighted delay of the system;

[0203] The formula is expressed as:

[0204] ;

[0205] In the formula: Indicates total weighted delay; weight Used to characterize the importance of different workpieces or orders; This indicates the completion time when the workpiece is finally delivered to the finished product export area; delivery date. This indicates the delivery time limit of the workpiece. By using the total weighted delay as the overall optimization objective, the scheduling strategy can balance prioritizing key workpieces with improving overall delivery performance in a dynamic environment.

[0206] To ensure the clarity of the model construction, the following assumptions are made in this embodiment: all initial workpieces, machines, and robots are available at the initial moment; the output buffer capacity of each machine is considered unlimited; robot and machine failures are not considered; the processing time of the workpiece on the designated machine and the transportation time between each location are known; the ongoing transportation and processing tasks are uninterrupted; each machine processes at most one workpiece at a time, and each gripper of the dual-gripper robot grasps at most one workpiece at a time; all workpiece processes must be strictly processed sequentially according to the predetermined process order. These assumptions enable the dynamic scheduling environment of the flexible processing unit to be clearly abstracted into a solvable event-driven Markov decision process model, providing a foundation for the subsequent construction of a hierarchical multi-agent reinforcement learning architecture.

[0207] II. Hierarchical Multi-Agent Deep Reinforcement Learning Architecture

[0208] In the specific implementation process, such as Figure 2 As shown, the processing steps for establishing a hierarchical multi-agent deep reinforcement learning architecture include:

[0209] S201: Based on the production logistics collaborative decision-making of the flexible processing unit environment, a task selection agent and a workpiece selection agent are constructed, and a hierarchical joint decision-making mechanism is established to generate joint scheduling actions; among them, the task selection agent is used to determine whether the robot should prioritize the picking task or the delivery task based on the current system state; the workpiece selection agent is used to select the specific service workpiece from the corresponding candidate workpiece set after the task selection result is given.

[0210] In this embodiment, both the task selection agent and the workpiece selection agent are constructed using an Actor-Critic structure. The task selection agent mainly receives global resource state features and candidate task state features, and outputs the task action probability distribution. This is used to determine the current transportation task type; the workpiece selection agent mainly receives global resource state features and candidate workpiece state features, and outputs the workpiece action probability distribution. This is used to determine the specific service artifact. Thus, the task selection agent first outputs the task action, and the artifact selection agent then outputs the artifact action. Together, they form a joint scheduling action, thereby realizing a hierarchical expression of the "select task first, then select artifact" scheduling logic.

[0211] S202: Establish a hierarchical constraint relationship between the task selection agent and the workpiece selection agent; wherein the task selection agent is located at the upper level, and its output results are used to limit the effective decision range of the workpiece selection agent; the workpiece selection agent is located at the lower level, and is used to make decisions among the candidate workpiece set determined by the task selection results.

[0212] In this embodiment, the task selection agent and the workpiece selection agent are coupled through a hierarchical constraint relationship. Specifically, the task selection agent is located at the upper level, and its output results are used to limit the effective decision range of the workpiece selection agent; the workpiece selection agent is located at the lower level and can only make decisions from the candidate workpiece set determined by the task selection results. The action space of the workpiece selection agent changes dynamically with the task selection results. When the task selection result is to prioritize the retrieval task, the system generates a candidate workpiece set in the retrieval mode; when the task selection result is to prioritize the delivery task, the system generates a candidate workpiece set in the delivery mode. This forms a serial decision chain in which the upper-level task actions guide the lower-level workpiece actions, which ensures the consistency of decision logic and reduces invalid searches.

[0213] S203: Design a shared environment interaction mechanism between the task selection agent and the workpiece selection agent; at the decision moment, the task selection agent outputs the task action, and the workpiece selection agent outputs the workpiece action from the constrained candidate workpiece set. The two form a joint scheduling action and act on the flexible processing unit environment.

[0214] In this embodiment, the task selection agent and the workpiece selection agent achieve collaborative interaction by sharing the same flexible machining unit environment. When there is at least one executable robot task in the system, this moment is defined as the decision moment. At the decision moment, the task selection agent first outputs the task action, and the workpiece selection agent subsequently outputs the workpiece action from the constrained candidate workpiece set. The two agents form a joint scheduling action that acts on the flexible machining unit environment. After the environment executes this joint scheduling action, the system transitions to the next state and feeds back a reward signal for joint training of the two agents. Furthermore, this embodiment employs a centralized training and distributed execution mechanism. During the training phase, collaborative strategies are learned through shared environment interaction information and reward feedback. During the execution phase, task actions and workpiece actions are output according to a predetermined hierarchical order, thereby achieving rapid joint scheduling decisions in a dynamic environment.

[0215] S204: Based on the joint scheduling actions between the task selection agent and the workpiece selection agent, as well as the hierarchical constraint relationship and the shared environment interaction mechanism, a hierarchical multi-agent deep reinforcement learning architecture is formed.

[0216] Specifically, the processing steps for the joint scheduling action between the task selection agent and the workpiece selection agent include:

[0217] S2011: Construct the Actor network and Critic network corresponding to the task selection agent, which are used to determine the type of transportation task to be executed first based on the current environment information of the flexible processing unit. The specific processing steps include:

[0218] (1) At the decision-making moment The task selection agent network (Actor network) selects agents based on input feature information. Output task action probability distribution:

[0219] ;

[0220] In the formula: Selecting an agent for a task at time Output task actions;

[0221] (2) The task selection agent Critic network evaluates the value of the state corresponding to the current input and outputs the result. ;

[0222] S2012: Construct the Actor network and Critic network corresponding to the workpiece selection agent, which are used to determine the specific service workpiece in the corresponding candidate workpiece set under the constraints of the transportation task type. The specific processing steps include:

[0223] (1) At the decision-making moment The workpiece selection intelligent agent Actor network selects based on input feature information. And the task actions output by the task selection agent. In the corresponding candidate workpiece set Probability distribution of workpiece actions in the output:

[0224] ;

[0225] In the formula: Select an intelligent agent for the workpiece at time. Output of workpiece motion; For the purpose of task action The determined set of candidate workpieces;

[0226] (2) The workpiece selection agent Critic network evaluates the state value corresponding to the current input and outputs: ;

[0227] S2013: Establish a hierarchical joint decision-making mechanism between the task selection agent and the workpiece selection agent, and combine the task actions output by the task selection agent with the workpiece actions output by the workpiece selection agent to generate joint scheduling actions.

[0228] The formula for joint scheduling actions is expressed as follows:

[0229] ;

[0230] The corresponding joint strategy is expressed as:

[0231] ;

[0232] After applying the joint scheduling action to the flexible processing unit environment, the system state transition is represented as follows:

[0233] .

[0234] III. System State Characteristics

[0235] Combination Figure 3 As shown, the system state characteristics are designed through the following steps:

[0236] In this embodiment, to enable the intelligent agent to fully characterize the operating state of the flexible processing unit in a dynamic environment, state features are extracted from three levels: global resource state, candidate task differences, and candidate workpiece differences, constructing a hierarchical state representation. Specifically, the system state features consist of global state features. Candidate task state characteristics and candidate workpiece state characteristics The composition can be represented as: .

[0237] like Figure 3 As shown, { , } serves as input to the task selection agent, supporting its determination of the transportation task type; , As input to the workpiece selection agent, it supports the agent in further determining the specific service workpiece under a given task type.

[0238] Specifically, the steps for processing the state characteristics of the design system include:

[0239] S301: Extracting information to characterize the decision moment Global resource status characteristics of the overall operation of the flexible processing unit Global resource status characteristics This includes the number of idle grippers on the robot, the number of idle robots, the robot's current position, workpiece completion rate statistics, and order completion rate;

[0240] The formula is expressed as:

[0241] ;

[0242] ;

[0243] ;

[0244] ;

[0245] ;

[0246] ;

[0247] In the formula, This represents the number of idle grippers on the robot. Indicates the total number of robot grippers; Indicates the first A robotic gripper at any time The occupancy status; This represents the number of idle machines. For the first The machine is at all times The occupancy status; For workpiece The completion rate; For workpiece At any moment Number of completed processes; For workpiece Total number of processes; This represents the average completion rate of the workpieces. For a moment Number of workpieces reached; The standard deviation of the workpiece completion rate; For order completion rate; This represents the number of completed orders. This represents the number of orders that have arrived.

[0248] S302: Extract candidate task state features to characterize the distribution of candidate tasks under different transportation task types at the current decision time. And candidate workpiece status features used to characterize differences in delivery time, transportation, waiting time, and processing of candidate workpieces. Candidate task state characteristics and candidate workpiece state characteristics Calculate the minimum, mean, maximum, or standard deviation of each feature separately;

[0249] The formula is expressed as:

[0250] ;

[0251] ;

[0252] ;

[0253] ;

[0254] ;

[0255] ;

[0256] ;

[0257] In the formula: The percentage of candidate pickup tasks; This is a set of candidate pickup tasks; The percentage of candidate delivery tasks; A set of candidate delivery tasks; For candidate workpieces At any moment Relaxation time; For workpiece Delivery date; For workpiece At any moment The estimated completion time; For candidate workpieces At any moment The delivery time; This is the robot's current position. For workpiece The current location; For workpiece Current process The target machine location; For loading and unloading time; For workpiece The completion time of the previous process; For workpiece The emergency weighting coefficient;

[0258] S303: Configure global resource status characteristics Candidate task state characteristics and candidate workpiece state characteristics Merge the data to generate system state characteristics. ;

[0259] The formula is expressed as:

[0260] ;

[0261] S304: System state characteristics Normalization is performed to reduce the impact of state variables with different dimensions on the agent's policy learning;

[0262] The normalized system state characteristics are represented as follows:

[0263] ;

[0264] In the formula: This represents the normalized system state feature vector; The mean of the state characteristics; The operational variance represents the state characteristics; To prevent constants with a denominator of zero.

[0265] IV. Movement Space

[0266] In this embodiment, action spaces corresponding to the decision-making tasks of the task selection agent and the workpiece selection agent are designed respectively. The task selection agent is responsible for selecting the preferred transportation task type between the pickup task and the delivery task, and its action space is represented as follows: .

[0267] Specifically, the steps for designing motion space processing include:

[0268] S311: Designing the action space of the task selection agent The task selection agent is responsible for choosing the priority transportation task between pickup and delivery tasks.

[0269] Action space The formula is expressed as:

[0270] ;

[0271] In the formula: To prioritize the pickup task, To prioritize delivery tasks;

[0272] S312: At the Decision Moment The Actor network, which selects tasks, outputs the policy probability distribution for each task action:

[0273] );

[0274] In the formula: ; Select input features for the agent after normalization of the task;

[0275] S313: Construct task action mask vectors based on the executability of candidate tasks in the current system. It also imposes constraints on unexecutable tasks; based on the policy distribution after mask constraints, it performs sampling or greedy selection to obtain the task selection agent at time step [time value missing]. Task Actions This method can compress the action space and reduce invalid searches.

[0276] After determining the task selection result, the system further identifies a set of candidate tasks for the current task mode. The task selection agent then chooses the final execution target from this set. To simultaneously represent common information and differentiated decision preferences across different task modes, the task selection agent employs a shared backbone network and a dual-output head structure for task selection.

[0277] S314: The workpiece selection agent uses a shared backbone network and a dual-output head structure for workpiece selection. The shared backbone network is used to extract the common hidden representation of the current state, and then selects the corresponding output head according to different task modes to calculate the weight vector. Then, candidate workpieces are calculated based on the weight vector. Based on the score, the candidate workpiece with the highest score is ultimately selected. ;

[0278] Specifically:

[0279] (1) Common hidden representation of the output of the shared backbone network for:

[0280] ;

[0281] In the formula: It is a shared feature extraction module consisting of two fully connected layers; Select the input features of the intelligent agent for the normalized workpiece;

[0282] (2) Activate the corresponding output head according to the task mode selection result and calculate the weight vector under the current task mode. ;

[0283] The formula is expressed as:

[0284] ;

[0285] In the formula: and These represent the weight matrix and bias vector of the output layer in the pickup task mode, respectively. and These represent the weight matrix and bias vector of the output layer in the delivery task mode, respectively.

[0286] (3) For each candidate workpiece Constructing eigenvectors And based on the weight vector Calculate candidate workpieces rating ;

[0287] The formula is expressed as:

[0288] ;

[0289] ;

[0290] In the formula: The ranking feature corresponding to the Maximum Weighted Delay Rule (MWT); The ranking feature corresponding to the maximum weighted delay rule (LWT); The ranking features corresponding to the shortest processing time rule (SPT); The ranking feature corresponding to the longest waiting time rule (LWT); The ranking features corresponding to the Minimum Transit Time (MTT) rule; The sorting features are those corresponding to the Minimum Remaining Time (MRT) rule.

[0291] S315: Construct task action mask vectors based on the executability of candidate tasks in the current system. Constraining unexecutable tasks; sampling or greedy selection based on the policy distribution after mask constraints to obtain the workpiece selection agent at time t. Workpiece movement ;

[0292] S316: Combine the task action of the task selection agent with the workpiece action output by the workpiece selection agent to form a joint scheduling action, as shown below:

[0293] ;

[0294] In the formula: This indicates the task action output by the task selection agent; This indicates the workpiece action output by the selected intelligent agent.

[0295] V. Reward Function

[0296] In this embodiment, to achieve the optimization objective of minimizing the total weighted delay, a reward function is constructed based on the change in the system's total estimated weighted delay. Specifically, the estimated completion time and estimated weighted delay of each workpiece at the current decision moment are first calculated. Then, an immediate reward is defined based on the change in the system's total estimated weighted delay to characterize the impact of the current joint scheduling action on future global performance. Through this reward design, the correlation between local transportation behavior and the global delivery target can be effectively established, guiding the two agents to collaboratively learn around the objective of minimizing the total weighted delay.

[0297] Among them, workpiece Delivery time It is determined based on its total processing time, total transportation and loading / unloading time, and delivery time coefficient, and is expressed as follows:

[0298] )

[0299] In the formula, Indicates workpiece Delivery time coefficient, Indicates workpiece Total processing time Indicates workpiece Total transportation and loading / unloading time.

[0300] In this embodiment, the workpiece is... Estimated completion time Defined as after its final processing step is completed, it is transported and unloaded by robots to the finished product area. The estimated completion time is determined by adding a virtual terminal process (without actual machining) at the end of each workpiece. For workpieces that are not yet completed, the estimated completion time is obtained by recursively extrapolating the estimated completion times of the current and subsequent processes; for workpieces already delivered to the finished goods area, their actual completion time is used directly as the estimated completion time. After obtaining the workpiece delivery date and estimated completion time, the workpiece... At the moment of decision The estimated weighted delay is expressed as:

[0301] ;

[0302] Then, regarding the decision moment The estimated weighted delay times of all workpieces that have arrived in the system but have not yet been delivered to the finished goods area are summed to obtain the total estimated weighted delay time of the system, expressed as:

[0303] ;

[0304] Based on this, an immediate reward function is constructed and the cumulative reward is calculated based on the change in the total estimated weighted delay of the system at adjacent decision times, expressed as:

[0305] ;

[0306] ;

[0307] In the formula, This is the feedback value for the improvement of the current joint scheduling action on the overall estimated weighted delay of the system; This refers to the cumulative reward value. As a discount factor, The total number of decision steps within a scheduling cycle; the task selection agent and the workpiece selection agent share the reward function and jointly optimize the total weighted delay target through collaborative learning.

[0308] Specifically, the steps involved in designing the reward function include:

[0309] S321: Calculate the workpiece Delivery time ;

[0310] The formula is expressed as:

[0311] ;

[0312] ;

[0313] );

[0314] In the formula: For workpiece Total processing time; For process Processing time on the target machine; respectively workpiece Total transportation and loading / unloading time; Raw material area; Finished product area; This is the delivery time factor; Indicates workpiece The location of the target machine in the first process; Indicates workpiece The total number of processes; Indicates the loading and unloading time of the workpiece;

[0315] S322: Place the workpiece Estimated completion time Defined as after its final processing step is completed, it is transported and unloaded by robots to the finished product area. The completion time for each workpiece; Add a virtual terminal process Its target location is And it does not include the processing procedure;

[0316] workpiece At the decision point of Represented as:

[0317] ;

[0318] In the formula: and Representing the workpieces respectively Actual and expected delivery Completion time;

[0319] S323: Calculate any incomplete process ( The estimated completion time is calculated using the following formula:

[0320] ;

[0321] ;

[0322] In the formula: For process The estimated completion time; For process Expected arrival time at the target machine; For the transportation time between machines in different locations; This refers to the earliest available time for the target machine; The estimated completion time for the virtual terminal process;

[0323] S324: Based on workpiece Estimated completion time and delivery date Calculation at the decision moment Total estimated weighted delay for all workpieces:

[0324] ;

[0325] ;

[0326] In the formula: For workpiece In Delay in time estimation; Indicates workpiece Importance weights; Weighted average delay for total forecast; For decision-making moments The number of workpieces that have arrived in the system but have not yet been delivered to the finished goods area;

[0327] S325: Construct an immediate reward function based on the change in the total estimated weighted delay of the system at adjacent decision times, and calculate the cumulative reward using the following formula:

[0328] ;

[0329] ;

[0330] In the formula: This is the feedback value for the improvement of the current joint scheduling action on the overall estimated weighted delay of the system; This refers to the cumulative reward value. Discount factor; The total number of decision steps within a scheduling cycle;

[0331] The task selection agent and the workpiece selection agent share the reward function and jointly optimize the total weighted delay target through collaborative learning.

[0332] VI. Training Algorithm Based on Multi-Agent Proximity Policy Optimization

[0333] This embodiment employs a training paradigm of centralized training and distributed execution, enabling the task selection agent and the workpiece selection agent to interact within a shared environment. At each decision-making moment, the task selection agent outputs a task action based on the current state, while the workpiece selection agent outputs a workpiece action under the constraints of the task action. Together, they form a joint scheduling action. After the joint scheduling action is applied to the flexible machining unit environment, the environment transitions to the next moment's state and provides an immediate reward. The current state, task action, workpiece action, immediate reward, and next moment's state constitute the decision-making experience sample. This is used for subsequent network parameter updates.

[0334] Specifically, the processing steps of the training algorithm based on multi-agent proximal policy optimization include:

[0335] S401: Employs a training paradigm of centralized training and distributed execution, enabling the task selection agent and the workpiece selection agent to interact in a shared environment and form joint scheduling actions. ; Upon application to a flexible manufacturing cell environment, an immediate reward is obtained. and the state at the next moment And constitute a sample of decision-making experience. ;

[0336] S402: Based on the decision-making experience samples collected in this embodiment, the immediate reward in each sample sequence is accumulated, and the cumulative discounted return from the current moment is calculated. This is used to characterize the long-term benefits of the current scheduling behavior. To reduce the impact of differences in numerical scales among different samples on the training process, this embodiment normalizes the cumulative discounted reward, keeping the reward values ​​at different decision stages within a relatively stable numerical range. This improves the numerical stability of the training process and provides a consistent training signal for subsequent advantage estimation and policy updates.

[0337] Based on the decision-making experience sample, the cumulative discount return (reward-to-go) is calculated and normalized to obtain the normalized cumulative discount return;

[0338] The formula is expressed as:

[0339] ;

[0340] ;

[0341] In the formula: For a moment Initial cumulative discount reward; This is the normalized cumulative discount return; Indicates the first The environment at any moment The instant reward received; This represents the discount factor, used to balance the importance of current rewards and future rewards; This is the mean of the discounted returns for all samples; The standard deviation of the discounted returns for all samples; To prevent constants with a denominator of zero;

[0342] S403: Calculate the dominance function for both the task selection agent and the workpiece selection agent. Specifically, first, calculate the time difference error based on the immediate reward, the next state value estimate, and the termination marker. Then, recursively calculate the dominance value using the generalized dominance estimation method. The algorithm is then normalized to improve the effectiveness and robustness of policy updates.

[0343] The generalized dominance estimation method is used to calculate and estimate the dominance of the task selection agent and the workpiece selection agent, respectively. and Then, normalization was performed to obtain...

[0344] ;

[0345] ;

[0346] ;

[0347] In the formula: For the agent in the first An environment, a moment The original advantage estimate; This is the normalized advantage estimate; For the Critic network under the strategy at time 10:00 State value estimation; This is a termination marker; The dominant attenuation coefficient; The mean of the odds across all samples; The standard deviation of the odds for all samples;

[0348] S404: Calculate the probability ratio of the current policy to the old policy for the task selection agent and the workpiece selection agent in terms of the actions already performed. This is used to characterize the magnitude of policy updates; at the same time, a pruning mechanism is used to limit the offset range between the old and new policies to alleviate the training instability caused by excessively rapid policy updates.

[0349] The probability ratio is obtained by subtracting the logarithmic probabilities of the current strategy and the old strategy for the actions already performed.

[0350] The formula is expressed as:

[0351] ;

[0352] In the formula: This represents the probability ratio between the agent's current policy and its old policy on the same executed action. Select an agent for the task and the current strategy for selecting an agent for the workpiece; The old strategy of selecting agents for tasks and selecting agents for workpieces; The pruning factor is used to limit the offset between the old and new policies through the pruning mechanism, thereby alleviating the training instability caused by excessively rapid policy updates.

[0353] S405: Based on the multi-agent proximal policy optimization objective function, the Actor network and Critic network of the task selection agent and the workpiece selection agent are jointly updated. The Actor network is used to improve the policy, and the Critic network is used to improve the accuracy of state value estimation. Furthermore, the policy exploration capability is enhanced by introducing an entropy regularization term, thereby achieving stable optimization of the parameters of the two agent networks.

[0354] The training algorithm based on multi-agent proximal policy optimization updates the Actor network and Critic network of the task selection agent and the workpiece selection agent by combining the total loss function.

[0355] Total loss function The formula is expressed as:

[0356] ;

[0357] ;

[0358] ;

[0359] ;

[0360] ;

[0361] In the formula: , , Let $\mathbf$ represent the policy loss, value loss, and entropy regularization term of the agent (task selection agent or job selection agent), respectively. These are the weighting coefficients for the strategy loss term; This refers to the weighting coefficient for the value loss item; The weighting coefficients for the entropy regularization term; For policy entropy; This indicates the estimated value after trimming; This represents the current state value estimate of the agent output by the Critic network; This represents the state value estimate of the old Critic network; This represents a clipping function, used to clip variables. Limited to the range Inside; Indicates the cutting factor; This represents the expectation of all time-step samples in the current training batch; This represents the mean squared error loss function;

[0362] S406: In this embodiment, the processes of experience sampling, cumulative discount reward calculation, advantage estimation, policy probability ratio pruning, and network parameter updating are repeatedly executed until the preset number of training rounds is reached or the policy convergence condition is met, thereby obtaining the optimal policy parameters for the task selection agent and the job selection agent. Through iterative training, the two agents can gradually learn the collaborative relationship between task selection and job selection, thus forming a joint scheduling strategy for dynamic disturbance environments.

[0363] Repeat steps S401 to S405 until the training reaches the preset number of iterations or the strategy converges, and obtain the optimal policy parameters for the task selection agent and the workpiece selection agent.

[0364] This invention addresses the challenges of deep coupling between production and logistics processes in the context of dual-gripper robots and the continuous changes in system state caused by random workpiece arrivals. It constructs an event-driven Markov decision process model, which uses key events such as machine processing completion, robot transportation completion, and the arrival of new workpieces as decision triggers. This model more accurately represents the evolution of system state in dynamic environments, improving the adaptability of scheduling methods to complex dynamic scenarios. Simultaneously, a hierarchical multi-agent deep reinforcement learning architecture is constructed, decomposing the original high-dimensional, strongly coupled joint scheduling decision into two levels: transportation task type selection and specific service workpiece selection. Through the division of labor and cooperation between task selection agents and workpiece selection agents, the action space scale and policy learning difficulty are reduced, overcoming the shortcomings of single-agent methods in complex dynamic scheduling problems, such as large decision dimensions, low training efficiency, and convergence difficulties.

[0365] This invention constructs a state feature system from three levels: global resource state, candidate task state, and candidate workpiece state, and improves training stability through normalization processing. It designs corresponding action spaces for each of the two agents and combines candidate task and candidate workpiece constraint mechanisms to reduce invalid action searches, improve decision-making efficiency, and enhance the feasibility of scheduling results. Furthermore, it constructs a reward function based on the weighted change in the system's total estimated delay, effectively linking local transportation behavior with the global delivery time optimization objective. This enables the agents to collaboratively learn around minimizing the total weighted delay, thereby improving overall scheduling performance.

[0366] This invention employs a multi-agent proximal policy optimization algorithm to jointly train two agents, combining centralized training with distributed execution mechanisms to enhance collaborative decision-making capabilities among multiple agents while ensuring training stability. This method can rapidly generate effective scheduling strategies under dynamic perturbation environments, reducing the difficulty of solving complex dynamic scheduling problems, improving the collaborative optimization capabilities of robot transportation and machining processes, and effectively improving the total weighted delay and overall operating efficiency of flexible machining units.

[0367] Example 2:

[0368] This embodiment discloses a dynamic control system for flexible processing units based on multi-agent deep reinforcement learning, which is implemented based on the dynamic control method for flexible processing units based on multi-agent deep reinforcement learning in Embodiment 1.

[0369] A flexible processing unit dynamic control system based on multi-agent deep reinforcement learning includes:

[0370] The MDP model building module is used to establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units.

[0371] The agent construction module is used to construct corresponding task selection agents and workpiece selection agents based on the production logistics collaborative decision-making of flexible processing units, forming a hierarchical multi-agent deep reinforcement learning architecture.

[0372] The agent design module, based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, designs the system state characteristics, action space and reward function of the task selection agent and the workpiece selection agent.

[0373] The model training module is used to jointly train the task selection agent and the workpiece selection agent based on the training algorithm of multi-agent proximal policy optimization, obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a cooperative scheduling strategy for the dynamic environment of random workpiece arrival.

[0374] The dynamic production scheduling module is used to achieve dynamic production scheduling of flexible processing units by having the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal strategy parameters obtained from training during the execution phase.

[0375] Example 3:

[0376] This implementation discloses a dynamic control model for flexible processing units based on multi-agent deep reinforcement learning.

[0377] The flexible fabrication unit dynamic control model includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps described in the embodiments of the flexible fabrication unit dynamic control method based on multi-agent deep reinforcement learning. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the aforementioned system embodiments.

[0378] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the intelligent question-and-answer system for equipment operation and maintenance.

[0379] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0380] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A dynamic control method for flexible processing units based on multi-agent deep reinforcement learning, characterized in that, include: S1: Establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units; S2: Based on the production logistics collaborative decision-making of flexible processing units, corresponding task selection agents and workpiece selection agents are constructed to form a hierarchical multi-agent deep reinforcement learning architecture. S3: Based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, design the system state features, action space and reward function of the task selection agent and the workpiece selection agent, and generate a multi-agent network. S4: The training algorithm based on multi-agent near-end policy optimization is used to jointly train the task selection agent and the workpiece selection agent in the multi-agent network to obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a cooperative scheduling strategy for the dynamic environment of random workpiece arrival. S5: During the execution phase, the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal policy parameters obtained from training, thereby realizing dynamic production scheduling of flexible processing units.

2. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 1, characterized in that: In step S1, the steps for establishing the event-driven Markov decision process model include: S101: Construct a dynamic scheduling environment for flexible processing units that includes flexible processing units, workpiece sets, machine sets, process routes, and dynamic arrival mechanisms for new workpieces; S102: Set the decision event type that triggers scheduling decisions based on the dynamic scheduling environment of flexible processing units; S103: Based on the dynamic scheduling environment of flexible processing units and the types of decision events that trigger scheduling decisions, construct an event-driven Markov decision process model for the dynamic scheduling process of flexible processing units. The formula for the event-driven Markov decision process model is expressed as follows: ； In the formula: For state space, For the action space, Let be the state transition probability. For the reward function, Discount factor; The optimization objective of the event-driven Markov decision process model is to minimize the total weighted delay of the system. The formula is expressed as: ； In the formula: Indicates the total weighted average delay; Indicate the importance of different workpieces or orders; This indicates the completion time when the workpiece is finally delivered to the finished product export area; Indicates the delivery time limit for the workpiece.

3. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 2, characterized in that: In step S2, the processing steps for establishing a hierarchical multi-agent deep reinforcement learning architecture include: S201: Based on the production logistics collaborative decision-making of the flexible processing unit environment, a task selection agent and a workpiece selection agent are constructed, and a hierarchical joint decision-making mechanism is established to generate joint scheduling actions; among them, the task selection agent is used to determine whether the robot should prioritize the picking task or the delivery task based on the current system state; the workpiece selection agent is used to select the specific service workpiece from the corresponding candidate workpiece set after the task selection result is given. S202: Establish a hierarchical constraint relationship between the task selection agent and the workpiece selection agent; wherein the task selection agent is located at the upper level, and its output results are used to limit the effective decision range of the workpiece selection agent; the workpiece selection agent is located at the lower level, and is used to make decisions among the candidate workpiece set determined by the task selection results. S203: Design a shared environment interaction mechanism between the task selection agent and the workpiece selection agent; at the decision moment, the task selection agent outputs the task action, and the workpiece selection agent outputs the workpiece action from the constrained candidate workpiece set. The two form a joint scheduling action and act on the flexible processing unit environment. S204: Based on the joint scheduling actions between the task selection agent and the workpiece selection agent, as well as the hierarchical constraint relationship and the shared environment interaction mechanism, a hierarchical multi-agent deep reinforcement learning architecture is formed.

4. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 3, characterized in that: In step S201, the processing steps for generating the joint scheduling action between the task selection agent and the workpiece selection agent include: S2011: Construct the Actor network and Critic network corresponding to the task selection agent, which are used to determine the type of transportation task to be executed first based on the current environment information of the flexible processing unit. The specific processing steps include: (1) At the decision-making moment The task selection agent network (Actor network) selects agents based on input feature information. Output task action probability distribution: ； In the formula: Selecting an agent for a task at time Output task actions; (2) The task selection agent Critic network evaluates the value of the state corresponding to the current input and outputs the result. ; S2012: Construct the Actor network and Critic network corresponding to the workpiece selection agent, which are used to determine the specific service workpiece in the corresponding candidate workpiece set under the constraints of the transportation task type. The specific processing steps include: (1) At the decision-making moment The workpiece selection intelligent agent Actor network selects based on input feature information. And the task actions output by the task selection agent. In the corresponding candidate workpiece set Probability distribution of workpiece actions in the output: ； In the formula: Select an intelligent agent for the workpiece at time. Output of workpiece motion; For the purpose of task action The determined set of candidate workpieces; (2) The workpiece selection agent Critic network evaluates the state value corresponding to the current input and outputs: ; S2013: Establish a hierarchical joint decision-making mechanism between the task selection agent and the workpiece selection agent, and combine the task actions output by the task selection agent with the workpiece actions output by the workpiece selection agent to generate joint scheduling actions. The formula for joint scheduling actions is expressed as follows: ； The corresponding joint strategy is expressed as: ； After applying the joint scheduling action to the flexible processing unit environment, the system state transition is represented as follows: 。 5. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 4, characterized in that: In step S3, the steps for processing the system state characteristics include: S301: Extracting information to characterize the decision moment Global resource status characteristics of the overall operation of the flexible processing unit Global resource status characteristics This includes the number of idle grippers on the robot, the number of idle robots, the robot's current position, workpiece completion rate statistics, and order completion rate; The formula is expressed as: ； ； ； ； ； ； In the formula, This represents the number of idle grippers on the robot. Indicates the total number of robot grippers; Indicates the first A robotic gripper at any time The occupancy status; This represents the number of idle machines. For the first The machine is at all times The occupancy status; For workpiece The completion rate; For workpiece At any moment Number of completed processes; For workpiece Total number of processes; This represents the average completion rate of the workpieces. For a moment Number of workpieces reached; The standard deviation of the workpiece completion rate; For order completion rate; This represents the number of completed orders. This represents the number of orders that have arrived. S302: Extract candidate task state features to characterize the distribution of candidate tasks under different transportation task types at the current decision time. And candidate workpiece status features used to characterize differences in delivery time, transportation, waiting time, and processing of candidate workpieces. Candidate task state characteristics and candidate workpiece state characteristics Calculate the minimum, mean, maximum, or standard deviation of each feature separately; The formula is expressed as: ； ； ； ； ； ； ； In the formula: The percentage of candidate pickup tasks; This is a set of candidate pickup tasks; The percentage of candidate delivery tasks; A set of candidate delivery tasks; For candidate workpieces At any moment Relaxation time; For workpiece Delivery date; For workpiece At any moment The estimated completion time; For candidate workpieces At any moment The delivery time; This is the robot's current position. For workpiece The current location; For workpiece Current process The target machine location; For loading and unloading time; For workpiece The completion time of the previous process; For workpiece The emergency weighting coefficient; S303: Configure global resource status characteristics Candidate task state characteristics and candidate workpiece state characteristics Merge the data to generate system state characteristics. ; The formula is expressed as: ； S304: System state characteristics Perform normalization processing; The normalized system state characteristics are represented as follows: ； In the formula: This represents the normalized system state feature vector; The mean of the state characteristics; The operational variance represents the state characteristics; To prevent constants with a denominator of zero.

6. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 5, characterized in that: In step S3, the steps for processing the motion space include: S311: Designing the action space of the task-selecting agent The task selection agent is responsible for choosing the priority transportation task between pickup and delivery tasks. Action space The formula is expressed as: ； In the formula: To prioritize the pickup task; To prioritize delivery tasks; S312: At the Decision Moment The Actor network, which selects tasks, outputs the policy probability distribution for each task action: )； In the formula: ; Select input features for the agent after normalization of the task; S313: Construct task action mask vectors based on the executability of candidate tasks in the current system. It also imposes constraints on unexecutable tasks; based on the policy distribution after mask constraints, it performs sampling or greedy selection to obtain the task selection agent at time step [time value missing]. Task Actions ; S314: The workpiece selection agent uses a shared backbone network and a dual-output head structure for workpiece selection. The shared backbone network is used to extract the common hidden representation of the current state, and then selects the corresponding output head according to different task modes to calculate the weight vector. Then, candidate workpieces are calculated based on the weight vector. Based on the score, the candidate workpiece with the highest score is ultimately selected. ; Specifically: (1) Common hidden representation of the output of the shared backbone network for: ； In the formula: It is a shared feature extraction module consisting of two fully connected layers; Select the input features of the intelligent agent for the normalized workpiece; (2) Activate the corresponding output head according to the task mode selection result and calculate the weight vector under the current task mode. ; The formula is expressed as: ； In the formula: and These represent the weight matrix and bias vector of the output layer in the pickup task mode, respectively. and These represent the weight matrix and bias vector of the output layer in the delivery task mode, respectively. (3) For each candidate workpiece Constructing eigenvectors And based on the weight vector Calculate candidate workpieces rating ; The formula is expressed as: ； ； In the formula: The ranking feature corresponding to the Maximum Weighted Delay Rule (MWT); The ranking feature corresponding to the maximum weighted delay rule (LWT); The ranking features corresponding to the shortest processing time rule (SPT); The ranking feature corresponding to the longest waiting time rule (LWT); The ranking features corresponding to the Minimum Transit Time (MTT) rule; The sorting features corresponding to the Minimum Remaining Time (MRT) rule; S315: Construct task action mask vectors based on the executability of candidate tasks in the current system. Constraining unexecutable tasks; sampling or greedy selection based on the policy distribution after mask constraints to obtain the workpiece selection agent at time [time value missing]. Workpiece movement ; S316: Combine the task action of the task selection agent with the workpiece action output by the workpiece selection agent to form a joint scheduling action, as shown below: ； In the formula: This indicates the task action output by the task selection agent; This indicates the workpiece action output by the selected intelligent agent.

7. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 1, characterized in that: Step S3, the steps for designing the reward function, include: S321: Calculate the workpiece Delivery time ; The formula is expressed as: ； ； )； In the formula: For workpiece Total processing time; For process Processing time on the target machine; respectively workpiece Total transportation and loading / unloading time; Raw material area; Finished product area; This is the delivery time factor; Indicates workpiece The location of the target machine in the first process; Indicates workpiece The total number of processes; Indicates the loading and unloading time of the workpiece; S322: Place the workpiece Estimated completion time After its final processing step is completed, it is transported and unloaded by robots to the finished product area. The completion time for each workpiece; Add a virtual terminal process Its target location is And it does not include the processing procedure; workpiece At the decision point of Represented as: ； In the formula: and Representing the workpieces respectively Actual and expected delivery Completion time; S323: Calculate any incomplete process The estimated completion time is calculated using the following formula: ； ； In the formula: For process The estimated completion time; For process Expected arrival time at the target machine; For the transportation time between machines in different locations; This refers to the earliest available time for the target machine; The estimated completion time for the virtual terminal process; S324: Based on workpiece Estimated completion time and delivery date Calculation at the decision moment Total estimated weighted delay for all workpieces: ； ； In the formula: For workpiece In Delay in time estimation; Indicates workpiece Importance weights; Weighted average delay for total forecast; For decision-making moments The number of workpieces that have arrived in the system but have not yet been delivered to the finished goods area; S325: Construct an immediate reward function based on the change in the total estimated weighted delay of the system at adjacent decision times, and calculate the cumulative reward using the following formula: ； ； In the formula: This is the feedback value for the improvement of the current joint scheduling action on the overall estimated weighted delay of the system; This refers to the cumulative reward value. Discount factor; The total number of decision steps within a scheduling cycle; The task selection agent and the workpiece selection agent share the reward function and jointly optimize the total weighted delay target through collaborative learning.

8. The method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in claim 7: In step S4, the processing steps of the training algorithm based on multi-agent proximal policy optimization include: S401: Employs a training paradigm of centralized training and distributed execution, enabling the task selection agent and the workpiece selection agent to interact in a shared environment and form joint scheduling actions. ; Upon application to a flexible manufacturing cell environment, an immediate reward is obtained. and the state at the next moment And constitute a sample of decision-making experience. ; S402: Based on the decision-making experience sample, calculate the cumulative discount return and normalize it to obtain the normalized cumulative discount return; The formula is expressed as: ； ； In the formula: For a moment Initial cumulative discount reward; This is the normalized cumulative discount return; Indicates the first The environment at any moment The instant reward received; This represents the discount factor, used to balance the importance of current rewards and future rewards; This is the mean of the discounted returns for all samples; The standard deviation of the discounted returns for all samples; To prevent constants with a denominator of zero; S403: Calculate and estimate the dominance of the task selection agent and the job selection agent respectively using the generalized dominance estimation method. and Then, normalization was performed to obtain... ； ； ； In the formula: For the agent in the first An environment, a moment The original advantage estimate; This is the normalized advantage estimate; For the Critic network under the strategy at time 10:00 State value estimation; This is a termination marker; The dominant attenuation coefficient; The mean of the odds across all samples; The standard deviation of the odds for all samples; S404: The probability ratio is obtained by subtracting the logarithmic probabilities of the current policy and the old policy on the actions already performed. The formula is expressed as: ； In the formula: This represents the probability ratio between the agent's current policy and its old policy on the same executed action. Select an agent for the task and the current strategy for selecting an agent for the workpiece; The old strategy of selecting agents for tasks and selecting agents for workpieces; This is the cutting factor; S405: A training algorithm based on multi-agent proximal policy optimization is used to update the Actor network and Critic network of the task selection agent and the workpiece selection agent by combining the total loss function. Total loss function The formula is expressed as: ； ； ； ； ； In the formula: , , These represent the agent's policy loss, value loss, and entropy regularization term, respectively. These are the weighting coefficients for the strategy loss term; This refers to the weighting coefficient for the value loss item; The weighting coefficients for the entropy regularization term; For policy entropy; This indicates the estimated value after trimming; This represents the current state value estimate of the agent output by the Critic network; This represents the state value estimate of the old Critic network; This represents a clipping function, used to clip variables. Limited to the range Inside; Indicates the cutting factor; This represents the expectation of all time-step samples in the current training batch; This represents the mean squared error loss function; S406: Repeat steps S401 to S405 until the training reaches the preset number of iterations or the policy converges, and obtain the optimal policy parameters for the task selection agent and the workpiece selection agent.

9. A flexible processing unit dynamic control system based on multi-agent deep reinforcement learning, characterized in that: The implementation of the flexible processing unit dynamic control method based on multi-agent deep reinforcement learning in claim 1 includes: The MDP model building module is used to establish an event-driven Markov decision process model based on the dynamic scheduling process of flexible processing units. The agent construction module is used to construct corresponding task selection agents and workpiece selection agents based on the production logistics collaborative decision-making of flexible processing units, forming a hierarchical multi-agent deep reinforcement learning architecture. The agent design module, based on the event-driven Markov decision process model and the hierarchical multi-agent deep reinforcement learning architecture, designs the system state characteristics, action space and reward function of the task selection agent and the workpiece selection agent. The model training module is used to jointly train the task selection agent and the workpiece selection agent based on the training algorithm of multi-agent proximal policy optimization, obtain the optimal policy parameters of the task selection agent and the workpiece selection agent, and generate a collaborative scheduling strategy for the dynamic environment of random workpiece arrival. The dynamic production scheduling module is used to achieve dynamic production scheduling of flexible processing units by having the task selection agent and the workpiece selection agent output task actions and workpiece actions respectively based on the optimal strategy parameters obtained from training during the execution phase.

10. A dynamic control model for flexible processing units based on multi-agent deep reinforcement learning, characterized in that, include: One or more processors; The processor is used to store one or more programs; When the one or more programs are executed by the one or more processors, the method for dynamic control of flexible processing units based on multi-agent deep reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images