Multi-agent-based dynamic flexible job shop multi-task scheduling method and system

Abstract

The invention belongs to the technical field of intelligent scheduling, and particularly relates to a multi-agent-based dynamic flexible job shop multi-task scheduling method and system, and the method comprises the steps: determining core attribute parameters of each machine tool in a dynamic flexible job shop; abstracting the dynamic flexible job shop into a workpiece agent and a machine tool agent, constructing a multi-agent collaborative scheduling architecture, and determining a global state space and a shared reward function according to core attribute parameters; respectively determining a first local decision model of the workpiece intelligent body and a second local decision model of the machine tool intelligent body; constructing a multi-agent soft actor-commentator model, and performing reinforcement learning training on the multi-agent soft actor-commentator model to obtain an optimal strategy parameter; and performing multi-task real-time integrated scheduling on the dynamic flexible job shop according to the optimal strategy parameters. According to the scheme, the scheduling efficiency and the reliability and adaptability of the scheduling link are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent scheduling technology, and in particular to a method and system for multi-task scheduling in a dynamic flexible workshop based on multiple agents. Background Technology

[0002] With the development of intelligent manufacturing technology, the flexible processing capabilities of intelligent equipment and the application of digital technology have significantly improved the flexibility of the workshop, enabling the manufacturing system to dynamically respond to changing production demands and environmental changes. The dynamic flexible workshop scheduling problem has become one of the key issues in manufacturing system job scheduling. To effectively solve this problem, existing research has proposed various methods, but early methods often assumed continuous and stable operation of machine tools, failing to fully consider the need for regular maintenance in actual production to ensure optimal operating conditions. Unexpected downtime caused by machine tool failures not only disrupts the established scheduling plan but may also lead to overall system failure, production delays, and increased delivery costs. Therefore, integrating preventative maintenance into the scheduling process to ensure efficient, reliable, and stable operation of the workshop has become an important research direction.

[0003] Currently, research on scheduling that comprehensively considers processing and maintenance tasks is gradually increasing, mainly divided into centralized and multi-agent-based distributed scheduling methods. On the one hand, centralized scheduling methods based on metaheuristic algorithms typically model the integrated scheduling problem of processing and maintenance as a mixed-integer programming model and add reliability constraints. Although it can provide an approximate optimal solution for static scheduling problems, it requires iterative calculations when dealing with dynamic changes in the workshop, which is time-consuming and easily affects the original scheduling plan, resulting in poor scheduling performance.

[0004] On the other hand, distributed scheduling methods based on multi-agent systems can respond to dynamic changes through local decision-making and adjust in real time according to actual conditions, resulting in stronger system adaptability. However, after introducing preventive maintenance and reliability constraints for machine tools, existing research neglects the coupling relationship between machining and maintenance tasks, often solving each sub-problem separately or converting maintenance tasks into machining constraints, making it difficult to optimize the overall system performance. At the same time, a large number of studies focus on multi-task integrated scheduling optimization in static environments, while research on multi-task integrated scheduling optimization in dynamic flexible workshops is still insufficient, lacking real-time response capabilities. In addition, current multi-task integrated scheduling optimization methods are mostly aimed at specific scenarios and problems, lacking adaptability and making it difficult to cope with complex and ever-changing intelligent manufacturing scenarios.

[0005] It is evident that traditional multi-task scheduling schemes suffer from technical problems such as low scheduling efficiency, insufficient reliability, and poor adaptability. Summary of the Invention

[0006] This invention provides a dynamic flexible workshop multi-task scheduling method and system based on multiple agents, which solves the shortcomings of traditional multi-task scheduling schemes such as low scheduling efficiency, insufficient reliability and poor adaptability.

[0007] On one hand, this invention provides a multi-agent-based dynamic flexible workshop multi-task scheduling method, comprising: determining the core attribute parameters of each machine tool in the dynamic flexible workshop; abstracting the dynamic flexible workshop into workpiece agents and machine tool agents, constructing a multi-agent collaborative scheduling architecture, and determining the global state space and shared reward function of the dynamic flexible workshop based on the core attribute parameters; determining a first local decision model for the workpiece agent and a second local decision model for the machine tool agent based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space, and shared reward function; constructing a multi-agent soft actor-critic model, and performing reinforcement learning training on the multi-agent soft actor-critic model based on the first local decision model and the second local decision model to obtain optimal policy parameters; and performing real-time integrated multi-task scheduling of the dynamic flexible workshop based on the optimal policy parameters.

[0008] According to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided by the present invention, the core attribute parameters of each machine tool in the dynamic flexible workshop are determined, including: establishing a machine tool reliability model, a machine tool life evolution model, and a maintenance time model for each machine tool in the dynamic flexible workshop; outputting the reliability index value of each machine tool based on the machine tool reliability model; calculating the remaining service life of each machine tool based on the machine tool life evolution model; outputting the maintenance time of each machine tool according to the maintenance time model; and using the reliability index value, remaining service life, and maintenance time as the core attribute parameters of each machine tool.

[0009] According to the multi-agent-based dynamic flexible job shop multi-task scheduling method provided by the present invention, the machine tool reliability model is as follows:

[0010] in, cr tk machine tool k In time t The reliability index value ranges from [0, 1]. g tk machine tool k In time t The cumulative effective service life; η k machine tool k The scale parameter of the Weibull distribution; β k machine toolk The shape parameters of the Weibull distribution.

[0011] According to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided by the present invention, the machine tool life evolution model includes: a life increase model and a life recovery model; When the machine tool performs the target operation, the lifespan increase model is as follows:

[0012] When the machine tool performs maintenance tasks, the life recovery model is as follows:

[0013] in, g tk1 machine tool k In performing the target process O ij The cumulative effective service life after that; g tk machine tool k The cumulative effective operating life at time t; pt ijk Indicate the target process O ij In machine tool k Processing time; g tk2 machine tool k The cumulative effective operating life after performing maintenance tasks; mr k machine tool k The maintenance rate ranges from [0, 1].

[0014] According to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided by the present invention, when establishing the maintenance time model, the maintenance level is pre-divided into basic maintenance level, improved maintenance level and failure maintenance level. The maintenance time model is as follows:

[0015] in, mt tk machine tool k In time t Required maintenance time; bmr k machine tool k The basic maintenance level reliability index value; bmt k This represents the time constant corresponding to the basic maintenance level; imt k This represents the time constant corresponding to the improved maintenance level; fmt k This represents the time constant corresponding to the failure maintenance level; g tk machine tool k In time t The cumulative effective service life; cr tk machine tool k In time t Reliability index values; fr k This indicates the reliability failure threshold of the machine tool.

[0016] According to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided by the present invention, the global state space and shared reward function of the dynamic flexible workshop are determined based on the core attribute parameters, including: determining the machine tool state based on the core attribute parameters, and determining the workpiece state, process state, and interaction state between the workpiece and the machine tool respectively; constructing the global state space of the dynamic flexible workshop based on the workpiece state, process state, machine tool state, and interaction state between the workpiece and the machine tool; and establishing a shared reward function with workpiece completion as the core incentive source.

[0017] According to the multi-agent-based dynamic flexible job shop multi-task scheduling method provided by the present invention, the first local decision model of the workpiece agent is determined, including: constructing a first observation space of the workpiece agent in the multi-agent collaborative scheduling architecture based on the machine tool state and interaction state in the global state space; determining a first action space of the workpiece agent using an implicit action selection mechanism based on attribute weighted scoring; determining the immediate reward of the workpiece agent when the last process of the entire workpiece is completed, with minimizing the weighted delivery delay as the optimization objective, and establishing a first reward function of the workpiece agent; and determining a partially observable Markov decision process of the workpiece agent based on the first observation space, the first action space, and the first reward function to obtain the first local decision model.

[0018] According to the multi-agent-based dynamic flexible job shop multi-task scheduling method provided by the present invention, the second local decision model of the machine tool agent is determined, including: constructing a second observation space of the machine tool agent in the multi-agent collaborative scheduling architecture based on the workpiece state, process state, interaction state in the global state space and the remaining service life in the core attribute parameters; determining the second action space of the machine tool agent using an end-to-end action selection mechanism based on attribute weighted scoring; determining the immediate reward of the machine tool agent with the optimization objective of maximizing reward and minimizing corresponding delay, and establishing a second reward function of the machine tool agent; and determining the partially observable Markov decision process of the machine tool agent based on the second observation space, the second action space and the second reward function to obtain the second local decision model.

[0019] According to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided by the present invention, after obtaining the second local decision model, the method further includes: during the offline training of the second local decision model, if the machine tool agent is not maintained and the current reliability cannot complete the processing task, the processing trajectory will be modified to an active maintenance trajectory and stored in the experience pool; the second local decision model will be optimized by replaying the experience in the experience pool.

[0020] On the other hand, the present invention also provides a multi-agent-based dynamic flexible workshop multi-task scheduling system, comprising: an acquisition module for determining the core attribute parameters of each machine tool in the dynamic flexible workshop; a construction module for abstracting the dynamic flexible workshop into workpiece agents and machine tool agents, constructing a multi-agent collaborative scheduling architecture, and determining the global state space and shared reward function of the dynamic flexible workshop based on the core attribute parameters; a processing module for determining a first local decision model for the workpiece agent and a second local decision model for the machine tool agent based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space, and shared reward function; a training module for constructing a multi-agent soft actor-critic model, and performing reinforcement learning training on the multi-agent soft actor-critic model based on the first local decision model and the second local decision model to obtain optimal policy parameters; and a scheduling module for performing real-time integrated multi-task scheduling of the dynamic flexible workshop based on the optimal policy parameters.

[0021] The present invention provides a multi-agent-based dynamic flexible workshop multi-task scheduling method and system. By determining the core attribute parameters of machine tools to lay the foundation for scheduling data, the workshop is abstracted into heterogeneous multi-agents, and a collaborative architecture and global state-reward mechanism are constructed. Local decision-making models are established for two types of agents, and then trained using a multi-agent soft actor-critic model to obtain the optimal strategy for real-time scheduling. This achieves efficient distributed real-time decision-making. Heterogeneous agents, based on the optimal strategy trained offline, make end-to-end decisions directly based on local observations during the online phase, without relying on centralized iterative optimization, significantly reducing decision latency and improving scheduling efficiency. Simultaneously, it ensures the consistency and autonomy of multi-agent collaborative optimization, effectively balancing processing efficiency and equipment reliability. It can guide machine tool agents to proactively insert preventative maintenance during high-risk periods, avoiding unplanned downtime, significantly reducing weighted delivery delays under typical dynamic disturbance scenarios, and improving the reliability and adaptability of the scheduling process. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the multi-task scheduling method for dynamic flexible workshops based on multiple agents provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the implementation principle of multi-task integrated scheduling in a dynamic flexible workshop. Figure 3 This is a schematic diagram of the reward training curve for the workpiece intelligent agent; Figure 4 This is a schematic diagram of the reward training curve for the machine tool intelligent agent; Figure 5 This is a schematic diagram of the weighted delivery delay training curve; Figure 6 This is a schematic diagram of the structure of a multi-agent-based dynamic flexible workshop multi-task scheduling system provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The following is combined Figures 1 to 6 This invention describes the detailed scheme of the dynamic flexible workshop multi-task scheduling method and system based on multi-agent technology provided in the embodiments of the present invention.

[0026] like Figure 1 As shown in the figure, the multi-agent-based dynamic flexible job shop multi-task scheduling method provided by the embodiments of the present invention mainly includes the following steps: Step 110: Determine the core attribute parameters of each machine tool in the dynamic flexible work workshop.

[0027] In this embodiment, a single machine tool is mainly used as the object. The core attribute parameters of the machine tool are quantified through multiple mathematical models, thereby providing data basis for multi-agent collaborative scheduling.

[0028] Step 120: Abstract the dynamic flexible workshop into workpiece intelligent agents and machine tool intelligent agents, construct a multi-agent collaborative scheduling architecture, and determine the global state space and shared reward function of the dynamic flexible workshop based on core attribute parameters.

[0029] In this embodiment, a multi-agent collaborative scheduling architecture is mainly built, which abstracts the workshop entity into workpiece intelligent agents and machine tool intelligent agents. The two are defined to interact through publish-subscribe or request-response mechanisms. At the same time, the workpiece state, process state, machine tool state, and interaction state are integrated to construct a global state space, and a shared reward function with workpiece completion as the core is designed.

[0030] Step 130: Based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space, and shared reward function, determine the first local decision model of the workpiece agent and the second local decision model of the machine tool agent.

[0031] It is understood that, by constructing a first local decision model for the workpiece intelligent agent, this embodiment can clarify the perception range, decision options, and optimization orientation of the workpiece intelligent agent, thereby achieving feasible machine tool allocation; by constructing a second local decision model for the machine tool intelligent agent, it can clarify the perception range, decision options, and optimization orientation of the machine tool intelligent agent, thereby achieving collaborative optimization of machining and maintenance tasks.

[0032] Step 140: Construct a multi-agent soft actor-critic model, and perform reinforcement learning training on the multi-agent soft actor-critic model based on the first local decision model and the second local decision model to obtain the optimal policy parameters.

[0033] It is understandable that by constructing a multi-agent soft actor-critic model, a decision-making computing platform can be built for the agent, and the algorithm optimization logic can be defined; then, through closed-loop reinforcement learning training of interaction-sampling-learning-update, the optimal policy parameters can be trained.

[0034] Step 150: Based on the optimal strategy parameters, perform real-time integrated scheduling of multiple tasks in the dynamic flexible workshop.

[0035] This embodiment applies the optimal strategy parameters obtained from offline training to a real-world or digital twin production system, enabling real-time integrated scheduling of multiple tasks in a dynamic, flexible workshop.

[0036] In one embodiment, determining the core attribute parameters of each machine tool in a dynamic flexible workshop specifically includes: First, establish machine tool reliability models, machine tool life evolution models, and maintenance time models for each machine tool in the dynamic flexible operation workshop.

[0037] On the one hand, based on the machine tool reliability model, the reliability index value of each machine tool is output.

[0038] In this embodiment, taking each machine tool as the object, a machine tool reliability model based on a two-parameter Weibull distribution can be established to characterize the nonlinear relationship between machine tool service life and reliability. Specifically, the machine tool reliability model can be expressed as: (1) in, cr tk machine tool k In time t The reliability index value ranges from [0, 1]. g tk machine tool k In time t The cumulative effective service life; η k machine tool k The scale parameter of the Weibull distribution; β k machine tool k The shape parameters of the Weibull distribution.

[0039] On the other hand, based on the machine tool life evolution model, the remaining service life of each machine tool is calculated.

[0040] In this embodiment, the dynamic evolution process of machine tool lifespan as machining and maintenance tasks occur can be defined. When the machine tool performs a target operation, its lifespan increases, satisfying the lifespan increase model; when the machine tool performs a maintenance task, its lifespan recovers, satisfying the lifespan recovery model. Therefore, the machine tool lifespan evolution model includes: a lifespan increase model and a lifespan recovery model.

[0041] When the machine tool performs the target operation, the lifespan increase model can be specifically represented as follows: (2) When a machine tool performs maintenance tasks, the life recovery model can be specifically represented as follows: (3) in, g tk1 machine tool k In performing the target process O ij The cumulative effective service life after that; g tk machine tool k In time t The cumulative effective service life; pt ijk Indicate the target process O ij In machine tool k Processing time; g tk2 machine tool k The cumulative effective operating life after performing maintenance tasks; mr k machine tool k The maintenance rate ranges from [0, 1].

[0042] In practical applications, a reliability failure threshold for the machine tool can be set. The formula for calculating the remaining service life of the machine tool is as follows: (4) in, rul tk Indicates the remaining service life of the machine tool; fr k This represents the reliability failure threshold of the machine tool, i.e., the minimum reliability index value at which the machine tool fails.

[0043] On the other hand, based on the maintenance time model, the maintenance time for each machine tool is output.

[0044] In a specific implementation, when establishing a maintenance time model, maintenance levels can be divided according to the current reliability level of the machine tool. Specifically, based on different reliability levels, maintenance time can be defined into three different maintenance levels, including basic maintenance level, improved maintenance level, and failure maintenance level.

[0045] Furthermore, the maintenance time model can be specifically expressed as: (5) in, mt tk machine tool k In time t Required maintenance time; bmr k machine tool k The basic maintenance level reliability index value; bmt k This represents the time constant corresponding to the basic maintenance level; imt k This represents the time constant corresponding to the improved maintenance level; fmt k This represents the time constant corresponding to the failure maintenance level; g tk machine tool k In time t The cumulative effective service life; cr tk machine tool k In time t Reliability index values; fr k This indicates the reliability failure threshold of the machine tool.

[0046] Finally, reliability index values, remaining service life, and maintenance time are used as core attribute parameters for each machine tool.

[0047] It should be noted that, to achieve integrated and optimized scheduling of processing and maintenance tasks under dynamic disturbance environments, this embodiment constructs a multi-agent collaborative scheduling architecture based on heterogeneous multi-agent deep reinforcement learning. This architecture abstracts the physical entities in the dynamic flexible workshop into agents with autonomous perception, decision-making, and communication capabilities, forming an intelligent scheduling system oriented towards state perception and real-time response. Based on this, the global state space and shared reward function of the dynamic flexible workshop are defined, providing a modeling foundation for subsequent learning-based collaborative decision-making.

[0048] In constructing an event-driven multi-agent cooperative scheduling architecture, two types of heterogeneous agents need to be defined: workpiece agents and machine tool agents, such as... Figure 2As shown, the workpiece intelligent agents JA1, JA2, ..., JA are obtained. i and machine tool intelligent agents MA1, MA2, ..., MA n Two types of heterogeneous intelligent agents are responsible for machine tool allocation and task sequencing, respectively, and each achieves policy optimization through experience pools combined with learners. The two interact and make collaborative decisions through publish-subscribe or request-response mechanisms.

[0049] In the workpiece agent definition phase, this embodiment dynamically creates a corresponding workpiece agent for each newly arrived workpiece. The workpiece agent's lifecycle begins when the workpiece enters the system and ends when all processes are completed and the workpiece is delivered. The workpiece agent is responsible for maintaining the workpiece's complete process route, delivery date, priority weight, and other attributes. After each process is completed, based on the current workshop status, it selects the optimal machine tool for the next process.

[0050] like Figure 2 As shown, in the machine tool allocation process, O is the main component. 11 O 12 O 13 Taking the process as the object, starting from the initial state S, select machine tools such as M1 and M2 for the process, and finally reach the end state E. Figure 2 ZhongX 111 =1 indicates the decision to assign the first process of workpiece 1 to machine tool 1.

[0051] In the machine tool intelligent agent definition stage, this embodiment creates a machine tool intelligent agent for each machine tool, which is responsible for managing the local scheduling decisions of the machine tool and solving two core sub-problems. The first is the processing task sequencing problem, which selects the next task to be executed from multiple waiting processing operations; the second is the maintenance task scheduling problem, which determines whether maintenance activities need to be inserted based on the current machine tool reliability status.

[0052] like Figure 2 As shown, in the task sequencing stage, the machine tool agent dynamically decides between executing machining tasks and scheduling maintenance tasks. Specifically, machine tool M1 selects to execute machining tasks, such as operation O. 11 O 12 O 13 Process sequentially. Alternatively, machine tool M1 can be selected to perform a maintenance task. Figure 2 OM indicates maintenance operation, during which processing is suspended to prioritize equipment reliability.

[0053] Figure 2 The middle part shows the O 11 O 21 O 22 O 23The complex scheduling problem of multiple processes reflects the complex coupling relationship between multiple processes and multiple machine tools. The machine tool allocation observation oj of the workpiece intelligent agent... ti Task sequencing observation of machine tool intelligent agents tk All of these stem from this step, which provides state information for intelligent agents to make decisions.

[0054] When a machine tool is idle and receives multiple task requests, the machine tool agent selects tasks according to the scheduling strategy. If a machining task is selected, the machine tool enters the machining state, executes the operation until completion, and then returns to the idle state to await the next round of scheduling decisions. If the machine tool agent determines that a maintenance task needs to be performed based on the reliability assessment results, it initiates the corresponding level of maintenance process, determines the maintenance duration, updates the machine tool status to available status after maintenance is completed, and notifies the coordination module to release resources.

[0055] Through the collaborative mechanism between the workpiece intelligent agent and the machine tool intelligent agent, this embodiment realizes the unified modeling and joint optimization of the three major sub-problems of machine tool allocation, machining task sequencing and maintenance task scheduling, and supports online scheduling and adaptive decision-making in uncertain environments.

[0056] In one embodiment, the global state space and shared reward function of the dynamic flexible workshop are determined based on core attribute parameters, specifically including: First, the machine tool status is determined based on the core attribute parameters, and then the workpiece status, process status, and interaction status between the workpiece and the machine tool are determined respectively.

[0057] The workpiece status includes delivery date and weight, as detailed below: (6) in, sj i Indicates the workpiece status; d i Indicates workpiece i Delivery time; jw i Indicates workpiece i The weight.

[0058] The process status is represented by the number of remaining processes and the average remaining processing time, as detailed below: (7) in, so ij Indicates the process status; nro ij Indicate the target process O ij The number of remaining processes; arpt ij Indicate the target process Oij The average remaining processing time.

[0059] Machine tool status includes current remaining machining time, total remaining machining time, number of workpieces to be processed in the buffer, and remaining service life, as detailed below: (8) in, sm tk Indicates the machine tool status; crpt tk machine tool k In time t The remaining processing time; trpt tk machine tool k In time t Total remaining processing time; nrj tk machine tool k In time t The number of workpieces to be processed; rul tk machine tool k In time t The remaining service life.

[0060] The interaction between the workpiece and the machine tool is defined by the processing time on a specific machine tool and the transportation time required for processing, as follows: (9) in, si tijk This indicates the interaction state between the workpiece and the machine tool; pt ijk Indicate the target process O ij In machine tool k Processing time; tt ijk Indicate the target process O ij Transport to machine tool k The time required.

[0061] Then, based on the workpiece state, process state, machine tool state, and the interaction state between the workpiece and the machine tool, a global state space for a dynamic flexible workshop is constructed.

[0062] Understandably, in order to support the perception and decision-making of intelligent agents, this embodiment defines a global state space for a dynamic flexible workshop, which is composed of workpiece state, process state, machine tool state, and the interaction state between the workpiece and the machine tool, forming a high-dimensional but structured state representation. In this embodiment, the global state space s can be specifically represented as follows: (10) At the same time, a shared reward function is established with the completion of the workpiece as the core incentive source.

[0063] Understandably, in order to guide the workpiece agent and the machine tool agent to collaboratively optimize system performance, this embodiment designs a shared reward function as a feedback signal from the environment to the joint decision-making behavior. All agents learn the optimal scheduling strategy by maximizing long-term cumulative rewards.

[0064] In this embodiment, the shared reward function uses job completion as the core incentive source, encouraging the early completion of high-priority tasks and reducing delays. Specifically, it can be represented as follows: (11) in, r ti This represents the reward value corresponding to the shared reward function; C 1 represents the normalization constant. c i Indicates workpiece i Completion time; d i Indicates workpiece i Delivery time.

[0065] In this embodiment, to address the characteristics of incomplete information and uncertain state evolution in a dynamic flexible workshop, the scheduling decision-making process of each workpiece agent is modeled as a partially observable Markov decision process to achieve autonomous decision-making for global performance optimization under limited perception conditions. This first local decision model ensures that the generated machine tool allocation scheme meets process constraints and resource availability, thereby guaranteeing scheduling feasibility, with minimizing the weighted delivery time delay of workpieces as the core optimization objective.

[0066] In practical applications, this embodiment specifically constructs a first observation space, a first action space, and a first reward function suitable for the workpiece intelligent agent, and ensures their consistency with the global state space and the system objective.

[0067] In one embodiment, determining the first local decision model of the workpiece agent specifically includes: On the one hand, based on the machine tool state and interaction state in the global state space, the first observation space of the workpiece agent in the multi-agent collaborative scheduling architecture is constructed.

[0068] In practical applications, since the workpiece agent cannot obtain the complete global state of the workshop, its decision-making relies on locally observable information. Therefore, this embodiment defines a workpiece agent. i In time t The first observation space is oj tiIt only includes the machine tool status related to the currently scheduled operation. sm tk With interaction state si tijk The details are as follows: (12) On the other hand, an implicit action selection mechanism based on attribute-weighted scoring is adopted to determine the first action space of the workpiece agent.

[0069] In practical applications, to achieve end-to-end intelligent scheduling, this embodiment adopts an implicit action selection mechanism based on attribute-weighted scoring, transforming the machine tool allocation problem into a priority ranking problem. Specifically, the first action space of the workpiece agent is defined as the scheduling priority weight vector of all feasible machine tools, as follows: (13) in, Represents the intelligent agent of the workpiece i In time t The first action space; These represent the attribute weights of the parameters: current remaining processing time, total remaining processing time, number of remaining processes, remaining service life, processing time, and transportation time. The values ​​of each weight are all within the range of [-1, 1].

[0070] The workpiece intelligent agent evaluates the priority of each machine tool based on observed features and attribute weights. pm tk The details are as follows: (14) Finally, the machine tool with the highest priority is selected to process this operation, as follows: (15) in, ma ti Represented as workpiece i The t The machine tool selected for each process step.

[0071] On the other hand, with the goal of minimizing the weighted delivery delay, the immediate reward of the workpiece agent is determined when the last process of the entire workpiece is completed, and the first reward function of the workpiece agent is established.

[0072] In practical applications, to guide the workpiece agent towards the goal of minimizing the weighted delivery delay, this embodiment designs a sparse but semantically clear reward signal, namely, the immediate reward for the workpiece agent. rj tij This is triggered only when the final process of the entire workpiece is completed, forming a terminal reward mechanism. Therefore, the first reward function can be specifically represented as follows: (16) in, rj tij Indicates the first j Instant rewards for each step of the process; p i Indicates workpiece i The number of processes; r ti Indicates workpiece i In the t Release time of each process step.

[0073] Finally, based on the first observation space, the first action space, and the first reward function, the partially observable Markov decision process of the workpiece agent is determined, and the first local decision model is obtained.

[0074] It should be noted that, to achieve efficient and reliable operation of machine tool resources in dynamic environments, this embodiment models the local scheduling decisions of each machine tool agent as a partially observable Markov decision process, enabling it to autonomously balance the relationship between machining task execution and preventative maintenance arrangements under conditions of incomplete information. By designing reasonable observation space, action space, and reward mechanisms, and introducing a failure trajectory modification strategy, it can be ensured that the machine tool agent indirectly supports the global optimization objective, namely minimizing the workpiece weighted delivery delay, while meeting reliability constraints. This effectively solves key problems such as uncertain maintenance timing and sparse learning samples.

[0075] In one embodiment, determining the second local decision model of the machine tool agent specifically includes: On the one hand, based on the workpiece state, process state, interaction state and remaining service life in the core attribute parameters in the global state space, a second observation space for the machine tool agent in the multi-agent collaborative scheduling architecture is constructed.

[0076] It is understandable that the machine tool agent's decision-making relies on its local perception of the current workshop state. Therefore, this embodiment defines its time at... t The first observation vector is om tk Specifically, this includes external observation. oe tk and internal observation oi tk External observation includes workpiece status. sj i Process status so ij and interaction state si tijk Internal observation represents the remaining lifespan of the machine tool's intelligent agent. rultk The details are as follows: (17) (18) On the other hand, an end-to-end action selection mechanism based on attribute-weighted scoring is adopted to determine the second action space of the machine tool agent.

[0077] In practical applications, a machine tool agent can simultaneously solve two sub-problems: whether to perform a maintenance task and, if not, which machining task to select. This embodiment employs an end-to-end action selection mechanism based on attribute-weighted scoring, transforming the complex decision into a priority comparison problem. Specifically, the second action space of the machine tool agent is defined as a dual-channel priority weight vector, as follows: (19) in, This represents the second action space of the machine tool intelligent agent; These represent the attribute weights of each parameter, including workpiece delivery date, workpiece weight, number of remaining processes, average remaining processing time, processing time, and maintenance operation. The values ​​of each weight are all within the range of [-1, 1].

[0078] In this embodiment, the machine tool agent evaluates whether to schedule a maintenance task based on maintenance weight. If the weight of the maintenance task exceeds a preset threshold, the machine tool agent selects to schedule and execute the maintenance task; otherwise, the machine tool agent evaluates the priority of each workpiece in the buffer and selects the workpiece with the highest priority for processing, as follows: (20) in, ts tk machine tool k In the t The actions to be taken at each decision-making moment; omt tk machine tool k In the t The decision-making moment is when to perform maintenance operations; w omt machine tool k The weights of the attributes that perform maintenance operations; thom k machine tool k The threshold for performing maintenance operations; pj ti Indicates workpiece i The t The processing priority of each step can be calculated as follows: (twenty one) On the other hand, with the optimization objective of maximizing the reward and minimizing the corresponding delay, the instantaneous reward of the machine tool agent is determined, and the second reward function of the machine tool agent is established.

[0079] Understandably, to guide the machine tool agent to support production goals while ensuring equipment reliability, this embodiment designs a multi-scale reward function, namely the second reward function, which makes local decisions negatively correlated with the global optimization objective; that is, maximizing the reward corresponds to minimizing the delay. The immediate reward of the machine tool agent is as follows: (twenty two) in, rm tk machine tool k In the t The immediate reward obtained at each decision-making moment; ts tk machine tool k In the t The action chosen at each decision-making moment; Indicates the penalty coefficient; p i Indicates workpiece i In the j The pre-set processing machine tools for each process.

[0080] As shown in the above formula, if the machine tool agent's action is to perform a maintenance task, the immediate reward is zero; if the machine tool's reliability is below a set threshold, the immediate reward is - If the workpiece processing task is successfully completed, the player will receive the same reward as if the workpiece were completed; otherwise, the immediate reward will be zero.

[0081] Finally, based on the second observation space, the second action space, and the second reward function, the partially observable Markov decision process of the machine tool agent is determined, and the second local decision model is obtained.

[0082] In some embodiments, after obtaining the second local decision model, the above method may further include: First, during the offline training of the second local decision model, if the machine tool agent does not maintain it and the current reliability cannot complete the machining task, the machining trajectory will be modified to an actively maintained trajectory and stored in the experience pool.

[0083] Then, the second local decision model is optimized by replaying the experience pool.

[0084] Understandably, due to the scarcity of fault samples in real-world environments, traditional reinforcement learning struggles to adequately learn when to proactively maintain systems to avoid faults. Therefore, this embodiment proposes a failure trajectory modification strategy to enhance the diversity and value density of experience during the training phase. This strategy dynamically intervenes in the experience generation process during offline training, as detailed below: (twenty three) in, tse tk machine tool k In the t The final action taken at each decision-making moment; ts tk machine tool k In the t The initial action chosen at each decision moment.

[0085] If the machine tool agent decides to perform a maintenance task, it executes it directly. Otherwise, the agent assesses the machine tool's current reliability. If the current reliability is sufficient to complete the current machining task, the task is executed directly; otherwise, a maintenance task is scheduled and executed. The modified trajectory will continue machining, replacing the proactive maintenance, and this experience will be stored in a dedicated experience pool for future use.

[0086] It should be noted that, to achieve efficient collaborative decision-making between the workpiece intelligent agent and the machine tool intelligent agent in complex dynamic environments, this embodiment constructs a policy learning framework based on deep neural networks and employs advanced multi-agent reinforcement learning algorithms for end-to-end training. This framework supports continuous action space modeling, stable convergence, and adaptive exploration, and is suitable for high-dimensional state input and long-term scheduling optimization tasks.

[0087] Specifically, in this embodiment, an independent policy network and value network are constructed for each agent, including the workpiece agent and the machine tool agent, using a fully connected feedforward neural network as the basic architecture. For example, the network architecture may include: The input layer has a dimension equal to the number of features in its respective observation space.

[0088] The hidden layer contains three fully connected layers, and all hidden layers use the ReLU function as the loss function to improve the nonlinear fitting capability.

[0089] The output layer of the policy network uses the Tanh activation function to map the output to [ The value network outputs a single scalar representing the Q-value estimate of a state-action pair within the interval [1, 1].

[0090] Furthermore, this embodiment employs a multi-agent soft actor-critic model that incorporates a parameterized temperature factor and a gradient-based automatic entropy adjustment mechanism to achieve a better balance between exploration and utilization.

[0091] To obtain the optimal scheduling strategy, this embodiment performs multiple rounds of iterative training on the multi-agent soft actor-critic model in an offline simulation environment until the strategy converges. The entire training process follows a closed-loop mechanism of interaction-sampling-learning-update, and the specific process is as follows: In the initialization phase, a dynamic flexible workshop simulation environment is constructed, and configurations such as machine tool parameters, process routes, and task arrival modes are loaded; the policy networks and value networks of all workpiece agents and machine tool agents are initialized, with weights randomly initialized; the experience replay pool is initialized to store transfer samples; and the initial temperature parameters and their learning rates are set.

[0092] During the training cycle, at the beginning of each training round, the simulation environment is reset and the queue of pending tasks is cleared. When a new workpiece arrives, the corresponding workpiece agent is instantiated and its observation and decision-making functions are activated. Each agent executes decisions in parallel. Specifically, the workpiece agent selects a feasible machine tool for its current process. The machine tool agent decides to execute a machining or maintenance task, advances the time step, updates state information including machine tool life and buffer status, returns a reward signal, and stores all transfer experience into the experience pool.

[0093] During the centralized network update phase, a centralized learning update is performed every certain number of time steps or after each batch interaction: a batch of samples is randomly sampled from the experience pool; the target Q value is calculated; the two Critic networks are updated, specifically, the Actor network is updated using gradient descent / ascent to update the network parameters; the temperature parameter is automatically adjusted to maintain the target entropy level; and the target network parameters are updated using a soft update method.

[0094] During the termination and output phase, the above process is repeated until the cumulative reward curve converges stably. The optimal strategy parameters after training are output and used for the machine tool allocation decision of the workpiece agent and the task sorting and maintenance arrangement decision of the machine tool agent, respectively.

[0095] In practical applications, after offline training is completed, the learned optimal policy parameters can be deployed to the actual or digital twin intelligent production system to achieve efficient real-time integrated scheduling of multiple tasks.

[0096] The following example, using a practical problem solved by a tool manufacturing company, evaluates the practical application effect of the multi-agent-based dynamic flexible job shop multi-task scheduling method provided by this invention. The dynamic flexible job shop has 8 machine tools providing processing services, and 20 batches of processing tasks dynamically enter the shop according to a predetermined time. Each batch of tasks contains 3 identical workpieces. Each workpiece has 3 operations, each operation can be processed on 2 machine tools, and the processing time on different machine tools is the same. The weights of 20%, 60%, and 20% of the workpieces are set to 1.0, 1.5, and 2.0, respectively. The delivery time of each workpiece is defined as follows: (twenty four) in, d i Indicates workpiece i Delivery time; Indicates workpiece i Arrival time; DF In this embodiment, the delay factor is represented. DF The value is 1.5; K ij Indicate process O ij A selection of machine tools; pt ijk Indicates workpiece i The j The first process is in the k Processing time on selectable machine tools; nm ij Indicate process O ij The number of selectable machine tools; O i Indicates workpiece i A set of processes.

[0097] Detailed processing task information for each processing stage is shown in Table 1.

[0098] Table 1 Detailed Processing Task Information

[0099] In practical applications, the transportation time between any two machine tools is shown in Table 2.

[0100] Table 2. Transportation time between machine tools

[0101] Based on the above settings, multi-task scheduling is performed according to the multi-agent-based dynamic flexible workshop multi-task scheduling method provided in this embodiment. Specifically, the scale parameter in the machine tool reliability model is set to 75, the shape parameter to 1.8, the maintenance rate in the life recovery model is set to 0.7, the machine tool reliability failure threshold in the remaining service life calculation is set to 0.9, the basic maintenance level reliability index value in the maintenance time model is set to 0.95, and the time constants corresponding to the basic maintenance level, improved maintenance level, and failed maintenance level are set to 0.5 minutes, 0.03 minutes, and 10 minutes, respectively. The normalization constant in the shared reward function is set to 1.0 × 10⁻⁶. 5 In the assessment of whether to schedule maintenance tasks, the threshold for the machine tool to perform maintenance operations is set at 0.5, and the penalty coefficient in the second reward function is set at 1.

[0102] In the network architecture construction phase, to input the observation data into the neural network for processing, this embodiment employs a manual feature method to transform the observation data. Specifically, the maximum, minimum, and average values ​​of each observation feature are used as the current observation vector, effectively transforming high-dimensional observation data into low-dimensional neural network input. Therefore, the observation vector of the workpiece agent is 18-dimensional, and the observation vector of the machine tool agent is 17-dimensional. Furthermore, the hidden layer contains three fully connected layers, composed of 128, 256, and 64 neurons, respectively.

[0103] The specific parameters for the intensive training phase are shown in Table 3.

[0104] Table 3 Specific parameters related to training

[0105] The model training process is as follows Figure 3 , Figure 4 and Figure 5 As shown, Figure 3 and Figure 4 The reward training curves of the workpiece agent and the machine tool agent are shown separately. It can be seen that both the machine tool agent and the workpiece agent can quickly obtain higher cumulative rewards and rapidly reach a stable state, demonstrating excellent learning efficiency. Furthermore, the reward training curves of the machine tool agent and the workpiece agent exhibit similar trends, indicating that the reward function can effectively guide the learning process of the machine tool agent, thereby satisfying machine tool reliability constraints and effectively avoiding machine tool failure problems.

[0106] Figure 5The training curves for the optimization objective, specifically the weighted delivery delay, are shown. It can be seen that the algorithm proposed in this invention can effectively shorten the weighted delivery delay, indicating that the designed reward function accurately represents the optimization objective and demonstrates the effectiveness of modeling heterogeneous multi-agent partially observable Markov decision processes.

[0107] The following section compares the performance of the above-mentioned multi-agent-based dynamic flexible workshop multi-task integrated scheduling method with traditional solutions.

[0108] Since traditional methods do not support the coordinated scheduling of processing and maintenance, all comparative methods assume that maintenance is performed before the machine tool is about to fail, i.e., they employ a condition-based maintenance strategy to ensure that reliability constraints are met. The algorithms relied upon by traditional methods include: Combined Heuristic Rule (CHR), Cooperative Coevolution Genetic Programming (CCGP), and Surrogated-assisted Multi-task Multitree Genetic Programming (SMT). 2 The solutions are shown in Table 4. The algorithm used is the GP algorithm and the Multi-Agent Soft Actor-Critic (MASAC-CBM) algorithm.

[0109] Table 4 Comparison Data of Results

[0110] It is easy to see that the multi-agent-based dynamic flexible job shop multi-task scheduling method (hereinafter referred to as "this method") provided in this embodiment shows a significant advantage in reducing weighted delivery delay. Compared with the best-performing traditional method, this method achieves a 14.53% reduction in weighted delivery delay, highlighting the effectiveness of heterogeneous multi-agent cooperative scheduling methods in solving practical scheduling problems with small optimization spaces.

[0111] In summary, the multi-agent-based dynamic flexible workshop multi-task integrated scheduling method provided by this invention effectively solves the challenges in real-world cases by coordinating machine tool allocation, machine tool processing task sequencing, and machine tool maintenance task arrangement. It demonstrates good optimization and responsiveness, showcasing the potential of this method in practical industrial applications. Actual comparative results all prove that this method can meet the real-time scheduling requirements of dynamic flexible workshops, ensuring efficient, reliable, and stable operation of the workshop.

[0112] Based on the same general inventive concept, this invention also protects a multi-agent dynamic flexible workshop multi-task scheduling system. The multi-agent dynamic flexible workshop multi-task scheduling system provided by this invention will be described below. The multi-agent dynamic flexible workshop multi-task scheduling system described below can be referred to in correspondence with the multi-agent dynamic flexible workshop multi-task scheduling method described above.

[0113] like Figure 6 As shown, the multi-agent-based dynamic flexible workshop multi-task scheduling system provided in this embodiment of the invention specifically includes: The acquisition module 210 is used to determine the core attribute parameters of each machine tool in the dynamic flexible work workshop.

[0114] Module 220 is used to abstract the dynamic flexible workshop into workpiece intelligent agents and machine tool intelligent agents, construct a multi-agent collaborative scheduling architecture, and determine the global state space and shared reward function of the dynamic flexible workshop based on core attribute parameters.

[0115] The processing module 230 is used to determine the first local decision model of the workpiece agent and the second local decision model of the machine tool agent based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space and shared reward function.

[0116] Training module 240 is used to construct a multi-agent soft actor-critic model, and to perform reinforcement learning training on the multi-agent soft actor-critic model based on the first local decision model and the second local decision model to obtain the optimal policy parameters.

[0117] The scheduling module 250 is used to perform real-time integrated scheduling of multiple tasks in a dynamic flexible workshop based on the optimal strategy parameters.

[0118] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments of the relevant methods, and will not be elaborated further here.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent-based dynamic flexible job shop multi-task scheduling method, characterized in that, include: Determine the core attribute parameters of each machine tool in the dynamic flexible operation workshop; The dynamic flexible workshop is abstracted into workpiece intelligent agents and machine tool intelligent agents, a multi-agent collaborative scheduling architecture is constructed, and the global state space and shared reward function of the dynamic flexible workshop are determined based on the core attribute parameters. Based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space, and shared reward function, the first local decision model of the workpiece agent and the second local decision model of the machine tool agent are determined respectively. A multi-agent soft actor-critic model is constructed, and the multi-agent soft actor-critic model is trained by reinforcement learning based on the first local decision model and the second local decision model to obtain the optimal policy parameters; Based on the optimal strategy parameters, multi-task real-time integrated scheduling is performed on the dynamic flexible work workshop.

2. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 1, characterized in that, Determine the core attribute parameters of each machine tool in the dynamic flexible work workshop, including: Establish machine tool reliability models, machine tool life evolution models, and maintenance time models for each machine tool in a dynamic flexible operation workshop; Based on the machine tool reliability model, output the reliability index value for each machine tool; Based on the machine tool life evolution model, the remaining service life of each machine tool is calculated. Based on the maintenance time model, output the maintenance time for each machine tool; The reliability index value, remaining service life, and maintenance time are used as the core attribute parameters of each machine tool.

3. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 2, characterized in that, The machine tool reliability model is as follows: in, cr tk machine tool k In time t The reliability index value ranges from [0, 1]. g tk machine tool k In time t The cumulative effective service life; η k machine tool k The scale parameter of the Weibull distribution; β k machine tool k The shape parameters of the Weibull distribution.

4. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 2, characterized in that, The machine tool life evolution model includes: a life increase model and a life recovery model; When the machine tool performs the target operation, the lifespan increase model is as follows: When the machine tool performs maintenance tasks, the life recovery model is as follows: in, g tk1 machine tool k In performing the target process O ij The cumulative effective service life after that; g tk machine tool k In time t The cumulative effective service life; pt ijk Indicate the target process O ij In machine tool k Processing time; g tk2 machine tool k The cumulative effective operating life after performing maintenance tasks; mr k machine tool k The maintenance rate ranges from [0, 1].

5. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 2, characterized in that, When establishing a maintenance time model, the maintenance levels are pre-defined as basic maintenance, improved maintenance, and failure maintenance. The maintenance time model is as follows: in, mt tk machine tool k In time t Required maintenance time; bmr k machine tool k The basic maintenance level reliability index value; bmt k This represents the time constant corresponding to the basic maintenance level; imt k This represents the time constant corresponding to the improved maintenance level; fmt k This represents the time constant corresponding to the failure maintenance level; g tk machine tool k In time t The cumulative effective service life; cr tk machine tool k In time t Reliability index values; fr k This indicates the reliability failure threshold of the machine tool.

6. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 1, characterized in that, Based on the aforementioned core attribute parameters, the global state space and shared reward function of the dynamic flexible workshop are determined, including: The machine tool status is determined based on the core attribute parameters, and the workpiece status, process status, and interaction status between the workpiece and the machine tool are determined respectively. Based on the workpiece state, process state, machine tool state, and the interaction state between the workpiece and the machine tool, a global state space for a dynamic flexible workshop is constructed. A shared reward function is established with the completion of the workpiece as the core incentive source.

7. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 6, characterized in that, Determine the first local decision model for the workpiece intelligent agent, including: Based on the machine tool state and interaction state in the global state space, the first observation space of the workpiece agent in the multi-agent cooperative scheduling architecture is constructed. An implicit action selection mechanism based on attribute-weighted scoring is adopted to determine the first action space of the workpiece agent; With minimizing the weighted delivery delay as the optimization objective, the immediate reward of the workpiece agent is determined when the last process of the entire workpiece is completed, and the first reward function of the workpiece agent is established. Based on the first observation space, the first action space, and the first reward function, the partially observable Markov decision process of the workpiece agent is determined, and the first local decision model is obtained.

8. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 6, characterized in that, Determine the second local decision model for the machine tool agent, including: Based on the workpiece state, process state, interaction state in the global state space, and the remaining service life in the core attribute parameters, a second observation space for the machine tool agent in the multi-agent collaborative scheduling architecture is constructed. An end-to-end action selection mechanism based on attribute-weighted scoring is adopted to determine the second action space of the machine tool agent; With the optimization objective of maximizing reward and minimizing corresponding delay, the immediate reward of the machine tool agent is determined, and the second reward function of the machine tool agent is established. Based on the second observation space, the second action space, and the second reward function, the partially observable Markov decision process of the machine tool agent is determined, and the second local decision model is obtained.

9. The multi-agent-based dynamic flexible job shop multi-task scheduling method according to claim 8, characterized in that, After obtaining the second local decision model, the method further includes: During offline training of the second local decision model, if the machine tool agent does not maintain it and the current reliability cannot complete the machining task, the machining trajectory will be modified to an active maintenance trajectory and stored in the experience pool. The second local decision model is optimized by replaying the experience pool.

10. A dynamic flexible workshop multi-task scheduling system based on multi-agent technology, characterized in that, include: The acquisition module is used to determine the core attribute parameters of each machine tool in the dynamic flexible work workshop; The module is used to abstract the dynamic flexible workshop into workpiece intelligent agents and machine tool intelligent agents, construct a multi-agent collaborative scheduling architecture, and determine the global state space and shared reward function of the dynamic flexible workshop based on the core attribute parameters. The processing module is used to determine the first local decision model of the workpiece agent and the second local decision model of the machine tool agent based on the multi-agent collaborative scheduling architecture, core attribute parameters, global state space and shared reward function. The training module is used to construct a multi-agent soft actor-critic model, and to perform reinforcement learning training on the multi-agent soft actor-critic model based on the first local decision model and the second local decision model to obtain the optimal policy parameters; The scheduling module is used to perform real-time integrated scheduling of multiple tasks in the dynamic flexible work workshop based on the optimal strategy parameters.

Images