ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning

Abstract

The application provides an ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning, which draws on the idea of hierarchical control and distributed problem solving, and designs a collaborative decision-making framework of "expert consensus priority, conflict hierarchical arbitration" by using multi-agent cooperation. In addition, multi-objective optimization scheduling should avoid strategy conflicts between different optimization objectives, which may lead to system decision paralysis or serious performance decline. Therefore, the application decomposes the complex global multi-objective optimization problem into multiple parallel single-objective optimization sub-problems, which are learned by independent expert agents, ensuring the depth of decision-making in each optimization dimension, and then a higher-level comprehensive decision-making agent arbitrates the decision conflicts between experts, thereby effectively resolving strategy conflicts and improving the overall decision-making quality and robustness of the system.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence. Specifically, this invention relates to the cross-application of multi-agent deep reinforcement learning technology in the fields of operations research and intelligent manufacturing, and is an ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning. Background Technology

[0002] In the context of next-generation intelligent manufacturing enterprises, workflow, as the core model for describing and executing business processes, is no longer merely an auxiliary tool for improving productivity. Instead, it is a key bottleneck and core engine directly determining enterprise operational efficiency, market responsiveness, and core competitiveness. Traditional workflow scheduling methods largely rely on static rules or heuristic algorithms, whose poor adaptability and limited optimization capabilities become increasingly apparent when facing increasingly complex and dynamic business environments. Therefore, designing intelligent workflow collaborative optimization models and methods based on digital worker technology and practical application scenarios, and establishing dynamic task allocation and cross-domain data flow real-time optimization models through intelligent collaborative scheduling algorithms, enables forward and reverse workflow collaboration, data sharing, and global optimization of business processes. This plays a crucial role in improving the overall operational efficiency of enterprise workflows and reducing operating costs.

[0003] In recent years, with the rapid improvement of computing power and the explosive growth of data, artificial intelligence technologies represented by reinforcement learning (RL) and deep learning (DL) have flourished. Reinforcement learning allows an agent to interact with its environment and make mistakes, adjusting its strategy based on rewards or penalties from environmental feedback, ultimately aiming to maximize cumulative rewards by autonomously learning a mapping strategy from system state to optimal decision. Deep neural networks, such as multilayer perceptrons (MLP), convolutional neural networks (CNN), recurrent neural networks (RNN), attention mechanisms, Transformers, and graph neural networks (GNN), possess powerful nonlinear function fitting capabilities and feature extraction capabilities for high-dimensional data.

[0004] Deep Reinforcement Learning (DRL) combines deep neural networks with classic reinforcement learning algorithms. Through an end-to-end learning approach, it enables direct control from raw input to output, allowing agents to learn control strategies for complex tasks directly from high-dimensional, raw sensory inputs (such as image pixels and sensor data). As a data-driven technology, it provides a powerful theoretical framework and technical path for solving workflow scheduling decision-making problems that are difficult to address with traditional methods.

[0005] In practice, as industrial software grows larger and more computational tasks are generated in parallel, the efficient scheduling of tasks to the execution phase becomes increasingly important, especially considering the characteristics of industrial scenarios. Workflows have become a core modeling tool for organizing and executing complex task processes. A workflow typically consists of a series of tasks with pre- and post-dependent dependencies, which need to be rationally allocated to a set of resources to achieve specific performance goals. In academic research and industrial practice, workflows are often modeled as a directed acyclic graph, denoted as […]. Each node in the diagram This represents a subtask in a workflow. Each task has its own attributes, such as production time and required resources. Each directed edge in the graph... Representative task and tasks The dependencies between them represent the tasks Must be in the task Execution can only begin after all other steps are completed. Known as The parent task. Known as The child tasks in a directed acyclic graph (DAG) naturally encode the sequential constraints of workflow execution.

[0006] The core problem of workflow scheduling is to map each task in the workflow to a suitable resource and determine its execution order on that resource, while satisfying the dependencies between tasks, thereby optimizing one or more global performance metrics. Key metrics include completion time (Makespan), total cost (Cost), reliability, and energy consumption. Due to the large number of tasks, the complexity of inter-task dependencies, and the heterogeneity and dynamism of computing environments, the workflow scheduling problem has been proven in computational theory to be an NP-hard (Non-deterministic Polynomial-time hard) problem. Workflows typically run in heterogeneous and distributed environments, with systems composed of multiple execution resources (such as execution units and processors), each differing in processing power and cost.

[0007] In recent years, the research objectives of workflow scheduling algorithms have gradually shifted from single-objective optimization to multi-objective optimization, better aligning with the requirements of practical industrial scheduling decisions. These algorithms aim to discover a set of optimal solutions under multiple objectives with inherent contradictions and conflicts. Deep reinforcement learning has demonstrated great potential in providing near-globally optimal, scalable, and generalizable solutions, offering a novel and powerful end-to-end learning framework for solving such complex combinatorial optimization problems. Especially in dynamic and complex environments, it redefines the scheduling problem as a sequential decision-making process, allowing agents to autonomously learn scheduling strategies containing complex trade-offs through direct interaction with the scheduling environment and repeated trial and error.

[0008] As a complex, multi-domain system, ERP (Enterprise Resource Planning) workflow scheduling systems have the following significant characteristics: First, ERP workflow scheduling involves a complex combination of executors, tasks, workers, dependencies, and time, requiring a high-dimensional, dynamic state space representation capable of comprehensively capturing key information. Second, human dynamic factors (such as fatigue levels, preferences, and satisfaction) are non-linear and time-varying, necessitating scientific and reasonable quantification and integration into the scheduling model and optimization objectives. Third, in actual production processes, the system will face changes in the state space and resource pool due to variations in production tasks and productivity, such as the arrival of new orders, the insertion of urgent tasks, machine failures, or the departure of key workers. It must be able to switch from a predetermined execution strategy to an emergency rescheduling mode in a timely and principled manner. Therefore, based on the characteristics of ERP workflow scheduling optimization, this solution proposes an intelligent ERP workflow scheduling method based on multi-agent deep reinforcement learning to improve the overall operational efficiency, reliability, and intelligence level of enterprise workflows. Summary of the Invention

[0009] The purpose of this invention is to address the technical challenges of existing ERP systems in workflow scheduling, namely poor adaptability to multi-objective conflicts and dynamic environmental changes, and the difficulty in balancing decision-making efficiency and quality. This invention fills the gap in the current lack of a systematic and intelligent collaborative scheduling framework for complex industrial scenarios. It proposes an intelligent collaborative scheduling method for ERP workflows based on multi-agent deep reinforcement learning. This method draws on the ideas of hierarchical control and distributed problem solving, utilizing a collaborative decision-making framework of "expert consensus first, conflict hierarchical arbitration" designed by multiple agents. Furthermore, multi-objective optimization scheduling must avoid policy conflicts between different optimization objectives that could lead to system decision paralysis or severe performance degradation. To this end, this invention decomposes the complex global multi-objective optimization problem into multiple parallel single-objective optimization sub-problems, which are then learned by independent expert agents, ensuring decision-making depth in each optimization dimension. A higher-level comprehensive decision-making agent then arbitrates the decision conflicts between experts, effectively resolving policy conflicts and improving the overall decision-making quality and robustness of the system.

[0010] Another objective of this invention is to provide a general and scalable intelligent scheduling framework for complex workflow scheduling problems, aiming to systematically improve the overall performance of scheduling systems in dynamic and multi-constraint environments. Therefore, this solution can also serve as an effective means to improve the implementation of deep reinforcement learning in complex industrial applications.

[0011] The technical solution of this invention is: an intelligent workflow scheduling method for ERP based on multi-agent deep reinforcement learning, comprising the following steps: Step 1) For the multiple conflicting optimization objectives in ERP workflow scheduling, such as minimizing completion time, minimizing resource consumption, and minimizing manpower time, design and train a parallel expert agent for each individual optimization objective. The expert agent is based on a deep reinforcement learning algorithm (such as PPO) framework. In a shared global simulation environment, it is independently trained by being given a unique reward function for its specific objective, so that it can fully learn and master the ability to make near-optimal decisions in the single objective dimension it is responsible for.

[0012] Step 2) Considering that different expert agents will have different decision preferences based on their single optimization objective, resulting in conflicting action instructions when making joint decisions, this solution designs an action fusion and filtering mechanism. This mechanism first compares the actions of all experts to filter out conflict-free actions that are unanimously agreed upon by all expert agents, forming a "consistent action set." Then, it filters the current action set for validity, removing invalid actions that conflict with the current system's hard constraints (such as task dependencies and resource consumption), ultimately forming a streamlined "candidate action pool" for subsequent decision-making.

[0013] Step 3) Above the expert agent level, construct a higher-level comprehensive decision-making agent. Its core task is to weigh and arbitrate the "candidate action pool" generated in Step 2). This comprehensive decision-making agent is trained and makes decisions based on a weighted multi-objective composite reward function, formalized as follows: ,in This is the normalized reward value for the corresponding sub-objective. Output a "comprehensive decision action" that achieves the best balance among multiple conflicting objectives, based on the weight coefficients of each objective.

[0014] Step 4) The final scheduling decision merges the "consistent action set" selected in Step 2) with the "comprehensive decision action" output by the comprehensive decision-making agent in Step 3) (union), forming the final, complete collaborative scheduling action, which serves as the final output of the system. Applying this method can effectively balance multiple conflicting scheduling objectives, significantly improving the overall operational efficiency of the workflow and its adaptability to dynamic environments.

[0015] This invention proposes an intelligent workflow scheduling method for ERP based on multi-agent deep reinforcement learning. Compared with existing methods for improving workflow scheduling performance, its advantages are as follows: Most current research focuses on improving single heuristic algorithms or using single-agent reinforcement learning models. Due to the multi-objective conflict characteristics and dynamic variability of real-world industrial scenarios, these methods often struggle to find a globally optimal solution and lack adaptability and robustness. This invention draws on the concept of hierarchical collaborative decision-making to systematically decompose and reconstruct the complex scheduling problem, providing a general and scalable solution for solving complex multi-objective optimization scheduling problems in the industrial field. Attached Figure Description

[0016] Figure 1 It is an ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning.

[0017] Figure 2 It is a method for implementing a single-objective expert agent based on the PPO algorithm.

[0018] Figure 3 This is a schematic diagram of a 10x10 workflow scheduling example.

[0019] Figure 4 yes Figure 3 The scheduling Gantt chart for the instance.

[0020] Figure 5a , Figure 5b , Figure 5c This is a comparison chart showing the effects of optimizing the target. Detailed Implementation

[0021] The invention will now be further described with reference to the accompanying drawings.

[0022] The core of this invention lies in providing and implementing an intelligent workflow scheduling method for ERP based on multi-agent deep reinforcement learning. It mainly comprises three core components: a simulation environment for heterogeneous ERP workflows, an expert decision-making layer composed of multiple parallel expert agents, and a comprehensive decision-making agent for conflict arbitration. The specific implementation methods of each component will be described in detail below.

[0023] I. Construction of an ERP Heterogeneous Workflow Scheduling Simulation Environment To effectively train and verify the intelligent scheduling method proposed in this invention, a simulation environment for heterogeneous workflow scheduling in an ERP system that can highly simulate real industrial production scenarios is first constructed, such as... Figure 3 As shown, this workflow instance contains 10 Jobs, with each color representing a Job. Each Job contains 10 Tasks. Each Task's information includes its sequence number, expected execution time, and the department that can handle it. For example, J0-T0 indicates that this is the first task in Job 0; Time: 6 indicates the expected execution time is 6 seconds; and M: 0, 4, 8, 9 indicates that the departments with sequence numbers 0, 4, 8, and 9 can handle the task. In this example, the environment is developed based on Python 3.8 and primarily relies on open-source scientific computing libraries such as NumPy and Simpy.

[0024] The ERP workflow scheduling problem is formalized as a Markov Decision Process (MDP), consisting of a quadruple... definition.

[0025] State transition probability: This indicates that after performing action A in state S, the state transitions to S. The probability is 1. In scheduling problems, due to the determinism of the environment (the outcome of an action is fixed), this probability is usually 1. State transitions are implicitly defined and executed by the internal logic of the simulation environment. For example, when an "approval" task is assigned to an employee, the state of the environment changes as follows: the task's state changes from "pending" to "in progress," the employee's state changes from "idle" to "busy," and the environment's global clock advances according to the task's expected processing time. This state transition process is implemented by the environment's `step()` function, which is the core interface through which the environment and the agent communicate and interact.

[0026] The state space (S) contains: 1) Task State Matrix: This is a dimensional matrix. The matrix, where This is the maximum number of tasks supported by the system. This describes the number of features for a single task. Each row represents a task, and its vector expression is:

[0027] The unique identifier for the task.

[0028] The business process ID to which the task belongs.

[0029] : Estimated task processing time (can be dynamic).

[0030] The current status of the task (e.g., pending, in progress, completed, suspended).

[0031] The deadline requirement for the task.

[0032] : A list of indices pointing to all its direct predecessor tasks, reflecting the DAG structure.

[0033] : A list of indices pointing to all its direct successor tasks.

[0034] If the current number of workflow tasks is less than the maximum number of tasks, use zero vectors to fill in the gaps to keep the dimensions fixed.

[0035] 2) Resource State Matrix: This is a dimensional matrix... The matrix, It is the maximum number of resources. These are resource feature numbers. Each row represents a physical or human resource, and its vector expression is:

[0036] A unique identifier for a resource.

[0037] Resource type (e.g., automated robots, junior employees, senior managers).

[0038] The current status of the resource (e.g., idle, busy, faulty, on vacation).

[0039] The cost of using resources per unit of time.

[0040] The efficiency coefficient of resources affects the actual execution time of a task.

[0041] 3) Human Factor State Vector: This is a vector vector with a dimension of... The matrix, It is the maximum number of human resources. It is the human factor characteristic number. Its vector expression is: .

[0042] Used to quantify the dynamic fatigue level of human resources, with a value range.

[0043] A vector representing the employee's skill proficiency across different task types.

[0044] The employee's current real-time workload.

[0045] These parameters will be dynamically updated during the simulation according to preset rules (such as working time and task difficulty).

[0046] Action Space (A): The action space contains all currently executable tasks extracted by the graph neural network. An action is defined as a scheduling decision, that is, assigning task i to resource j for execution. An action is valid if: All prerequisite tasks for this task have been completed.

[0047] This resource is currently idle.

[0048] The type and skills of the resource meet the task requirements.

[0049] Reward Function (R): The reward function is designed individually for different optimization objectives. After the environment receives and executes an action, it calculates and returns the corresponding reward value. For example, for the cost optimization objective, the reward is -(machine operating cost + overdue cost); for the time optimization objective, the reward is -(new Makespan - old Makespan). These reward signals are the direct basis for the agent's learning, and the connection between them lies in the fact that different reward functions shape the specific behavioral patterns of different expert agents.

[0050] II. Design and Implementation of Parallel Expert Agents The first layer of this invention's framework is the expert decision-making layer, which consists of N parallel expert agents. Each expert agent is a complete deep reinforcement learning agent, responsible for learning and making decisions based on a single, well-defined optimization objective. Taking three core optimization objectives as an example, three expert agents are designed: Agent1 (minimizes the maximum completion time), Agent2 (minimizes the total resource consumption), and Agent3 (minimizes the total manpower time).

[0051] All expert agents employ the following... Figure 2 The Proximal Policy Optimization (PPO) algorithm is shown below. Each expert agent consists of two core neural networks: a Policy Network and a Value Network. The Policy Network receives the complete state vector provided by the environment at its input layer. The input layer dimension is equal to the vectorized dimension of the state space S. The first hidden layer consists of multiple fully connected layers and a ReLU activation function, mapping the original state to 256 dimensions for deep feature crossing and abstraction. Finally, through linear layers and Softmax, the 256-dimensional features are mapped to n dimensions, outputting the probability distribution of each action, where n is the number of specific executable tasks. The Value Network maintains the same structure as the Policy Network in its input and hidden layers. At its output layer, it maps the 256-dimensional features to an evaluation of the current decision, i.e., the state value.

[0052] The expert agent obtains the current workflow state from the environment, samples the effects of different actions, and predicts the action probability and state value through a policy network and a value network. It then selects the action with the highest probability to execute. Data generated by the agent's interaction with the environment is temporarily stored in a data collection area. Within a collection cycle, the agent's (observation, action, prob, value, reward, done) tuples (observation state, action, probability of each action, value, reward, execution effect) are stored in a buffer. When the buffer is full, advantage calculation is performed, and the calculated advantage and reward values ​​are used for subsequent network training. After data collection is complete, the training phase begins. The training process is repeated n times. In each round, the data in the buffer is randomly divided into multiple batches. For each batch of data, the total loss is calculated, and gradient backpropagation is performed.

[0053] All expert agents share the same PPO algorithm framework, and the differences in their learning behavior are achieved by designing different state spaces and reward functions.

[0054] Agent1 (Minimize Maximum Completion Time): The goal of this agent is to complete all tasks as quickly as possible, and its reward function is... As shown in the following formula:

[0055]

[0056]

[0057]

[0058] Rewards based on time and idle penalty It consists of two parts: a time reward, which is the difference between the execution time of the two execution points; and an idle penalty, which is the total number of tasks. Half of the idle penalty parameter The node, if the current number of tasks to be executed If the amount exceeds half of the total task volume, a higher penalty parameter will be applied. This ensures that the utilization rate of the execution part is maximized when the workload is high, while a low penalty parameter is used when the workload is low. In this example ( The value is (100, 50).

[0059] Agent2 (Minimize Total Resource Consumption): This agent's goal is to save costs. At each decision step, when an action is executed, the environment calculates the cost incurred by that step, and the cost reward... Operating costs Negative values ​​and delay costs It consists of two parts: the operating cost, which is the product of the running time and the unit time cost of the execution part, and the delay cost, which is the penalty for timed-out tasks. In the example, the value is 10 times the average execution cost of the task, which prompts the agent to focus on timely completion of tasks while reducing costs.

[0060]

[0061] Agent3 (Minimize Human Factors): The objective of this agent is to average the workload of human agents; the reward function is... Based on the degree of human fatigue Load standard deviation It is made from two parts together.

[0062]

[0063] III. Implementation of the Action Fusion and Filtering Module This module processes the outputs of the three expert agents, and the specific steps are as follows: Input: Receive the action probabilities from Agent1, Agent2, and Agent3 in their current state. A set of candidate actions as output. .

[0064] Consistent action filtering: Identify actions recommended by all experts through the intersection of sets. This leads directly to the final decision.

[0065] Validity filtering: Traverse each action in the action space and construct a new action space based on the resources and tasks occupied by the current consistent action set. Output: This module ultimately outputs two sets: a conflict-free "consistent action set". And the new action space after processing.

[0066] IV. Design and Implementation of Integrated Decision-Making Agent This is the second-layer decision-making core of the framework, responsible for scheduling decisions in the new action space. Agent4 is also based on the PPO framework, but its action space excludes consistent action sets and uses a composite reward function to guide the agent to balance three objectives according to certain weights. The final scheduling instruction issued to the execution system consists of two combined parts: , in, The supplementary decisions made by Agent4 based on the new state and action space, and the final overall scheduling scheme's Gantt chart is as follows: Figure 4 As shown, each workflow is represented by a color, with the vertical axis representing the ten execution parts and the horizontal axis representing time. The execution order and execution machine are determined based on the topological relationship between tasks and the expected execution time. The bottom corresponds to the load and fatigue of the ten execution parts.

[0067] By comparing with Shortest Processing Time First (SPT), Most Remaining Tasks First (MTR), Earliest Delivery Date First (EDD), and random algorithms, the results are as follows: Figure 5a , Figure 5b , Figure 5c As shown, the vertical axis represents the three optimization objectives: execution time, execution cost, and human factors. This solution has achieved good results in all three optimization objectives, realizing the coordinated scheduling of multiple optimization objectives.

[0068] In summary, this invention constructs a hierarchical, collaborative multi-agent deep reinforcement learning framework to effectively decompose and intelligently integrate the complex ERP workflow scheduling problem, ultimately achieving an efficient, robust, and scalable intelligent scheduling method.

[0069] The above description provides a detailed account of the ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning described in this invention. However, it is clear that the specific implementation of this invention is not limited thereto. For those skilled in the art, various obvious modifications made to this invention without departing from the spirit and scope of the claims are within the protection scope of this invention.

Claims

1. An intelligent workflow scheduling method for ERP based on multi-agent deep reinforcement learning, characterized in that: Includes the following steps: Step 1) For the multiple conflicting optimization objectives in ERP workflow scheduling, including minimizing completion time, minimizing resource consumption, and minimizing manpower time, design and train a parallel expert agent for each single optimization objective. The expert agent is based on a deep reinforcement learning algorithm framework and is independently trained in a shared global simulation environment by being given a unique reward function for a specific objective. It fully learns and masters the ability to make near-optimal decisions in the single objective dimension it is responsible for. Step 2) Considering that different expert agents will have different decision preferences based on a single optimization objective, which may lead to conflicting action instructions when making joint decisions, an action fusion and filtering mechanism is designed. This mechanism first compares the actions of all expert agents and filters out conflict-free actions that are agreed upon by all expert agents to form a consistent action set. Then, the current action set is filtered for validity, and invalid actions that conflict with the hard constraints of the current system are removed. Finally, a concise candidate action pool is formed for subsequent decision-making. Step 3) Above the expert agent level, construct a higher-level comprehensive decision-making agent. Its core task is to weigh and arbitrate the candidate action pool generated in Step 2). This comprehensive decision-making agent is trained and makes decisions based on a weighted multi-objective composite reward function, formalized as follows: ,in, This is the normalized reward value for the corresponding sub-objective. Output a comprehensive decision action that achieves the best balance among multiple conflicting objectives, based on the weight coefficients of each objective; Step 4) The final scheduling decision merges the set of consistent actions selected in Step 2) with the comprehensive decision actions output by the comprehensive decision agent in Step 3) to form the final, complete collaborative scheduling action, which is the final output of the system.

2. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: Construction of an ERP heterogeneous workflow scheduling simulation environment: First, a simulation environment for ERP heterogeneous workflow scheduling is built to simulate a real industrial production scenario. The workflow contains 10 Jobs, and each Job contains 10 Tasks. The simulation environment is developed based on Python 3.8 or higher. The ERP workflow scheduling problem is formalized as a Markov Decision Process (MDP), consisting of a quadruple... definition.

3. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that: State transition probability: This indicates that after performing action A in state S, the state transitions to S. The probability; state transitions are implicitly defined and executed by the internal logic of the simulation environment; when an approval task is assigned to an employee, the state of the environment will change as follows: the task's state changes from pending to in progress, the employee's state changes from idle to busy, and the environment's global clock will advance according to the task's expected processing time. This state transition process is implemented by the environment's step() function.

4. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that: The state space S contains: Task state matrix: is a matrix with one dimension. The matrix, where, This is the maximum number of tasks supported by the system. This describes the number of features for a single task; each row represents a task, and the vector expression is: ； The unique identifier for the task; The business process ID to which the task belongs; Estimated task processing time; The current state of the task; Task deadline requirements; : A list of indices pointing to all direct preceding tasks, reflecting the DAG structure; : A list of indices pointing to all direct successor tasks; If the current number of workflow tasks is less than the maximum number of tasks, use zero vectors to pad to keep the dimensions fixed; Resource status matrix: This is a dimensional matrix. The matrix, It is the maximum number of resources. These are resource feature numbers; each row represents a physical or human resource, and the vector expression is: ； A unique identifier for a resource; Resource type; The current state of the resource; The cost of using resources per unit of time; The efficiency coefficient of resources affects the actual execution time of a task; Human factor parameter state vector: is a vector with dimension 1. The matrix, It is the maximum number of human resources. It is the number of human factors; the vector expression is: ； : Used to quantify the dynamic fatigue level of human resources, with a value range; A vector representing an employee's skill proficiency across different task types; : The current real-time workload of employees; These parameters will be dynamically updated according to preset rules during the simulation.

5. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that: Action Space A: The action space contains all currently executable tasks extracted by the graph neural network. An action is defined as a scheduling decision, that is, assigning task i to resource j for execution. The condition for an action to be valid is: All prerequisite tasks have been completed; The resource is currently idle; The types of resources and skills meet the task requirements; Reward function R: The reward function is designed separately according to different optimization objectives. After the environment receives an action and executes it, it calculates and returns the corresponding reward value. For cost optimization objectives, the reward is machine running cost + delinquency cost; for time optimization objectives, the reward is new Makespan - old Makespan.

6. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: Each expert agent is a complete deep reinforcement learning agent, responsible for learning and making decisions based on a single, well-defined optimization objective; three expert agents are designed: Agent1, which minimizes the maximum completion time; Agent2, which minimizes the total resource consumption; and Agent3, which minimizes the total manpower time. All expert agents are implemented using the Proximal Policy Optimization (PPO) algorithm. Each expert agent contains two core neural networks: a Policy Network and a Value Network. The Policy Network receives the complete state vector provided by the environment at the input layer. The input layer dimension is equal to the vectorized dimension of the state space S. The first hidden layer consists of multiple fully connected layers and a ReLU activation function, mapping the original state to 256 dimensions for deep feature crossing and abstraction. Finally, through linear layers and Softmax, the 256-dimensional features are mapped to n dimensions, outputting the probability distribution of each action, where n is the number of specific executable tasks. The Value Network maintains the same structure as the Policy Network in the input and hidden layers. At the output layer, it maps the 256-dimensional features to an evaluation of the current decision, i.e., the state value.

7. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 6, characterized in that: The expert agent obtains the current workflow state in the environment, samples the effects of different actions, and predicts the action probability and state value through the policy network and value network. It then selects the action with the highest probability to execute. The data generated by the agent's interaction with the environment is temporarily stored in the data collection area. Within a collection cycle, the (observation, action, prob, value, reward, done) tuple of each step of the agent is stored in the buffer, which contains the observation state, action, probability of each action, value, reward, and execution effect. When the buffer is full, advantage calculation is performed. The calculated advantage value and reward value are used for subsequent network training. Once the data collection is complete, the training phase begins. The training process is repeated n times. In each round, the data in the buffer is randomly divided into multiple batches. For each batch of data, the total loss is calculated and gradient backpropagation is performed.

8. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 7, characterized in that: All expert agents share the same PPO algorithm framework, and the differentiation of learning behavior is achieved by designing different state spaces and reward functions; Agent1: The goal is to complete all tasks as quickly as possible. The reward function is shown in the following formula, which consists of two parts: time reward and idle penalty. Half of the total task volume is used as the node for the idle penalty parameter to ensure that the utilization rate of the execution part is maximized when the task volume is large. ； ； ； ； Rewards based on time and idle penalty It consists of two parts: a time reward, which is the difference between the execution time of the two execution points; and an idle penalty, which is the total number of tasks. Half of the idle penalty parameter The node, if the current number of tasks to be executed If the amount exceeds half of the total task volume, a higher penalty parameter will be applied. This ensures that the utilization rate of the execution part is maximized when the workload is high, while a low penalty parameter is used when the workload is low. ; Agent2: The goal is to save costs. At each decision step, when an action is executed, the environment calculates the cost incurred. The cost reward consists of two parts: the negative value of the running cost and the delay cost. The running cost is the product of the running time and the unit time cost of the execution part, while the delay cost is the penalty for timed-out tasks. ； Agent3: The objective is to average the workload of the workers. The reward function is composed of two parts: the degree of worker fatigue and the standard deviation of the workload. 。 9. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The outputs of the three expert agents are processed in the following steps: Input: Receive the action probabilities from Agent1, Agent2, and Agent3 in their current state. A set of candidate actions as output. ; Consistent action filtering: Identify actions recommended by all experts through the intersection of sets. It proceeds directly to the final decision-making stage; Validity filtering: Traverse each action in the action space and construct a new action space based on the resources and tasks occupied by the current consistent action set. Output: The final output consists of two sets: a set of conflict-free, consistent actions. And the new action space after processing.

10. The ERP intelligent workflow scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The design and implementation of the integrated decision-making agent is as follows: The integrated decision-making agent is responsible for scheduling decisions in the new action space. Agent4 is also based on the PPO framework, but the action space of the integrated decision-making agent excludes the consistent action set. It adopts a composite reward function to guide the integrated decision-making agent to balance three objectives according to their weights. The final scheduling instruction issued to the execution system consists of two combined parts: , in, It is a supplementary decision made by Agent4 based on the new state and action space.

Images