Human-machine collaborative dynamic scheduling method fusing double-layer optimization mechanism

Abstract

The application provides a man-machine collaborative dynamic scheduling method fusing double-layer optimization mechanism, and relates to the field of man-machine collaborative dynamic scheduling.The double-layer optimization mechanism combines an offline optimization layer and an online real-time scheduling layer organically: firstly, the offline optimization layer is based on workpiece sets, stations and robot resources, and combines a simulated annealing algorithm for adaptive optimization to obtain a global optimal solution;then, the offline optimization layer outputs the global optimal solution as an initial scheduling scheme of the online real-time scheduling layer, and a multi-agent adaptive near-end strategy optimization algorithm is adopted for real-time adjustment.The application avoids the inefficiency of online scheduling from zero exploration, and ensures the efficiency of decision-making.In addition, the online real-time scheduling layer constructs a collaborative decision-making system composed of task allocation agents, resource scheduling agents and disturbance response agents, significantly reduces the decision dimension pressure faced by a single agent through specialized division of labor, and realizes more fine processing of complex interaction in man-machine collaborative scheduling.

Description

Technical Field

[0001] This invention relates to the field of human-machine collaborative dynamic scheduling, and specifically to a human-machine collaborative dynamic scheduling method that integrates a two-layer optimization mechanism. Background Technology

[0002] Human-machine collaborative dynamic scheduling is a key link in intelligent manufacturing. It involves the rational allocation and scheduling of human resources, robots and tasks to improve production efficiency and reduce costs.

[0003] In the field of production scheduling within intelligent manufacturing environments, traditional single-agent architectures employ a centralized decision-making model, requiring the simultaneous handling of complex multi-dimensional relationships. This single-decision architecture can lead to an explosion of policy network parameters, resulting in a severe curse of dimensionality when dealing with high-dimensional state spaces. Furthermore, these methods typically learn from scratch, lacking prior knowledge guidance, leading to low training efficiency, slow convergence, and difficulty in rapid deployment in real-world production. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a human-machine collaborative dynamic scheduling method that integrates a two-layer optimization mechanism, solving the technical problems of single decision-making architectures being unable to effectively coordinate complex multi-dimensional interactions and having low training efficiency.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A human-machine collaborative dynamic scheduling method integrating a dual-layer optimization mechanism, wherein the dual-layer optimization mechanism includes an offline optimization layer and an online real-time scheduling layer, and the human-machine collaborative dynamic scheduling method includes: Acquire the set of workpieces to be processed, workstations, and robot resources; each workpiece includes several tasks to be processed. In the offline optimization layer, an initial solution is generated based on the workpiece set, workstation, and robot resources; and the initial solution is adaptively optimized using a simulated annealing algorithm, and the global optimal solution is obtained by comparing the optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, and the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; In the online real-time scheduling layer, the global optimal solution is used as the initial scheduling scheme, and a multi-agent adaptive proximal strategy optimization algorithm is employed to adjust the current scheduling scheme in real time. The multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent. The task allocation agent is used for selecting the processing method; the resource scheduling agent is used to assign tasks to corresponding workers and / or robots and select workstations; the disturbance response agent is used to detect disturbance events and invoke the task allocation agent and resource scheduling agent to reallocate resources.

[0006] Preferably, the process of obtaining the global optimal solution includes: The initial solution is used as input to the simulated annealing algorithm; During the cooling process, a perturbation is applied to the current optimal solution based on a hybrid neighborhood search strategy to generate a new solution; the perturbation operation probability is dynamically updated based on the current temperature. Determining whether to accept a new solution includes: based on the optimization objective, calculating the increments of minimizing the maximum completion time, energy consumption, and worker load balancing between the new solution and the current optimal solution; if the new solution is better, then accept the new solution; otherwise, use the Metropolis criterion to calculate the probability of accepting the new solution. Repeat the above perturbation and acceptance process until the termination temperature is reached, then terminate the algorithm and output the global optimal solution.

[0007] Preferably, the framework for the interaction between the task allocation agent, resource scheduling agent, disturbance response agent, and environment includes: (1) State space: The task allocation agent includes a task queue, task priority, required skill level for the task, processing time for different processing methods, and occupancy status of each workstation. The resource scheduling agent includes the task processing method, task priority, required skill level, occupancy status of each workstation, worker skill level, robot fault status, real-time load of each worker, and real-time cumulative energy consumption of the robot, all output by the task allocation agent. For the disturbance response agent, this includes the robot's fault status, the current set of newly added workpieces, and the urgency of each new workpiece. (2) Action space: For the task allocation agent, it refers to the selection of the processing method; The resource scheduling agent refers to the task-resource matching decision used to instruct the assignment of tasks to the corresponding workers and / or robots and to select workstations; For the disturbance response agent, it refers to the dynamic disturbance recovery action, which includes no operation, robot failure-task reassignment, immediate handling of inserting an emergency workpiece, and inserting the emergency workpiece at the end of the queue. (3) Policy Network: For the task allocation agent, a fully connected neural network is used; The resource scheduling agent adopts a Transformer encoder architecture; For the disturbance response agent, a lightweight fully connected neural network is used; (4) Shared reward function: The minimum maximum completion time, energy consumption, worker load balancing increments before and after the decision, as well as the penalty term, are weighted and fused together; wherein the penalty term is used to characterize the decision violation of constraints.

[0008] Preferably, the calculation process for the penalty term includes: If a hard constraint is violated, the penalty term is set to a constant value; the hard constraints include worker skill mismatch, task priority relationship, robot malfunction but still assigning tasks, robot quantity limit, worker quantity limit, and workstation quantity limit; If the soft constraint is violated, the penalty term is calculated based on the current worker load balance and the average load of all workers; wherein the soft constraint refers to the current worker's load exceeding the limit.

[0009] Preferably, the training process of the multi-agent adaptive proximal policy optimization algorithm includes: S10. Make decisions based on the policy networks corresponding to the task allocation agent, resource scheduling agent, and disturbance response agent respectively, so as to generate experience data and store it in the shared experience replay pool. S20. Sample data from the experience replay pool, calculate the time-series difference error for each sample, and perform generalized dominance estimation. S30. Based on computational generalized advantage estimation, the task allocation agent, resource scheduling agent, and disturbance response agent adaptively update the parameters of the corresponding policy network with the objective of maximizing the shearing objective function. S40. The maximum completion time, energy consumption, and worker load balance are comprehensively evaluated through the value network, and the system state prediction value is output. The actual return value is calculated using an n-step time-series difference algorithm. The mean square error between the system state prediction value and the actual return value is minimized, and the mean square error is solved using a gradient descent algorithm to optimize the parameters of the value network. S50. Repeat S10 to S40. If the fluctuation of the actual return value is less than the preset threshold for a preset number of consecutive rounds and the number of constraint violations is less than the preset number, then convergence is determined.

[0010] Preferably, the adaptive shearing objective function in S30 is expressed as: in, For the first Trainable parameters of the policy network of an agent; Let be the expected value at time step t; min is the function that takes the smaller value. This is the clipping function; This represents the ratio of the current strategy to the old strategy. Indicates the current policy in state Select Action The probability, Indicates the old strategy in state Select Action The probability of; For time step t, the generalized advantage is estimated; The dynamic shearing threshold is t.

[0011] Preferably, the dynamic shear threshold is expressed as: in, The initial maximum threshold; The minimum protection threshold is exp; exp is an exponential function. The Wasserstein distance between the old and new strategies; Distance threshold; The coefficient is linear growth. This is the attenuation coefficient.

[0012] A human-machine collaborative dynamic scheduling system integrating a dual-layer optimization mechanism, wherein the dual-layer optimization mechanism includes an offline optimization layer and an online real-time scheduling layer, and the human-machine collaborative dynamic scheduling system includes: The acquisition module is used to acquire the set of workpieces to be processed, workstations, and robot resources; each workpiece includes several processing tasks. An offline optimization module is used to generate an initial solution based on the workpiece set, workstation, and robot resources in the offline optimization layer; and to adaptively optimize the initial solution using a simulated annealing algorithm, and obtain the global optimal solution by comparing optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; An online real-time scheduling module is used in the online real-time scheduling layer to take the global optimal solution as the initial scheduling scheme and to adjust the current scheduling scheme in real time using a multi-agent adaptive proximal strategy optimization algorithm. The multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent. The task allocation agent is used for selecting the processing method; the resource scheduling agent is used for assigning tasks to corresponding workers and / or robots and selecting workstations; the disturbance response agent is used for detecting disturbance events and invoking the task allocation agent and resource scheduling agent to reallocate resources.

[0013] A storage medium storing a computer program for human-machine collaborative dynamic scheduling that incorporates a two-layer optimization mechanism, wherein the computer program causes a computer to execute the human-machine collaborative dynamic scheduling method as described above.

[0014] An electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the human-machine collaborative dynamic scheduling method as described above.

[0015] (III) Beneficial Effects This invention provides a human-machine collaborative dynamic scheduling method that integrates a two-layer optimization mechanism. Compared with existing technologies, it has the following advantages: This invention proposes a two-layer optimization mechanism that organically combines an offline optimization layer and an online real-time scheduling layer. First, the offline optimization layer adaptively optimizes based on the workpiece set, workstation, and robot resources, combined with simulated annealing, to obtain a globally optimal solution. Then, the globally optimal solution output by the offline optimization layer is used as the initial scheduling scheme for the online real-time scheduling layer, which employs a multi-agent adaptive proximal strategy optimization algorithm for real-time adjustments. This invention avoids the inefficiency of exploring from scratch in online scheduling, ensuring efficient decision-making. Furthermore, the online real-time scheduling layer constructs a collaborative decision-making system composed of a task allocation agent, a resource scheduling agent, and a disturbance response agent. This specialized division of labor significantly reduces the decision-making dimensional pressure faced by individual agents, enabling more refined handling of complex interactions in human-machine collaborative scheduling. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a structural block diagram of a two-layer optimization mechanism provided in an embodiment of the present invention; Figure 2 A block diagram illustrating a human-machine collaborative dynamic scheduling method that integrates a two-layer optimization mechanism, provided in an embodiment of the present invention. Figure 3 A diagram illustrating a human-machine collaborative production mode provided in an embodiment of the present invention; Figure 4 A flowchart of a MAAPPO algorithm provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the interaction between multiple agents and the environment, provided as an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This application provides a human-machine collaborative dynamic scheduling method that integrates a two-layer optimization mechanism, solving the technical problem that a single decision-making architecture is difficult to effectively coordinate complex multi-dimensional interaction relationships and has low training efficiency.

[0020] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: (1) Collaborative architecture of multi-agent deep reinforcement learning and dynamic optimization This invention innovatively combines multi-agent systems with deep reinforcement learning to achieve joint optimization of multiple objectives (maximum completion time, energy consumption, and load balancing).

[0021] (2) Two-layer collaborative mechanism This invention creatively employs a two-layer architecture combining an offline optimization layer and an online real-time scheduling layer, such as... Figure 1 As shown, in the offline optimization layer, the improved simulated annealing algorithm generates a globally optimal solution for a high-quality scheduling scheme. The online real-time scheduling layer, based on a multi-agent adaptive proximal policy optimization algorithm, uses the high-quality scheduling scheme generated by the offline optimization layer as the initial scheduling scheme and makes real-time dynamic adjustments. This method avoids the inefficiency of exploring from scratch in online scheduling and solves the problem that traditional methods struggle to balance global optimization and real-time response.

[0022] (3) Specialized division of labor in intelligent agent collaborative decision-making system This invention constructs a collaborative decision-making system composed of a task allocation agent, a resource scheduling agent, and a disturbance response agent. This architecture, through specialized division of labor, allows each agent to focus on solving sub-problems in a specific domain. For example, the task allocation agent is responsible for selecting processing methods, the resource scheduling agent handles resource matching, and the disturbance response agent responds to sudden disturbances. This maintains the flexibility of distributed decision-making while ensuring the effective integration of global information. This design significantly reduces the decision-making dimensional pressure faced by individual agents, enabling the system to handle the complex interactive relationships in human-machine collaborative scheduling with greater precision.

[0023] (4) Rapid response mechanism for dynamic disturbances To address the uncertainties in the production environment, this invention employs a specialized disturbance response agent and a corresponding rapid response mechanism. In the event of unexpected situations such as robot malfunctions or emergency plug-ins, the disturbance response agent can quickly generate a recovery plan.

[0024] (5) Multi-agent adaptive proximal policy optimization algorithm This invention makes several innovative improvements to the traditional Proximal Policy Optimization (PPO) algorithm, proposing a Multi-Agent Adaptive Proximal Policy Optimization (MAAPPO) algorithm specifically suitable for human-machine collaborative dynamic scheduling problems. The improvements include: combining the n-step temporal difference (n-step TD) algorithm and generalized advantage estimation (GAE) to significantly improve sampling efficiency and training stability; and introducing the Wasserstein distance metric to dynamically and adaptively adjust the shearing threshold, enabling the algorithm to automatically select appropriate update magnitudes at different training stages.

[0025] (6) Refined human factors modeling This invention innovatively introduces quantitative modeling of multi-dimensional human factors into the traditional scheduling model. By establishing precise matching constraints between worker skill levels and task requirements, it ensures that tasks are assigned to the most suitable workers. In particular, the system accurately models the time difference between human-machine collaborative processing and independent work, allowing the advantages of human-machine collaboration to be fully utilized.

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] Example 1: like Figure 2As shown, this embodiment of the invention provides a human-machine collaborative dynamic scheduling method that integrates a dual-layer optimization mechanism. The dual-layer optimization mechanism includes an offline optimization layer and an online real-time scheduling layer. The human-machine collaborative dynamic scheduling method includes: S1. Obtain the set of workpieces to be processed, workstations, and robot resources; where each workpiece includes several tasks to be processed. S2. In the offline optimization layer, an initial solution is generated based on the workpiece set, workstation, and robot resources; and the initial solution is adaptively optimized using a simulated annealing algorithm, and the global optimal solution is obtained by comparing the optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, and the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; S3. In the online real-time scheduling layer, the global optimal solution is used as the initial scheduling scheme, and a multi-agent adaptive proximal strategy optimization algorithm is used to adjust the current scheduling scheme in real time; wherein the multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent; the task allocation agent is used for processing mode selection; the resource scheduling agent is used for assigning tasks to corresponding workers and / or robots and selecting workstations; the disturbance response agent is used for detecting disturbance events and calling the task allocation agent and the resource scheduling agent to reallocate resources.

[0028] This invention integrates a two-layer optimization mechanism, using the globally optimal solution output by the offline optimization layer as the initial scheduling scheme for the online real-time scheduling layer. This avoids the inefficiency of exploring from scratch in online scheduling and ensures efficient decision-making. Furthermore, the online real-time scheduling layer constructs a collaborative decision-making system composed of a task allocation agent, a resource scheduling agent, and a disturbance response agent. This specialized division of labor significantly reduces the decision-making pressure faced by individual agents, enabling more refined handling of complex interactions in human-machine collaborative scheduling.

[0029] It should be noted that the assembly line balancing problem (ALBP-HRC) addressed in this embodiment of the invention, which involves human-robot collaboration, can be briefly described as: in an assembly line equipped with… One workstation. workers and In the production workshop of collaborative robots, it is necessary to... Task allocation and scheduling are performed on each workpiece to be processed, such as... Figure 3As shown, each workpiece contains a varying number of tasks to be processed, each with different levels of difficulty, requiring different skill levels from the workers. Task processing methods include independent worker operation, independent robot operation, and human-robot collaborative operation, with different processing methods requiring different times for the same task. Simultaneously, the production process may encounter dynamic disturbances such as urgent workpiece insertion and robot malfunctions. This invention utilizes a multi-agent deep reinforcement learning algorithm to rationally plan and arrange processing tasks, thereby achieving joint optimization of multiple objectives (maximum completion time, energy consumption, and load balancing), effectively improving the flexibility and responsiveness of the production system.

[0030] The following sections will detail the above scheme in the following order: first, the assumptions made; second, the definitions of some mathematical symbols used in the embodiments of this invention; and finally, a detailed description of the two-layer optimization mechanism and the multi-agent adaptive proximal strategy optimization algorithm adopted in the online real-time scheduling layer. Specifically: First, the embodiments of the present invention make the following assumptions about the problem under study: 1) All workpieces to be processed and their associated task information (including task priority relationships, processing time, and required skill level) are known in advance.

[0031] 2) Each workstation can only process one task and select one processing mode at a time.

[0032] 3) Any robot can work at any workstation without functional differences.

[0033] 4) The human-machine collaborative operation mode requires a worker and a robot to work together at the same workstation.

[0034] 5) Each workstation can be assigned a maximum of one robot and one worker.

[0035] 6) Dynamic disturbance events (such as robot malfunctions or emergency plug-ins) occur randomly.

[0036] 7) In the initial state, all robots are usable and have no faults.

[0037] 8) After a robot malfunctions, it will be unable to continue providing services until the malfunction is repaired.

[0038] Secondly, for the sake of consistent description, please refer to Table 1 and define the following mathematical symbols: Table 1 Furthermore, the embodiments of the present invention also define the following optimization objectives: in, The objective function value, Weighted by the maximum completion time, As energy consumption weight, For load balancing weights.

[0039] in, This represents the average load for all workers.

[0040] Finally, see Figure 2 The optimization algorithm that integrates the two-layer optimization mechanism proposed in this embodiment of the invention is as follows: By collaborating between an offline optimization layer and an online real-time scheduling layer, the offline optimization layer generates a globally optimal solution based on an improved simulated annealing algorithm, which serves as the starting point and initial scheduling scheme for the online real-time scheduling layer. The online real-time scheduling layer, relying on a multi-agent reinforcement learning framework, achieves real-time response to dynamic disturbances and dynamic resource allocation through collaborative decision-making by three types of agents: task allocation, resource scheduling, and disturbance response. This significantly improves the system's adaptability and scheduling efficiency in dynamic environments while ensuring the global optimization objective. Specifically: As stated in step S2, in the offline optimization layer, an initial solution is generated based on the workpiece set, workstation, and robot resources; and the initial solution is adaptively optimized using a simulated annealing algorithm, and the global optimal solution is obtained by comparing the optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, and the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing.

[0041] For the offline optimization layer, an initial solution is generated based on the workpiece set, workstation, and robot resources. The relevant steps are as follows: (1) Input Receive basic data: workpiece set Task Set Workstation Collection Workers gathering Collaborative robot collection Task Priority Relationship , workpiece Task Required skill level Worker skill level .

[0042] (2) Initial solution generation The aim here is to generate a feasible initial solution that satisfies all hard constraints (worker skill mismatch, task priority relationship, robot failure but still assigning tasks, robot number limit, worker number limit, and workstation number limit, etc.). This serves as the starting point for subsequent optimization algorithms. The specific process for generating the initial solution includes: ①Task sorting and workstation allocation Based on the set of priority relationships between tasks, tasks are arranged in a topological order (Directed Acyclic Graph, DAG) to ensure that all tasks are processed sequentially according to their priority. To maximize resource utilization and minimize the maximum completion time, tasks on the critical path are preferentially assigned to currently idle workstations, while ensuring that each workstation can only process one independent task at a time.

[0043] ② Processing method selection and dynamic resource binding For each task to be assigned Based on its characteristics (required skill level), determine its processing method and bind it to specific resources. If Preferred High-skill tasks ( ) Priority selection ; the rest of the options .

[0044] When the processing method is determined to be or At that time, a worker needs to be assigned to the task, and a minimum heap should be used to start from the skill level required ( The system selects the worker with the lowest current load from the worker pool and assigns them to the worker pool to achieve the goal of load balancing.

[0045] In addition, the following constraints must be met: the total number of robots assigned to all workstations shall not exceed the total number of collaborative robots, and the total number of workers assigned to all workstations shall not exceed the total number of workers.

[0046] ③ Output Finally, output a complete initial solution. The solution clearly specifies the workstation to which the task belongs. Processing methods adopted The assigned workers and / or the assigned robot .

[0047] Furthermore, the simulated annealing algorithm is used to adaptively optimize the initial solution, and the global optimal solution is obtained by comparing the optimization objective. The relevant steps are as follows: S100. Use the initial solution as input to the simulated annealing algorithm; S200. During the cooling process, a perturbation is applied to the current optimal solution based on a hybrid neighborhood search strategy to generate a new solution; wherein the perturbation operation probability is dynamically updated based on the current temperature. S300, Determine whether to accept the new solution, including: Based on the optimization objective, calculate the increments in minimizing the maximum completion time, energy consumption, and worker load balancing between the new solution and the current optimal solution: in, The increment of the objective function value. The increment of the maximum completion time, For the increase in energy consumption, For incremental load balancing.

[0048] If the new solution is better, then accept the new solution; otherwise, use the Metropolis criterion to calculate the probability of accepting the new solution: in, This represents a function that takes the smaller value. It is an exponential function. Let be the current temperature. The above formula indicates that if There is a certain probability that a new solution will be accepted to promote global exploration, and the lower the temperature, the better the chance of success under the same conditions. The lower the probability of accepting a new solution under certain circumstances, the better the local convergence is guaranteed. S400. Repeat the above perturbation and acceptance process until the termination temperature is reached, then terminate the algorithm and output the global optimal solution.

[0049] As stated in step S3, in the online real-time scheduling layer, the global optimal solution is used as the initial scheduling scheme, and a multi-agent adaptive proximal strategy optimization algorithm is used to adjust the current scheduling scheme in real time; wherein the multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent; the task allocation agent is used for selecting the processing method; the resource scheduling agent is used for assigning tasks to the corresponding workers and / or robots and selecting workstations; the disturbance response agent is used for detecting disturbance events and calling the task allocation agent and the resource scheduling agent to reallocate resources.

[0050] The embodiments of this invention employ a multi-agent adaptive proximal policy optimization algorithm, which integrates three types of agents: Task Allocation Agent (TAA), Resource Scheduling Agent (RSA), and Disturbance Response Agent (DRA) to construct a collaborative decision-making system.

[0051] Within the "Centralized Training with Decentralized Execution" (CTDE) framework: During the training phase, a centralized value network acquires global state information, providing unified value estimation and advantage assessment for the policy networks of the three agents, enabling collaborative policy optimization. During the execution phase, each agent independently and in parallel generates actions based on local observations and its trained policy network, without the need for central controller intervention or global communication support. The three agents collaborate through a shared reward function and communication mechanism, maintaining decision-making autonomy while ensuring efficient and stable overall system response.

[0052] Specifically, the framework for the interaction between the task allocation agent, resource scheduling agent, disturbance response agent, and environment includes: (1) State space: The task allocation agent (local state) includes the task queue, task priority, skill level required for the task, processing time for different processing methods, and occupancy status of each workstation. For the resource scheduling agent (local state), it includes the task processing method, task priority, skill level required for the task, occupancy status of each workstation, worker skill level, robot fault status, real-time load of each worker, and real-time cumulative energy consumption of the robot, which are decided and output by the task allocation agent. The disturbance response agent (local state) includes the robot fault state, the currently added set of workpieces, and the urgency of each new workpiece.

[0053] If we understand, the corresponding global state includes the task queue, task priority, skill level required for the task, processing time for different processing methods, occupancy status of each workstation, worker skill level, robot fault status, current (time t) set of newly added workpieces and urgency of each new workpiece, real-time load of each worker, and real-time cumulative energy consumption of the robot.

[0054] (2) Action space: For the task allocation agent, it refers to the selection of the processing method; The resource scheduling agent refers to the task-resource matching decision used to instruct the assignment of tasks to the corresponding workers and / or robots and to select workstations; The disturbance response agent refers to dynamic disturbance recovery actions, including no operation, robot fault-reassignment of tasks, immediate handling of inserting emergency workpieces, and inserting emergency workpieces at the end of the queue.

[0055] If understood, the joint actions include processing method selection, task-resource matching decision-making, and dynamic disturbance recovery actions.

[0056] (3) Policy Network: For the task allocation agent, a fully connected neural network is used; The resource scheduling agent adopts a Transformer encoder architecture; For the disturbance response agent, a lightweight fully connected neural network is used.

[0057] It is understood that the policy network selection for each of the above-mentioned intelligent agents is not unique, and those skilled in the art can make substitutions according to actual needs.

[0058] Furthermore, it is necessary to point out that the disturbance response agent outputs the probability distribution of each action through its policy network, and the agent samples the aforementioned disturbance response actions accordingly: (1) Robot malfunction The unfinished tasks of the faulty robot are sorted according to the shortest remaining processing time (SRPT), and the resource scheduling agent RSA is invoked to reallocate resources.

[0059] (2) Emergency workpiece insertion If the urgency of the new workpiece If a task is forced to be prioritized, the task allocation agent (TAA) and resource scheduling agent (RSA) will be invoked to reallocate resources; otherwise, the task will be added to the end of the normal queue. Urgency threshold (4) Shared reward function: The minimum maximum completion time, energy consumption, worker load balancing increments before and after the decision, along with the penalty term, are weighted and fused; the penalty term is used to characterize the decision violation of constraints, and is expressed as: in, Let be the reward value at time t. As a penalty item, The weight of the penalty item.

[0060] Specifically, the penalty item The calculation process includes: 1) If a hard constraint is violated, the penalty term is taken as a constant value; the hard constraints include worker skill mismatch, task priority relationship, robot malfunction but still assigning tasks, robot quantity limit, worker quantity limit, and workstation quantity limit, then: in, This represents the penalty coefficient for violating hard constraints.

[0061] 2) If the soft constraint is violated, the penalty term is calculated based on the current worker load balance and the average load of all workers; where the soft constraint refers to the current worker's load exceeding the limit, then: in, The penalty coefficient for violating soft constraints, and ; This is the coefficient representing the average load of all workers.

[0062] In an alternative implementation, such as Figure 4 As shown, the training process of the multi-agent adaptive proximal policy optimization algorithm includes: S10. Based on the policy networks corresponding to the task allocation agent, resource scheduling agent, and disturbance response agent, decisions are made respectively to generate empirical data (state). Joint Action ,award New status And store it in the shared experience replay pool.

[0063] S20. Sample data from the experience replay pool and calculate the time difference error for each sample. And perform generalized dominance estimation. .

[0064] Among them, discount factor and Generalized Advantage Estimation (GAE) parameters Dynamically adjusted, decaying as training progresses; Output for value networks; For multi-step forward-looking window, For time step indexing.

[0065] S30. Based on computational generalized advantage estimation, the task allocation agent, resource scheduling agent, and disturbance response agent adaptively update the parameters of their respective policy networks with the objective of maximizing the shearing objective function.

[0066] The adaptive shearing objective function is expressed as: in, For the first Trainable parameters of the policy network of an agent; Let be the expected value at time step t; min is the function that takes the smaller value. This is the clipping function; This represents the ratio of the current strategy to the old strategy. Indicates the current policy in state Select Action The probability, Indicates the old strategy in state Select Action The probability of; For time step t, the generalized advantage is estimated; The dynamic shearing threshold is t.

[0067] Furthermore, the dynamic shearing threshold is expressed as: in, The initial maximum threshold; The minimum protection threshold is exp; exp is an exponential function. The Wasserstein distance between the old and new strategies; Distance threshold; The coefficient is linear growth. This is the attenuation coefficient.

[0068] Understandably, when the strategy changes drastically... (For example, when dealing with robot malfunctions), the strategy needs to be adjusted quickly, automatically widening the shearing range to accelerate adaptation; when The strategy updates are stable, gradually tightening for refined optimization.

[0069] S40. Through the value network, comprehensively evaluate the maximum completion time, energy consumption, and worker load balance, and output the system state prediction value. The actual return value is calculated using an n-step time-series difference algorithm. ; and by minimizing the mean square error between the predicted system state and the actual return value. The gradient descent algorithm is used to solve for the mean square error to optimize the parameters of the value network, so as to ensure that the value function can accurately evaluate the long-term expected return of the system state and provide a reliable benchmark for the advantage estimation of the policy network.

[0070] in, The overall output of the value network is the predicted long-term expected return of the system state. In the state The projected return based on the maximum time to completion in the future; In the state The following is a forecast of the return on total future energy consumption; In the state The following are the projected returns based on the future level of worker workload balancing.

[0071] S50. Repeat S10 to S40. If the fluctuation of the actual return value is less than the preset threshold for a preset number of consecutive rounds and the number of constraint violations is less than the preset number, then convergence is determined.

[0072] In addition, such as Figure 5 As shown, in the multi-agent cooperative scheduling framework, the task allocation agent (TAA), the resource scheduling agent (RSA), and the disturbance response agent (DRA) achieve dynamic decision-making through continuous interaction with the shared environment. The specific interaction process of the three agents is as follows: ① Observation: Each agent obtains the local state information required for its decision-making from the shared environment.

[0073] ② Decision-making and execution: Each agent makes independent decisions and generates joint actions based on its own policy network.

[0074] TAA: Selects the optimal processing method for the task and transmits this decision to RSA.

[0075] RSA: Receives instructions from TAA, comprehensively considers factors such as resource status and load balancing, makes the final task-resource binding decision, assigns the task to a specific worker or robot for execution, and selects a specific workstation.

[0076] DRA: When a disturbance event is detected, TAA and RSA are invoked to reallocate resources.

[0077] ③ Feedback: After a joint action changes the environmental state, the environment updates to a new global state, feeding back a shared reward signal that integrates multiple objectives to all agents. This reward signal is used to evaluate the merits of the joint action and drive the collaborative optimization of the policies of all agents.

[0078] Thus, this embodiment of the invention completes the entire process of the human-machine collaborative dynamic scheduling method integrating a two-layer optimization mechanism. This method uses core enterprise production indicators—maximum completion time, energy consumption, and load balancing—as optimization objectives. Through a multi-agent collaborative decision-making mechanism, it significantly improves scheduling efficiency. Compared to traditional methods, it not only converges faster and effectively avoids getting trapped in local optima, but also achieves adaptive optimization of objectives in dynamic environments, thereby meeting the demands of intelligent manufacturing for high efficiency, flexibility, and stability.

[0079] Example 2: This invention provides a human-machine collaborative dynamic scheduling system that integrates a dual-layer optimization mechanism, comprising an offline optimization layer and an online real-time scheduling layer. The human-machine collaborative dynamic scheduling system includes: The acquisition module is used to acquire the set of workpieces to be processed, workstations, and robot resources; each workpiece includes several processing tasks. An offline optimization module is used to generate an initial solution based on the workpiece set, workstation, and robot resources in the offline optimization layer; and to adaptively optimize the initial solution using a simulated annealing algorithm, and obtain the global optimal solution by comparing optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; An online real-time scheduling module is used in the online real-time scheduling layer to take the global optimal solution as the initial scheduling scheme and to adjust the current scheduling scheme in real time using a multi-agent adaptive proximal strategy optimization algorithm. The multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent. The task allocation agent is used for selecting the processing method; the resource scheduling agent is used for assigning tasks to corresponding workers and / or robots and selecting workstations; the disturbance response agent is used for detecting disturbance events and invoking the task allocation agent and resource scheduling agent to reallocate resources.

[0080] Example 3: This invention provides a storage medium storing a computer program for human-machine collaborative dynamic scheduling that integrates a two-layer optimization mechanism, wherein the computer program causes a computer to execute the human-machine collaborative dynamic scheduling method as described in Embodiment 1.

[0081] Example 4: This invention provides an electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the human-machine collaborative dynamic scheduling method as described in Embodiment 1.

[0082] It is understood that the human-machine collaborative dynamic scheduling system, storage medium and electronic device with integrated dual-layer optimization mechanism provided in the embodiments of the present invention correspond to the human-machine collaborative dynamic scheduling method with integrated dual-layer optimization mechanism provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts in the sub-methods, and will not be repeated here.

[0083] In summary, compared with existing technologies, it has the following beneficial effects: 1. An innovative two-layer optimization mechanism was designed, organically combining an offline optimization layer and an online real-time scheduling layer. This architecture provides a high-quality initial solution through the offline optimization layer, while the online real-time scheduling layer enables fine-tuning and rapid decision-making response. This ensures both the overall optimality of the scheduling scheme and the efficiency of real-time decision-making, effectively solving the problem that traditional methods struggle to balance global optimization and real-time response.

[0084] 2. A multi-agent collaborative optimization mechanism was developed. Through the division of labor and cooperation among task allocation agents, resource scheduling agents, and disturbance response agents, the collaborative optimization of multiple objectives such as production efficiency, energy consumption, and load balancing was achieved. This innovative design breaks through the limitations of traditional methods, which often suffer from the inconsistency of addressing one aspect at the expense of another, enabling the production system to ensure efficiency while simultaneously taking into account energy consumption control and personnel load balancing.

[0085] 3. A dynamic response mechanism was proposed, which can respond to various sudden disturbances in the production process in real time, significantly improving the robustness and stability of the production system and ensuring the continuity of the production process.

[0086] 4. A refined human factors model has been constructed, enabling intelligent allocation of human resources. This system can dynamically adjust task allocation based on workers' skill levels and real-time status, achieving complementary advantages between humans and machines.

[0087] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-machine collaborative dynamic scheduling method integrating a two-layer optimization mechanism, characterized in that, The dual-layer optimization mechanism includes an offline optimization layer and an online real-time scheduling layer, and the human-machine collaborative dynamic scheduling method includes: Acquire the set of workpieces to be processed, workstations, and robot resources; each workpiece includes several tasks to be processed. In the offline optimization layer, an initial solution is generated based on the workpiece set, workstation, and robot resources; and the initial solution is adaptively optimized using a simulated annealing algorithm, and the global optimal solution is obtained by comparing the optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, and the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; In the online real-time scheduling layer, the global optimal solution is used as the initial scheduling scheme, and a multi-agent adaptive proximal strategy optimization algorithm is employed to adjust the current scheduling scheme in real time. The multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent. The task allocation agent is used for selecting the processing method; the resource scheduling agent is used to assign tasks to corresponding workers and / or robots and select workstations; the disturbance response agent is used to detect disturbance events and invoke the task allocation agent and resource scheduling agent to reallocate resources. The framework for the interaction between the task allocation agent, resource scheduling agent, disturbance response agent, and environment includes: (1) State space: The task allocation agent includes a task queue, task priority, required skill level for the task, processing time for different processing methods, and occupancy status of each workstation. The resource scheduling agent includes the task processing method, task priority, required skill level, occupancy status of each workstation, worker skill level, robot fault status, real-time load of each worker, and real-time cumulative energy consumption of the robot, all output by the task allocation agent. For the disturbance response agent, this includes the robot's fault status, the current set of newly added workpieces, and the urgency of each new workpiece. (2) Action space: For the task allocation agent, it refers to the selection of the processing method; The resource scheduling agent refers to the task-resource matching decision used to instruct the assignment of tasks to the corresponding workers and / or robots and to select workstations; The disturbance response agent refers to dynamic disturbance recovery actions, which include no operation, robot failure-reassignment of tasks, immediate handling of inserted emergency workpieces, and inserting emergency workpieces at the end of the queue. (3) Policy Network: For the task allocation agent, a fully connected neural network is used; The resource scheduling agent adopts a Transformer encoder architecture; For the disturbance response agent, a lightweight fully connected neural network is used; (4) Shared reward function: The minimum maximum completion time, energy consumption, worker load balancing increments before and after the decision, as well as the penalty term, are weighted and fused together; wherein the penalty term is used to characterize the decision violation of constraints. The training process of the multi-agent adaptive proximal policy optimization algorithm includes: S10. Make decisions based on the policy networks corresponding to the task allocation agent, resource scheduling agent, and disturbance response agent respectively, so as to generate experience data and store it in the shared experience replay pool. S20. Sample data from the experience replay pool, calculate the time-series difference error for each sample, and perform generalized dominance estimation. S30. Based on computational generalized advantage estimation, the task allocation agent, resource scheduling agent, and disturbance response agent adaptively update the parameters of the corresponding policy network with the objective of maximizing the shearing objective function. S40. By comprehensively evaluating the maximum completion time, energy consumption, and worker load balance through the value network, output the system state prediction value, and calculate the actual return value using an n-step time-series difference algorithm; and optimize the parameters of the value network by minimizing the mean square error between the system state prediction value and the actual return value and solving the mean square error using a gradient descent algorithm. S50. Repeat S10 to S40. If the fluctuation of the actual return value is less than the preset threshold for a preset number of consecutive rounds and the number of constraint violations is less than the preset number, then convergence is determined.

2. The human-machine collaborative dynamic scheduling method as described in claim 1, characterized in that, The process of obtaining the global optimal solution includes: The initial solution is used as input to the simulated annealing algorithm; During the cooling process, a perturbation is applied to the current optimal solution based on a hybrid neighborhood search strategy to generate a new solution; the perturbation operation probability is dynamically updated based on the current temperature. Determining whether to accept a new solution includes: based on the optimization objective, calculating the increments of minimizing the maximum completion time, energy consumption, and worker load balancing between the new solution and the current optimal solution; if the new solution is better, then accept the new solution; otherwise, use the Metropolis criterion to calculate the probability of accepting the new solution. Repeat the above perturbation and acceptance process until the termination temperature is reached, then terminate the algorithm and output the global optimal solution.

3. The human-machine collaborative dynamic scheduling method as described in claim 1, characterized in that, The calculation process for the penalty item includes: If a hard constraint is violated, the penalty term is set to a constant value; the hard constraints include worker skill mismatch, task priority relationship, robot malfunction but still assigning tasks, robot quantity limit, worker quantity limit, and workstation quantity limit; If the soft constraint is violated, the penalty term is calculated based on the current worker load balance and the average load of all workers; wherein the soft constraint refers to the current worker's load exceeding the limit.

4. The human-machine collaborative dynamic scheduling method as described in claim 1, characterized in that, The shearing objective function in S30 is expressed as follows: in, For the first Trainable parameters of the policy network of an agent; Let be the expected value at time step t; min is the function that takes the smaller value. This is the clipping function; This represents the ratio of the current strategy to the old strategy. Indicates the current policy in state Select Action The probability, Indicates the old strategy in state Select Action The probability of; For time step t, the generalized advantage is estimated; The dynamic shearing threshold is t.

5. The human-machine collaborative dynamic scheduling method as described in claim 4, characterized in that, The dynamic shearing threshold is expressed as: in, The initial maximum threshold; The minimum protection threshold is exp; exp is an exponential function. The Wasserstein distance between the old and new strategies; Distance threshold; The coefficient is linear growth. This is the attenuation coefficient.

6. A human-machine collaborative dynamic scheduling system integrating a dual-layer optimization mechanism, characterized in that, The dual-layer optimization mechanism includes an offline optimization layer and an online real-time scheduling layer. The human-machine collaborative dynamic scheduling system is used to execute the human-machine collaborative dynamic scheduling method as described in any one of claims 1 to 5, including: The acquisition module is used to acquire the set of workpieces to be processed, workstations, and robot resources; each workpiece includes several processing tasks. An offline optimization module is used to generate an initial solution based on the workpiece set, workstation, and robot resources in the offline optimization layer; and to adaptively optimize the initial solution using a simulated annealing algorithm, and obtain the global optimal solution by comparing optimization objectives; wherein the initial solution includes the workstation to which the task belongs, the processing method, the assigned workers and / or robots, the processing method includes worker, robot, and human-machine collaborative processing; the optimization objectives include minimizing the maximum completion time, energy consumption, and worker load balancing; An online real-time scheduling module is used in the online real-time scheduling layer to take the global optimal solution as the initial scheduling scheme and to adjust the current scheduling scheme in real time using a multi-agent adaptive proximal strategy optimization algorithm. The multi-agent includes a task allocation agent, a resource scheduling agent, and a disturbance response agent. The task allocation agent is used for selecting the processing method; the resource scheduling agent is used for assigning tasks to corresponding workers and / or robots and selecting workstations; the disturbance response agent is used for detecting disturbance events and invoking the task allocation agent and resource scheduling agent to reallocate resources.

7. A storage medium, characterized in that, It stores a computer program for human-machine collaborative dynamic scheduling that integrates a two-layer optimization mechanism, wherein the computer program causes the computer to execute the human-machine collaborative dynamic scheduling method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the human-machine collaborative dynamic scheduling method as described in any one of claims 1 to 5.

Images