Human-robot collaborative assembly line scheduling method and system based on reinforcement learning

Abstract

The application provides a kind of man-machine collaborative assembly line scheduling method and system based on reinforcement learning, it is related to intelligent manufacturing technical field.The application is completed scheduling decision by the cooperation of job agent, worker agent and robot agent in the multi-agent collaborative reinforcement learning framework.The application constructs a multi-objective dynamic scheduling model, and solves the multi-objective dynamic scheduling model by online scheduling method based on multi-agent collaborative reinforcement learning;The objective function of the model is to minimize the weighted sum of average completion time and worker fatigue penalty, and the update formula of worker fatigue degree is defined to predict the fatigue value;In the global reward of agent collaborative reinforcement learning framework, the worker agent reward is designed according to the worker fatigue penalty.The application ensures that the worker is not arranged task when fatigue is close to threshold value through fatigue value prediction and fatigue threshold control, and avoids the health risk caused by overwork.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a human-machine collaborative assembly line scheduling method and system based on reinforcement learning. Background Technology

[0002] In the process of upgrading the intelligent manufacturing industry, workshop production models are gradually moving away from the traditional "reliance on full automation" and towards a hybrid human-machine collaborative model. This means that fully automated machines handle simple, repetitive processes to ensure basic production efficiency, while semi-automated machines require human intervention to complete highly flexible and complex core processes (such as precision assembly and customized processing). This model retains the high efficiency of automated equipment while leveraging the irreplaceable value of workers in dynamic decision-making and complex operations. It is widely used in multi-variety, small-batch production fields such as electronics manufacturing, aerospace, and automotive parts. The corresponding flexible workshop scheduling problem has also become a key technological breakthrough for enhancing the competitiveness of manufacturing systems.

[0003] Most existing flexible workshop scheduling schemes do not take worker fatigue into account. Some schemes do consider worker fatigue, but only as a constraint, without linking fatigue to the quantitative relationship between processing time. They only consider taking rest measures when the worker's fatigue value exceeds a set threshold. The optimized scheduling scheme obtained in this way may result in workers being assigned new tasks when they are close to their fatigue limit, which can easily lead to fatigue exceeding the limit, affecting work efficiency and increasing safety risks.

[0004] As described above, the existing flexible workshop scheduling scheme does not link the quantitative relationship between fatigue and processing time, resulting in a large deviation between the calculated scheduling scheme and the actual efficiency. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a human-machine collaborative assembly line scheduling method and system based on reinforcement learning, which solves the technical problem that existing flexible workshop scheduling schemes do not link the quantitative relationship between fatigue and processing time.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: Firstly, the present invention provides a human-machine collaborative assembly line scheduling method based on reinforcement learning. This method utilizes a multi-agent collaborative reinforcement learning framework to collaboratively complete scheduling decisions involving a task agent, a worker agent, and a robot agent. The human-machine collaborative assembly line scheduling method includes: S1. Perform environmental status awareness. If a new job arrives and there is an available workstation, proceed to the next step; otherwise, continue executing S1. S2. Multiple agents allocate workstations and select processing modes for new tasks. The processing modes include worker processing mode, robot processing mode, and human-machine collaborative processing mode. S3. The workstation allocation and processing mode selection are sent to the assembly line for execution, and the actions, next global state, rewards of each agent and global reward are collected as training data for the multi-agent collaborative reinforcement learning framework update training. The multi-agent cooperative reinforcement learning framework includes: constructing a multi-objective dynamic scheduling model, solving the multi-objective dynamic scheduling model through an online scheduling method based on multi-agent cooperative reinforcement learning; the objective function of the multi-objective dynamic scheduling model is to minimize the weighted sum of the average completion time and the worker fatigue penalty, and an update formula for worker fatigue is defined to predict the fatigue value; the constraints of the model include ensuring that the worker's fatigue value is always controlled below a safety threshold; the global reward in the agent cooperative reinforcement learning framework is set according to the objective function, and the worker agent reward is designed based on the worker fatigue penalty.

[0007] Preferably, the expression of the objective function is as follows: in, Indicate homework Completion time; Represents a set of jobs. , n Indicates the total number of assignments; Indicates the assembly of workers. ; Indicates workers w In time t Fatigue level; This indicates the safe threshold for worker fatigue.

[0008] Preferably, the task in the objective function Completion time The calculation method is as follows: in, Indicate homework Start processing time; Indicate homework At workstation The actual processing time for the upper processing; The calculation methods include: When the processing mode is worker or human-machine collaborative processing mode: When the processing mode is robot processing mode: In the formula: in, This indicates the effect of fatigue on processing time; Indicate homework At workstation The baseline processing time; This represents the learning rate parameter; Indicates the fatigue effect coefficient; Indicates workers The initial skill level; Indicates processing mode The efficiency coefficient; Assignment: 0-1 variables +1 at the workstation The above pattern The value is 1 if processed, otherwise it is 0; Indicates workers In time Fatigue level; Indicates workers In time Proficiency; Indicates workers In time Proficiency; Indicates the time Until then, the workers The total number of assignments completed; Indicates the time Until then, the workers The total number of assignments completed; Indicate homework ; Indicate homework Start processing time; Indicate homework Start processing time; Indicate homework Start processing time; Indicate homework j＋1 The start time is greater than the assignment j The value is 1 if the end time is 1, otherwise it is 0; For worker mode, Human-machine collaboration mode.

[0009] Preferably, the update formula for worker fatigue includes a fatigue accumulation model and a fatigue recovery model, specifically: After the work is completed, the fatigue level of the workers is updated using a fatigue accumulation model. Before processing, a fatigue recovery model is used to update worker fatigue levels. This update is applied when workers are performing processing tasks. j The last assignment before was i ,but The expression is as follows: in, Indicate homework ; Indicate homework Start processing time; Represents a set of jobs. , n Indicates the total number of assignments; Indicates the assembly of workers. ; Indicates workers w After completing the assignment Post-fatigue level; Indicates workers w In processing operations j Previous worker fatigue; Refers to the natural constant; This represents the fatigue accumulation rate parameter; This represents the fatigue recovery rate parameter.

[0010] Preferably, the task agent, worker agent, and robot agent in the multi-agent cooperative reinforcement learning framework adopt an improved centralized training and distributed execution paradigm, specifically: During the training phase, global state information is obtained through a central Critic network to evaluate the overall state value and calculate the global advantage. The global advantage is then fused with the local advantages of the agents to guide the optimization of each agent's strategy to be consistent with the system goal. During the execution phase, each agent makes decisions independently based solely on its own observations.

[0011] Preferably, the constraints of the multi-objective dynamic scheduling model include: Constraint 1.1 stipulates that the start time of the operation must not be less than the arrival time of the operation; Constraint 1.2 stipulates that each operation can only be uniquely assigned to one workstation and selected from one processing mode; Constraint 1.3 stipulates that if the operation... At workstation When selecting worker or human-machine collaboration mode, there must be an idle worker; constraint 1.4 stipulates that if the operation... At workstation When selecting robot or human-robot collaboration mode, there must be an idle robot; Constraint 1.5 stipulates that a workstation can process a maximum of one job at the same time, and the processing cannot be interrupted; Constraint 1.6 stipulates that a worker can work at a maximum of one workstation at the same time; Constraint 1.7 stipulates that a robot can work at a maximum of one workstation at the same time; Constraint 1.8 stipulates that the worker's fatigue cannot exceed a threshold when the job is completed. in, Represents a set of jobs. , n Indicates the total number of assignments; Indicates the assembly of workers. , Indicates the total number of workers; Represents a collection of robots. , Indicates the total number of robots; Indicate homework The set of available workstations; Represents a set of processing modes. }, For worker mode, For robot mode, Human-machine collaboration mode; Indicates the current system time; Indicate homework Start processing time; Indicate homework Arrival time; Operation This indicates the safe threshold for worker fatigue. Assignment: 0-1 variables At workstation The above pattern The value is 1 if processed, otherwise it is 0; For 0-1 variables, in time ,Worker Assigned to work station It is 1 if it is true, otherwise it is 0. For 0-1 variables, in time ,robot Assigned to work station It is 1 if it is true, otherwise it is 0. Indicates workers In time Fatigue level; Indicate homework At workstation The actual processing time for the upper processing; Indicates workers w In processing operations j Subsequent worker fatigue; Indicates the current system time exist and It is 1 when it is between 0 and 1, otherwise it is 0.

[0012] Preferably, the global reward of the agent cooperative reinforcement learning framework includes: In the formula, For global rewards, This represents the average completion time of the currently completed tasks. Indicates hyperparameters, Indicates workers w In time t Fatigue level; This indicates the safe threshold for worker fatigue.

[0013] Secondly, the present invention provides a human-machine collaborative assembly line scheduling system based on reinforcement learning, wherein the human-machine collaborative assembly line scheduling system is used to execute the human-machine collaborative assembly line scheduling method based on reinforcement learning as described above.

[0014] Thirdly, the present invention provides a computer-readable storage medium storing a computer program for human-machine collaborative assembly line scheduling based on reinforcement learning, wherein the computer program causes a computer to execute the human-machine collaborative assembly line scheduling method based on reinforcement learning as described above.

[0015] Fourthly, the present invention provides an electronic device, comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing reinforcement learning-based human-machine collaborative assembly line scheduling as described above.

[0016] (III) Beneficial Effects This invention provides a human-machine collaborative assembly line scheduling method and system based on reinforcement learning. Compared with existing technologies, it has the following advantages: This invention uses fatigue value prediction and fatigue threshold control to ensure that workers are not assigned tasks when their fatigue level is close to the threshold, thus avoiding the health risks associated with overwork. This mechanism effectively protects worker health and prevents workplace accidents. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a multi-agent collaborative reinforcement learning framework employing centralized training and distributed execution in an embodiment of the present invention; Figure 2 This is a schematic diagram showing the decision-making time points of three types of heterogeneous intelligent agents. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This application provides a reinforcement learning-based human-machine collaborative assembly line scheduling method and system, which solves the technical problem that existing flexible workshop scheduling schemes do not link the quantitative relationship between fatigue and processing time. It introduces a dynamic fatigue threshold control mechanism to ensure a balance between production efficiency and worker health.

[0021] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: Existing flexible workshop scheduling solutions have the following main drawbacks: 1. Most existing methods do not consider worker fatigue, or only implement rest measures when worker fatigue exceeds a set threshold, without proactive control during the task allocation phase. In this model, workers are still assigned new tasks when they are close to their fatigue limit (but have not yet reached the threshold), leading to continuous accumulation of fatigue during task execution and exceeding the safety threshold. Both of these situations ultimately lead to common problems: worker fatigue causes a decrease in processing speed and an increase in operational errors, directly affecting work efficiency and resulting in a large deviation between the calculated scheduling scheme and the actual efficiency; furthermore, exceeding fatigue limits significantly increases the probability of workers' inattention and operational errors, thereby increasing safety risks such as workplace injuries.

[0022] 2. Existing scheduling models typically use a fixed baseline processing time, neglecting the learning effect of workers gradually improving their skills and shortening processing time through repetitive tasks, and also failing to consider the fatigue effect caused by long processing times. This static modeling approach leads to biases in processing time prediction, reducing the accuracy and effectiveness of scheduling decisions.

[0023] 3. Existing methods often treat robots as independent resources for allocation, failing to achieve dynamic collaboration by considering worker fatigue levels. High-workload workers continue to be assigned tasks despite accumulating fatigue risks, while robots fail to adequately share the load, leading to a decrease in the overall collaborative efficiency of the system.

[0024] 4. Existing research on the selection of processing modes—manual, robotic, and human-machine collaborative—generally lacks a unified optimization framework that links mode decision-making with worker fatigue prediction, skill learning dynamics, and long-term production goals such as worker skill improvement. Many methods still determine processing modes based on static rules or isolated optimization sub-problems, failing to balance short-term capacity, long-term skill accumulation, and health risks at the system level, thus resulting in resource utilization and long-term performance not being simultaneously optimal.

[0025] In summary, existing technologies fail to achieve the organic integration of worker fatigue control, dynamic skill evolution, and multi-mode resource scheduling, resulting in insufficient overall system efficiency and difficulty in effectively mitigating worker health risks. To address these issues, this invention proposes a reinforcement learning-based human-machine collaborative assembly line scheduling method and system based on deep reinforcement learning and a dynamic fatigue threshold control mechanism. By constructing a ternary coupling model of "worker skill evolution—fatigue control—processing mode selection," combined with a multi-agent online decision-making architecture and a multi-objective optimization mechanism, the core objectives are achieved: significantly improving the real-time performance and robustness of scheduling while ensuring dynamic job arrival and uninterrupted processing constraints; balancing production efficiency and worker health; and achieving long-term stable operation of the production system. Specific objectives include: 1. Introduce a dynamic fatigue threshold control mechanism: Proactively predict worker fatigue levels during the job assignment phase. When a worker's fatigue level approaches a safe threshold, no new tasks will be assigned. Combined with rest strategies, this will ensure worker health and reduce the risk of excessive fatigue.

[0026] 2. Establish a worker skill evolution model: By introducing a quantitative model of learning effects, the prediction of job processing time is dynamically adjusted, so that the scheduling system can reflect the changes in workers' skill levels with the accumulation of experience, thereby improving the accuracy and efficiency of task allocation.

[0027] 3. Design a multi-mode processing scheduling mechanism: dynamically switch between three modes: manual, robot, and human-machine collaboration, and use strategies obtained through reinforcement learning training to balance efficiency, health, and fairness, so as to achieve reasonable allocation of different resources and improve the overall collaborative efficiency of the system.

[0028] 4. Construct an online decision-making architecture based on deep reinforcement learning: By utilizing deep reinforcement learning methods and combining worker skill dynamics, fatigue status, and the uncertainty of job arrival, the scheduling strategy is adjusted in real time to achieve rapid response to dynamic events such as random job arrival and uneven workstation load, thereby reducing capacity loss and improving the real-time performance and robustness of the intelligent manufacturing system.

[0029] 5. Achieve multi-objective comprehensive optimization: Through a multi-objective optimization framework, the main objective is to minimize the average completion time, and worker fatigue penalty is incorporated into the optimization objective function to ensure a long-term balance between production efficiency and worker health, and avoid the efficiency-health imbalance problem under the guidance of a single objective.

[0030] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0031] This invention provides a reinforcement learning-based human-machine collaborative assembly line scheduling method. This method utilizes a multi-agent collaborative reinforcement learning framework, in which job agents, worker agents, and robot agents collaborate to make scheduling decisions. The reinforcement learning-based human-machine collaborative assembly line scheduling method includes: S1. Perform environmental status awareness. If a new job arrives and there is an available workstation, proceed to the next step; otherwise, continue executing S1. S2. Multiple agents allocate workstations and select processing modes for new tasks. The processing modes include worker processing mode, robot processing mode, and human-machine collaborative processing mode. S3. The workstation allocation and processing mode selection are sent to the assembly line for execution, and the actions, next global state, rewards of each agent and global reward are collected as training data for the multi-agent collaborative reinforcement learning framework update training. The multi-agent cooperative reinforcement learning framework includes: constructing a multi-objective dynamic scheduling model, solving the multi-objective dynamic scheduling model through an online scheduling method based on multi-agent cooperative reinforcement learning; wherein the objective function of the multi-objective dynamic scheduling model is to minimize the weighted sum of the average completion time and worker fatigue penalty, and a fatigue update formula is defined to predict the fatigue value; the model's constraints include ensuring that the worker's fatigue value is always controlled below a safe threshold; the global reward in the agent cooperative reinforcement learning framework is set according to the objective function, and the worker agent reward is designed based on the worker fatigue penalty. The multi-agent cooperative reinforcement learning framework is as follows: Figure 1 As shown.

[0032] This invention, through fatigue value prediction and fatigue threshold control, ensures that workers are not assigned tasks when their fatigue level is close to the threshold, thus avoiding the health risks associated with overwork. This mechanism effectively protects worker health and prevents workplace accidents.

[0033] The following is a detailed description of each step: In one embodiment, S1 involves environmental status awareness. If a new task arrives and an available workstation exists, the next step is executed; otherwise, S1 continues. The specific implementation process is as follows: In reinforcement learning, the environmental state refers to the current situation of the environment in which the agent exists, and it serves as the basis for the agent's decision-making. It contains all the key information related to the decision. In this embodiment of the invention, the environmental state refers to the global state space of the workshop, including features such as job arrival status, worker skill level, worker fatigue value, robot availability, and workstation load rate.

[0034] In one embodiment, S2, a multi-agent system, allocates workstations and selects processing modes for new tasks. The processing modes include worker processing mode, robot processing mode, and human-machine collaborative processing mode. The specific implementation process is as follows: Activate the work agent to select the workstation and processing mode. When the selected processing mode is worker processing mode, activate the worker agent to select a suitable worker; when the selected processing mode is robot processing mode, activate the robot agent to select a suitable robot; when the human-machine collaborative processing mode is selected, activate the worker agent to select a suitable worker and activate the robot agent to select a suitable robot.

[0035] In this embodiment of the invention, a multi-objective dynamic scheduling model is constructed within a multi-agent cooperative reinforcement learning framework. The objective function of this model is to minimize the weighted sum of the average completion time and worker fatigue penalty. The global reward of the agent cooperative reinforcement learning framework is set according to the objective function, and the worker agent reward is designed based on the worker fatigue penalty. The three types of agents—task agent, worker agent, and robot agent—each include independent Actor networks and Critic networks. Simultaneously, the multi-agent cooperative reinforcement learning framework includes a centralized Critic network for fusing global information and assisting in collaborative decision-making.

[0036] The multi-objective dynamic scheduling model includes an objective function and constraints, as detailed below: in, Indicates worker w at time t The fatigue level is determined based on the processing time t. The fatigue level is updated once before the processing begins and once after processing is completed, as detailed below: After the work is completed, the worker fatigue level is updated using a fatigue accumulation model: Before the operation, the fatigue level of the workers is updated using a fatigue recovery model. This assumes that the workers... In processing operations The last assignment before was : In this embodiment of the invention, the objective function also considers the impact of worker skill evolution and fatigue accumulation on average completion time, specifically reflected in... The calculation method is as follows: in, Indicate homework At workstation The actual processing time for the upper processing. Indicate homework Start processing time; Calculate the actual processing time by considering overall skill level, learning effect, fatigue effect, and processing mode. : In worker or human-machine collaboration mode: Robot Mode: Among them, formula (7) quantifies the impact of fatigue on processing time; formula (8) defines the worker learning effect formula, which shows that as the number of jobs completed by the worker increases, the worker's skills gradually improve and the processing time decreases; formula (9) defines the update of the number of processing times; formulas (8) and (9) are the learning effect model formulas constructed using the first formal learning curve model Wright.

[0037] The constraints include: Constraint (10) stipulates that the start time of the job must be greater than the arrival time of the job; Constraint (11) stipulates that each job can only be uniquely assigned to one workstation and selected one processing mode; Constraint (12) stipulates that if the job At workstation When selecting worker or human-machine collaboration mode, there must be an idle worker; constraint (13) stipulates that if the operation At workstation When selecting robot or human-machine collaboration mode, there must be an idle robot; constraint (14) stipulates that a workstation can process at most one job at the same time, and the processing cannot be interrupted; constraint (15) stipulates that a worker can work at most one workstation at the same time; constraint (16) stipulates that a robot can work at most one workstation at the same time; constraint (17) stipulates that the worker's fatigue cannot exceed the threshold when the job is completed.

[0038] in, Represents a set of jobs. , n Indicates the total number of assignments; Represents a collection of robots. , Indicates the total number of robots; Indicates the assembly of workers. , Indicates the total number of workers; Indicate homework The set of available workstations; Represents a set of processing modes. }, For worker mode, For robot mode, Human-machine collaboration mode; Indicates the current system time; Indicate homework Start processing time; Indicate homework Arrival time; Operation Indicate homework At workstation The baseline processing time; This represents the learning rate parameter; This represents the fatigue accumulation rate parameter; This parameter represents the fatigue recovery rate. This indicates the safe threshold for worker fatigue. Indicates the fatigue effect coefficient; Indicates workers The initial skill level; Indicates processing mode The efficiency coefficient; Assignment: 0-1 variables At workstation The above pattern The value is 1 if processed, otherwise it is 0; For 0-1 variables, in time ,Worker Assigned to work station It is 1 if it is true, otherwise it is 0. For 0-1 variables, in time ,robot Assigned to work station It is 1 if it is true, otherwise it is 0. Indicates workers In time Fatigue level; Indicate homework At workstation The actual processing time for the upper processing; Indicates workers In time Proficiency (learning effect) Indicates workers In time Proficiency; Indicates workers In time Proficiency; Indicates the time Until then, the workers The total number of assignments completed; Indicates the time Until then, the workers The total number of assignments completed; Indicates the time Until then, the workers The total number of assignments completed; Indicates the current system time Is it in and If the value is between 1 and 0, then the value is 1; otherwise, the value is 0.

[0039] To achieve multi-objective dynamic optimization scheduling, this invention proposes a multi-agent proximal policy optimization algorithm (a reinforcement learning algorithm) based on collaborative rewards and information sharing. This algorithm maps complex scheduling decision-making processes through multiple collaborative agents and ensures that the scheduling system improves efficiency while protecting worker health through a unique reward design mechanism.

[0040] The specific design of the multi-agent proximal policy optimization algorithm is as follows: Three types of heterogeneous intelligent agents collaborate to complete scheduling decisions, with decision-making times as follows: Figure 2 As shown.

[0041] The three types of heterogeneous intelligent agents include task agents, worker agents, and robot agents. The task agent, as the core decision-making unit, is activated whenever a new task arrives and an available workstation exists. Its responsibility is to select a processing workstation and decide on the processing mode based on the current state. The worker agent, as an auxiliary decision-making unit, is activated only when the task agent selects "worker" or "human-machine collaboration mode." Its responsibility is to select the most suitable worker from the currently available workers and assign it to the task. The robot agent, also as an auxiliary decision-making unit, is activated only when the task agent selects "robot" or "human-machine collaboration mode." Its responsibility is to select the most suitable robot from the currently available robots and assign it to the task. The state space of the three types of heterogeneous intelligent agents consists of a global state and various local states, ensuring that the decision-making has both a global perspective and local specificity. The state characteristics are designed as shown in Table 1.

[0042] Table 1 State Feature Design Each agent's action space consists of a set of predefined scheduling rules. The policy network learns which rule is optimal under what conditions, as shown in Table 2.

[0043] Table 2 Motion Space Design In this embodiment of the invention, a Centralized Training with Decentralized Execution (CTDE) paradigm is used to train and utilize three intelligent agents—a worker agent and a robot agent. Furthermore, a collaborative reward and information sharing mechanism is introduced, forming the CTDE-Co-RSI learning paradigm, the core architecture of which is as follows: Figure 1 As shown.

[0044] During the training phase, a centralized network of critics can acquire global state information, including the load of all workstations, worker fatigue, robot status, and job queues. It is used to evaluate the value of the joint actions of three intelligent agents: the task agent, the worker agent, and the robot agent, thereby guiding the collaborative optimization of the strategies of each agent.

[0045] During the execution phase, each agent makes decisions independently based solely on its own observations, without relying on a central Critic, thus enhancing the system's reliability and real-time performance.

[0046] Each agent's Actor network observations contain statistical features of the global state, enabling the agent to perceive the overall operating status of the system when making decisions.

[0047] Meanwhile, in this embodiment of the invention, to address the problem of reward sparsity caused by different decision frequencies, a unique reward function is designed for each of the three heterogeneous agents, providing precise guidance signals for their behavior. Specifically: The reward function consists of two parts: a global reward and a local reward. These two parts together guide the agent in learning the target policy. The global reward, consistent with the target function, is used for training the central Critic network. In the formula, For global rewards, The hyperparameter represents the average completion time of currently completed tasks. Used to measure the importance of completion time and worker fatigue control.

[0048] Customized rewards are provided for each intelligent agent, offering precise decision-making signals: Rewards for the task-oriented intelligent agent: Worker agent rewards: Robot agent rewards: In the formula, Indicates workers Fatigue level, This indicates the time elapsed from the end of the robot's last task to the current decision-making moment.

[0049] In reinforcement learning, the core of policy optimization lies in accurately calculating the advantage function, which measures the performance of a state. Take action below Compared to the superiority of the average strategy, this invention employs a hybrid advantage estimation method and introduces a collaborative fusion mechanism to address the varying decision frequencies of agents in the system.

[0050] The task agent is activated at each decision point, and its decision trajectory is continuous. The Generalized Advantage Estimation (GAE) method is used to balance the low bias and high variance of the estimation.

[0051] The task agent at time step Advantage estimation Calculation formula and timing difference error The calculation formula is: in, This is a discount factor used to measure the importance of current rewards versus future rewards; These are GAE parameters used to adjust the tradeoff between variance and bias in advantage estimation; For the task agent at time step The instant reward received; For the Critic network of the task agent in the state The output state value function estimate represents the expected cumulative return that can be obtained by following the current policy.

[0052] Worker agents and robot agents are activated only after a specific pattern is selected, and their decision trajectories are sparse and discontinuous. To address the problem of reward sparsity, an advantage estimation method based on Monte Carlo rewards and value baselines is adopted.

[0053] worker agents at time steps Advantage estimation Calculation formula and cumulative return on discount The calculation formula is: in, The discounted cumulative reward for the worker agent from the start to the end of the current time step is used as an estimate of the action value function; For worker intelligent agents in time step The rewards received; For worker agents, the Critic network in state The output is the estimated value of the state value function.

[0054] The formulas for estimating the advantage of robotic agents and calculating the cumulative discounted returns are consistent with those for worker agents, so they will not be elaborated further.

[0055] It should be noted that, in specific embodiments, to promote cooperation among different agents and prevent them from pursuing individual rewards at the expense of the overall interests, this invention introduces a collaborative advantage fusion mechanism. The final advantage function of each agent is jointly determined by its own advantage and the overall system advantage.

[0056] Overall advantage And the final advantage estimation of the agent The formula is: in, The advantage estimates calculated for each agent; The global advantage estimate calculated for the central Critic network is used to measure the degree of advantage of the current state in achieving the overall system goal; , Let be the collaborative weight coefficient, and satisfy... .

[0057] In one embodiment, S3, the workstation allocation and processing mode selection are sent to the assembly line for execution, and the actions, next global state, rewards for each agent, and global reward are collected as training data for updating the multi-agent collaborative reinforcement learning framework. The specific implementation process is as follows: Actor network parameters for each agent Update by maximizing the pruning objective function of proximal policy optimization (PPO): Among them, the probability ratio The calculation formula is: in, and The policy network before and after the update is in state. Select action The probability of; This is the clipping range parameter, used to limit the magnitude of each policy update.

[0058] The pseudocode for the training and execution process of the algorithm described above in this embodiment of the invention is shown in Table 3: Table 3. Pseudocode of the training and execution process of the multi-agent proximal policy optimization algorithm. This invention also provides a reinforcement learning-based human-machine collaborative assembly line scheduling system, which is used to execute the reinforcement learning-based human-machine collaborative assembly line scheduling method described above.

[0059] This invention also provides a computer-readable storage medium storing a computer program for reinforcement learning-based human-machine collaborative assembly line scheduling, wherein the computer program causes a computer to execute the reinforcement learning-based human-machine collaborative assembly line scheduling method as described above.

[0060] This invention also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the reinforcement learning-based human-machine collaborative assembly line scheduling method as described above.

[0061] In summary, compared with existing technologies, it has the following beneficial effects: 1. This invention, through fatigue value prediction and fatigue threshold control, ensures that workers are not assigned tasks when their fatigue level is close to the threshold, thus avoiding the health risks associated with overwork. This mechanism effectively protects worker health and prevents production accidents.

[0062] 2. This invention combines the learning effect with fatigue control, helping workers continuously improve their skills while avoiding fatigue accumulation caused by overwork. Improved worker skills shorten processing time and reduce the negative impact of fatigue on production efficiency.

[0063] 3. This invention employs multi-agent deep reinforcement learning, which adaptively adjusts job allocation and resource scheduling strategies through real-time feedback, enhancing the production line's responsiveness and adaptability to changing demands. Simultaneously, through multi-objective optimization, it considers worker fatigue control while optimizing production efficiency. This balance effectively avoids fatigue accumulation caused by single-objective optimization, thereby improving the system's long-term stability. Furthermore, rational scheduling based on the characteristics of each job and the availability of workstations, through efficient resource matching, increases job completion speed while avoiding resource idleness, ensuring production continuity and maximizing resource utilization.

[0064] 4. This invention, through the introduction of a multi-Critic collaborative fusion mechanism, achieves more accurate credit allocation and more efficient agent collaboration. Dedicated Critics ensure that the learning signals received by each agent not only focus on the overall system goal but also better reflect the unique contribution of its specific decisions to the goal, thereby accelerating the strategy optimization process and improving the quality of the final strategy. Compared to a single global Critic, this mechanism can more effectively balance the conflicts between multiple goals such as production efficiency and worker health, avoiding agent behavior from falling into local optima or exhibiting "free-riding" phenomena. Ultimately, this enables the scheduling system to exhibit higher collaborative intelligence, decision-making accuracy, and overall stability in dynamic and complex environments.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human-robot collaborative assembly line scheduling based on reinforcement learning, characterized in that, The human-machine collaborative assembly line scheduling method utilizes a multi-agent collaborative reinforcement learning framework to collaboratively complete scheduling decisions among the task agent, worker agent, and robot agent. The human-machine collaborative assembly line scheduling method includes: S1. Perform environmental status awareness. If a new job arrives and there is an available workstation, proceed to the next step; otherwise, continue executing S1. S2. Multiple agents allocate workstations and select processing modes for new tasks. The processing modes include worker processing mode, robot processing mode, and human-machine collaborative processing mode. S3. The workstation allocation and processing mode selection are sent to the assembly line for execution, and the actions, next global state, rewards of each agent and global reward are collected as training data for the multi-agent collaborative reinforcement learning framework update training. The multi-agent cooperative reinforcement learning framework includes: constructing a multi-objective dynamic scheduling model, solving the multi-objective dynamic scheduling model through an online scheduling method based on multi-agent cooperative reinforcement learning; the objective function of the multi-objective dynamic scheduling model is to minimize the weighted sum of the average completion time and the worker fatigue penalty, and an update formula for worker fatigue is defined to predict the fatigue value; the constraints of the model include ensuring that the worker's fatigue value is always controlled below a safe threshold; the global reward in the agent cooperative reinforcement learning framework is set according to the objective function, and the worker agent reward is designed based on the worker fatigue penalty; The expression for the objective function is as follows: wherein, represents a completion time of a job; represents a set of jobs, , n represents a total number of jobs; represents a set of workers, ; represents a fatigue degree of a worker w at a time t ; represents a fatigue degree safety threshold of a worker;​ The task in the objective function Completion time The calculation method is as follows: in, Indicate homework Start processing time; Indicate homework At workstation The actual processing time for the upper processing; The calculation methods include: When the processing mode is worker or human-machine collaborative processing mode: When the processing mode is robot processing mode: In the formula: in, This indicates the effect of fatigue on processing time; Indicate homework At workstation The baseline processing time; This represents the learning rate parameter; Indicates the fatigue effect coefficient; Indicates workers The initial skill level; Indicates processing mode The efficiency coefficient; Assignment: 0-1 variables +1 at the workstation The above pattern The value is 1 if processed, otherwise it is 0; Indicates workers In time Fatigue level; Indicates workers In time Proficiency; Indicates workers In time Proficiency; Indicates the time Until then, the workers The total number of assignments completed; Indicates the time Until then, the workers The total number of assignments completed; Indicate homework ; Indicate homework Start processing time; Indicate homework Start processing time; Indicate homework Start processing time; Indicate homework j＋1 The start time is greater than the assignment j The value is 1 if the end time is 1, otherwise it is 0; For worker mode, Human-machine collaboration mode; The update formula for worker fatigue includes a fatigue accumulation model and a fatigue recovery model, specifically: After the work is completed, the fatigue level of the workers is updated using a fatigue accumulation model. Before processing, a fatigue recovery model is used to update worker fatigue levels. This update is applied when workers are performing processing tasks. j The last assignment before was i ,but The expression is as follows: in, Indicate homework ; Indicate homework Start processing time; Represents a set of jobs. , n Indicates the total number of assignments; Indicates the assembly of workers. ; Indicates workers w After completing the assignment Post-fatigue level; Indicates workers w In processing operations j Previous worker fatigue; Indicates workers w After completing the assignment Post-fatigue level; Refers to the natural constant; This represents the fatigue accumulation rate parameter; This represents the fatigue recovery rate parameter.

2. The reinforcement learning-based human-machine collaborative assembly line scheduling method as described in claim 1, characterized in that, The multi-agent cooperative reinforcement learning framework employs an improved centralized training and distributed execution paradigm for its task agent, worker agent, and robot agent, specifically: During the training phase, global state information is obtained through a central Critic network to evaluate the overall state value and calculate the global advantage. The global advantage is then fused with the local advantages of the agents to guide the optimization of each agent's strategy to be consistent with the system goal. During the execution phase, each agent makes decisions independently based solely on its own observations.

3. The reinforcement learning-based human-machine collaborative assembly line scheduling method as described in any one of claims 1 to 2, characterized in that, The constraints of the multi-objective dynamic scheduling model include: Constraint 1.1 stipulates that the start time of the operation must not be less than the arrival time of the operation; Constraint 1.2 stipulates that each operation can only be uniquely assigned to one workstation and selected from one processing mode; Constraint 1.3 stipulates that if the operation... At workstation When selecting worker or human-machine collaboration mode, there must be an idle worker; constraint 1.4 stipulates that if the operation... At workstation When selecting robot or human-robot collaboration mode, there must be an idle robot; Constraint 1.5 stipulates that a workstation can process a maximum of one job at the same time, and the processing cannot be interrupted; Constraint 1.6 stipulates that a worker can work at a maximum of one workstation at the same time; Constraint 1.7 stipulates that a robot can work at a maximum of one workstation at the same time; Constraint 1.8 stipulates that the worker's fatigue cannot exceed a threshold when the job is completed. in, Represents a set of jobs. , n Indicates the total number of assignments; Indicates the assembly of workers. , k Indicates the total number of workers; Represents a collection of robots. , Indicates the total number of robots; Indicate homework The set of available workstations; Represents a set of processing modes. }, For worker mode, For robot mode, Human-machine collaboration mode; Indicates the current system time; Indicate homework Start processing time; Indicate homework Arrival time; Operation This indicates the safe threshold for worker fatigue. Assignment: 0-1 variables At workstation The above pattern The value is 1 if processed, otherwise it is 0; For 0-1 variables, in time ,Worker Assigned to work station It is 1 if it is true, otherwise it is 0; For 0-1 variables, in time ,robot Assigned to work station It is 1 if it is true, otherwise it is 0. Indicates workers In time Fatigue level; Indicate homework At workstation The actual processing time for the upper processing; Indicates workers w In processing operations j Subsequent worker fatigue; Indicates the current system time exist and It is 1 when it is between 0 and 1, otherwise it is 0.

4. The reinforcement learning-based human-machine collaborative assembly line scheduling method as described in any one of claims 1 to 2, characterized in that, The global reward of the agent-cooperative reinforcement learning framework includes: In the formula, For global rewards, This represents the average completion time of the currently completed tasks. Indicates hyperparameters, Indicates workers w In time t Fatigue level; This indicates the safe threshold for worker fatigue.

5. A human-machine collaborative assembly line scheduling system based on reinforcement learning, characterized in that, The human-machine collaborative assembly line scheduling system is used to execute the reinforcement learning-based human-machine collaborative assembly line scheduling method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, It stores a computer program for reinforcement learning-based human-machine collaborative assembly line scheduling, wherein the computer program causes the computer to execute the reinforcement learning-based human-machine collaborative assembly line scheduling method as described in any one of claims 1 to 4.

7. An electronic device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing reinforcement learning-based human-machine collaborative assembly line scheduling as described in any one of claims 1 to 4.

Images