Man-machine cooperation task allocation method based on reinforcement learning

Abstract

The invention discloses a man-machine cooperation task allocation method based on reinforcement learning, and the method comprises the steps: constructing a reinforcement learning model which comprises a task definition, a resource definition, a state space, an action set and a reward function, and carrying out the modeling of a cooperation relation among a task, a worker and a robot in man-machine cooperation through a heterogeneous graph neural network, and extracting high-order features of the state space. Based on the characteristics, a strategy network and a value network are trained through a near-end strategy optimization algorithm, and learning of an efficient task allocation strategy is achieved. The reward function combines the task completion duration and the resource utilization rate, and gives consideration to the production efficiency and resource balance. In the system operation process, workshop state data are collected in real time to dynamically update the state space, and the reinforcement learning model generates an optimal allocation action according to the latest state and issues an instruction to a worker or a robot for execution. The method has the advantages of high adaptability, high distribution efficiency, high scheduling intelligence degree and the like, and is suitable for task scheduling optimization in complex dynamic environments such as intelligent manufacturing and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing and industrial automation, and specifically relates to a human-machine collaborative task allocation method based on reinforcement learning. Background Art

[0002] In modern industrial production, human-robot collaborative workshops are becoming the mainstream production model. In this model, workers and robots work together to complete production tasks, leveraging their respective strengths. Workers demonstrate high flexibility and adaptability when handling complex, unstructured tasks, while robots maintain high efficiency and precision in repetitive, regular tasks. However, how to rationally allocate tasks between workers and robots to maximize production efficiency while fully leveraging the advantages of human-robot collaboration remains a pressing issue.

[0003] Traditional task allocation methods primarily rely on fixed rules and heuristic algorithms. While these methods can meet production needs to a certain extent, they often exhibit significant limitations in the face of dynamically changing production environments. For example, when production tasks suddenly increase or equipment malfunctions, traditional task allocation methods struggle to adapt quickly, resulting in decreased production efficiency. Furthermore, these methods often fail to fully consider the real-time status of workers and robots, such as workload and fatigue levels. This can lead to irrational task allocation, impacting production efficiency and worker health.

[0004] In the prior art, Chinese patent CN118521130A discloses a fuzzy shop scheduling method and system based on deep reinforcement learning and graph neural networks, belonging to the field of fuzzy shop scheduling technology. This invention utilizes deep reinforcement learning and graph neural networks to effectively address the complexity and ambiguity of fuzzy shop scheduling problems. It extracts state change features during the scheduling process through a three-channel graph neural network, resolving the difficulties posed by ambiguity in feature extraction and graph information embedding. Furthermore, it utilizes a proximal strategy optimization algorithm to obtain an optimal decision network, achieving an efficient solution to the fuzzy shop scheduling problem and improving production efficiency.

[0005] However, this approach has the following limitations: Fuzzy shop floor scheduling methods primarily address task scheduling in traditional manufacturing environments and are unable to fully adapt to the complex demands of dynamic human-machine interaction and real-time state changes in modern collaborative workshops. Secondly, while graph neural networks can extract complex state features, they are still insufficient in modeling dynamic factors such as individual differences in workers, fatigue levels, and real-time performance fluctuations in robots. This can lead to irrational allocations in scheduling decisions in practical applications. Furthermore, this approach has poor adaptability to environmental changes and struggles to respond to emergencies such as equipment failures and urgent order adjustments, limiting its flexible application in actual production lines.

[0006] Therefore, how to design a human-machine collaborative workshop task scheduling method that can not only reflect the status of workers and robots in real time, adapt to dynamically changing environments, but also have efficient decision-making capabilities has become a key technical problem in improving the level of intelligence in modern manufacturing. Summary of the Invention

[0007] The purpose of the present invention is to provide a human-machine collaborative task allocation method based on reinforcement learning in order to overcome the defects of the above-mentioned prior art.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] The present invention provides a human-machine collaborative task allocation method based on reinforcement learning, comprising the following steps:

[0010] Build a reinforcement learning model for human-machine collaborative workshop reinforcement learning, including task definition, resource definition, state space, action set, and reward function;

[0011] A heterogeneous graph neural network is used to model the collaborative relationship of the state space of the reinforcement learning model, and the reinforcement learning model is trained with a proximal policy optimization algorithm to obtain a trained reinforcement learning model based on the state space input and output task allocation strategy.

[0012] The sensors and monitoring system in the workshop collect real-time workshop production data, including the current location of tasks, completed tasks, unfinished tasks, workers' working status, and robots' working status. The state space of the reinforcement learning model is updated based on the collected workshop production data.

[0013] The updated state space is input into the trained reinforcement learning model, which outputs the optimal task allocation action based on the current state space through the policy network.

[0014] According to the optimal task allocation action, the task instructions are sent to the corresponding workers or robots and other execution units through the workshop communication network to complete the task allocation and scheduling;

[0015] The task execution results are monitored in real time, and the actual execution effects are fed back to the reinforcement learning model as environmental interaction feedback to further optimize and adjust the task allocation strategy.

[0016] Furthermore, the task definition includes:

[0017] Task sequence op i,j , represents the jth operation in the i-th job;

[0018] Processing time matrix P = {p i,j,k}, indicating the task opi,j The time required to be executed by operator k, including workers and robots;

[0019] Task dependency matrix D = {d i,j,p,q}, where d i,j,p,q =1 indicates task op i,j Must be in op p,q You can start only after you finish it.

[0020] Furthermore, the resource definition includes:

[0021] Worker set W = {w1,w2,...w m}, where m is the number of workers;

[0022] Robot set R = {r1, r2, ...r n}, where n is the number of robots;

[0023] Collaboration unit set C = {c i,j |c i,j =(w i ,r j )}, indicating worker w i With robot r j The collaborative units are composed of i,j Including the collaborative efficiency coefficient η i,j , defined as:

[0024]

[0025] Among them, α is the collaboration weight coefficient, Represents workers w i and robot r j efficiency parameters.

[0026] Furthermore, the state space S includes:

[0027] Basic time state t current , indicating the current time;

[0028] Task state vector TS = {ts1, ts2, ... ts K}, where K is the total number of tasks, ts i Indicates the status of the i-th task. The status values ​​include: 0 for not started, 1 for in progress, and 2 for completed;

[0029] Worker state vector WS = {ws1, ws2, ..., ws m}, ws i represents the state of the i-th worker;

[0030] Robot state vector RS = {rs1,rs2,...rs n},rs j represents the state of the j-th robot;

[0031] Collaboration unit state matrix CS = {cs i,j}, indicating worker w i With robot r j The status of the composed collaborative units;

[0032] Worker busy end time vector Indicates the expected busy end time of each worker;

[0033] Robot busy end time vector Indicates the estimated busy end time of each robot;

[0034] Task completion time vector FT = {ft1, ft2, ..., ft K}, where ft i Indicates the completion time of task i. If the task is not completed, the value is 0 or the preset maximum value;

[0035] Worker utilization rate Robot utilization in and Represents workers w i With robot r j Busy time, T total Indicates the total production time.

[0036] Furthermore, the action set includes a task assignment action and a waiting action, wherein the task assignment action includes the following forms:

[0037] The task op i,j Assigned to worker w k , put the task op i,j Assigned to robot r l , put the task op i,j Assigned to collaborative unit c k,l ;

[0038] The waiting action indicates that no task is currently assigned, that is, no task assignment operation is performed;

[0039] The action is expressed in vector form as:

[0040] a=(op idx ,executor idx ,executor type )

[0041] Among them, opidx Indicates the index of the task to be assigned, executor idx Indicates the index of the executor, executor type Indicates the executor type. The value is: 0 for worker; 1 for robot; 2 for collaborative unit.

[0042] Furthermore, the reward function is:

[0043] R(s,a,s′)=-λ1·Makespan+λ2·U

[0044] Where R(s,a,s′) represents the reward value obtained by transferring from state space s to state space s′ through action a; Makespan represents the time point when the last task is completed in each scheduling scheme; λ1 and λ2 are weighting coefficients; and U is the resource utilization index:

[0045] U=β·U W +(1-β)·U R

[0046] in, is the average utilization rate of all workers, is the average utilization of all robots, and Represents workers w i With robot r j Busy time, T total represents the total production time, and β is the weight coefficient.

[0047] Furthermore, the use of heterogeneous graph neural networks to perform collaborative relationship modeling on the state space of the reinforcement learning model specifically includes:

[0048] According to the task definition and resource definition, the state space S of the human-machine collaborative workshop is modeled as a heterogeneous graph G = (V, E);

[0049] A heterogeneous graph neural network is used to model the heterogeneous graph G = (V, E), where each layer of the neural network performs message passing and feature update through the following steps:

[0050] For each pair of connected nodes v i ←v j , define the message passing function according to its node type and edge type:

[0051]

[0052] in, and They are node v i ,v jIn the feature representation of the l-1 layer, is the edge feature connecting them, φ (l) is a learnable message function such as a multilayer perceptron at layer l;

[0053] For node v i , collects messages from all its neighboring nodes and performs weighted aggregation based on node type:

[0054]

[0055] in, Indicates that in the l-th layer network, all neighbor nodes are aggregated to node v i The message result, N(v i ) is the node v i The neighbor set of is the attention weight, defined as

[0056]

[0057] Among them, W and a are the learnable parameter matrix and vector;

[0058] According to the aggregated message results Update each node feature:

[0059]

[0060] Where σ is the activation function, and Is for node type tgpe(v i )’s weight matrix and bias vector;

[0061] After L layers of message passing, the final node representation is obtained

[0062] The global graph representation is obtained through pooling operation:

[0063]

[0064] Pool can be average pooling, maximum pooling or attention pooling.

[0065] Furthermore, the heterogeneous graph G = (V, E), where:

[0066] The node set is V = V T ∪V W ∪V R ,include:

[0067] Task Node Set Each node represents a specific task op i,j, the task node characteristics include the current task state ts i , estimated processing time vector {p i,j,k}, represents the time required for execution by various operators and task dependency information, according to the task dependency matrix D = {d i,i,p,q}Get, task completion time ft i ;

[0068] Worker node set Each node represents a worker w i , the worker node features include the worker’s current state ws i , Workers' busy time ends Worker efficiency parameters

[0069] Robot Node Set Each node represents a robot r i , the robot node features include the robot’s current state rs j , Robot busy end time Robot efficiency parameters

[0070] The edge set is E=E T-T ∪E T-W ∪E T-R ∪E W-R ,include:

[0071] Inter-task dependency edge E T-T , represents the dependency relationship between tasks, if d i,j,p,q =1, that is, task op i,j Must be in op p,q Only after completion can it start, establishing directional edges between task nodes. The characteristics of inter-task dependency edges include dependency strength or priority;

[0072] Task-Worker Side E T-w , indicating that the task can be executed by workers, if the worker w k Can execute mission op i,j , then the task node and the worker node are connected, and the task-worker edge features include the worker processing time p i,j,k ;

[0073] Mission-Machine Edge E T-R , indicating that the task can be performed by the robot. If the robot r l Can execute mission op i,j , then the task node and the robot node are connected, and the task-machine edge features include the robot processing time p i,j,l ;

[0074] Worker-Robot Edge E W-R, indicating a relationship that can form a collaborative unit, if worker w i With robot r j Can form collaborative units cs i,j , then connect the worker node and the robot node, the worker-robot edge features include the collaboration efficiency coefficient η i,j .

[0075] Furthermore, the reinforcement learning model includes a policy network and a value network; and the training of the reinforcement learning model in combination with the proximal policy optimization algorithm specifically includes:

[0076] Obtain the global graph representation h after the state space is processed by heterogeneous graph neural network modeling G , input to the policy network and value network to generate policy distribution π θ (a|s) and state value V φ (s);

[0077] The policy network is:

[0078] π θ (a|s)=SoftmaxLayer(MLP(h G ))

[0079] Where MLP is the first multi-layer perceptron, θ is the policy network parameter, π θ (a|s) is the probability of selecting action a from the action set in state space s;

[0080] The value network is:

[0081] V φ (s) = MLP′(h G )

[0082] Among them, MLP′ is the second multi-layer perceptron, φ is the value network strategy, V φ (s) is the value estimate of the state space s;

[0083] The proximal policy optimization algorithm is used to train the policy network, and the objective function is:

[0084]

[0085] in, is the importance weight, is the advantage function estimate, ∈ is the clipping factor, clip is the clipping function, is the expected estimate at time step t;

[0086] The mean square error is used to train the value network, and the value function loss is:

[0087]

[0088] Among them, Vt t target is the estimated return, obtained according to the reward function R(s,a,s′),

[0089] Introduce the policy entropy regularization term:

[0090]

[0091] Among them, S[π θ |(s t )] is the strategy regardless of the state s t Entropy under

[0092] Construct a joint loss function based on the objective function, value function loss and policy entropy regularization term:

[0093] L(θ|φ)=L cLIP (θ)-c1L VF (φ)+c2S[π θ ]

[0094] Among them, c1 and c2 are hyperparameters that control the value loss and entropy regularization strength, and L(θ|φ) is the total loss function;

[0095] Backpropagation and gradient update are performed through the total loss function to optimize the policy network and value network parameters to obtain the trained reinforcement learning model.

[0096] Furthermore, the updated state space is input into the trained reinforcement learning model, and the reinforcement learning model outputs the optimal task allocation action through the policy network according to the current state space, specifically including:

[0097] The updated state space is processed by heterogeneous graph neural network modeling to obtain the global graph representation h G , the global graph is represented by h G Input into the policy network of the trained reinforcement learning model to obtain the optimal task allocation action.

[0098] Compared with the prior art, the present invention has the following advantages:

[0099] (1) The present invention solves the problem that the traditional task allocation method in the prior art relies on fixed rules and heuristic algorithms, is difficult to adapt to the dynamically changing production environment, and cannot fully consider the real-time status of workers and robots by constructing an environmental model for reinforcement learning in human-machine collaborative workshops. By clarifying the tasks, resources, status, actions and reward functions, the task allocation problem is transformed into a sequential decision problem, and dynamic adjustment and optimization of task allocation are achieved. The real-time status of workers and robots can be fully considered to achieve more reasonable and efficient task allocation. It has the advantages of being able to respond to changes in the production environment in real time, guiding the model to optimize task allocation through the reward function, and improving global production efficiency.

[0100] (2) The present invention solves the problem that traditional methods in the prior art cannot effectively handle the complex task and resource relationships in human-machine collaborative workshops and find the optimal task allocation strategy by utilizing heterogeneous graph neural networks and proximal policy optimization reinforcement learning algorithms. Heterogeneous graph neural networks can model and extract features of different types of nodes and edges, while the proximal policy optimization algorithm can quickly find the optimal task allocation strategy by optimizing the policy network. The use of this algorithm model can accurately capture the complex relationship between tasks and resources, provide a more comprehensive decision-making basis for task allocation, quickly respond to environmental changes, adjust task allocation in real time, and ensure the continuity and efficiency of the production process.

[0101] (3) This invention uses sensors and monitoring systems to collect real-time workshop production data and updates it into the state space of the reinforcement learning model. This solves the problem in existing technologies where traditional methods lack real-time monitoring and feedback of the production process, making it impossible to adjust task allocation strategies in a timely manner, resulting in reduced production efficiency. Real-time data collection and updating enables the reinforcement learning model to allocate tasks based on the latest workshop status, improving the model's decision-making accuracy and adaptability.

[0102] (4) By inputting the real-time updated state into a trained reinforcement learning model, the present invention addresses the problem of traditional methods in the prior art being slow to adjust task allocation in a dynamically changing production environment and unable to meet real-time production needs. The reinforcement learning model can evaluate the current state in real time and quickly output the optimal task allocation decision, improving the real-time performance and accuracy of task allocation.

[0103] (5) Based on the model's decision-making results, the present invention sends task assignment instructions to the corresponding execution units via the workshop's communication network, monitors the execution of tasks in real time, and feeds the execution results back to the reinforcement learning model. This solves the problem in the prior art that traditional methods lack real-time monitoring and feedback of task execution and are unable to optimize task assignment strategies based on execution results. Through real-time monitoring and feedback mechanisms, the reinforcement learning model can continuously adjust and optimize task assignment strategies based on task execution, achieving continuous optimization of the system.

[0104] (6) The present invention aims at the various types of entities (such as workers, robots, tasks, equipment, etc.) involved in the human-machine collaborative workshop and the complex and diverse relationships between them, and proposes a method to model the state space of the scheduling problem as a heterogeneous graph, and further adopts a heterogeneous graph neural network for information processing and feature extraction. This technical feature has significant innovation and advantages in scheduling modeling and intelligent decision-making methods compared with the comparative documents. In the existing technology, although graph neural networks and deep reinforcement learning technologies are also used for fuzzy workshop scheduling modeling, the graph structure used is usually a homogeneous graph. This modeling method has certain effects when processing simple production tasks, but when faced with complex scenarios with multiple entities and multiple relationships in the human-machine collaborative environment, its expression ability is obviously insufficient, and it is difficult to reflect the heterogeneity between different types of nodes (such as workers and robots) and different types of relationships (such as task execution relationships, resource occupancy relationships, and collaborative dependency relationships), thereby limiting the accuracy and generalization ability of the scheduling strategy. To this end, this paper proposes a modeling approach based on heterogeneous graphs, which clearly distinguishes and constructs multiple entity types (nodes) and relationship types (edges), such as worker nodes, robot nodes, and task nodes, as well as multiple edge types such as worker-task execution relationships, robot-task execution relationships, and worker-robot collaboration relationships. Through this heterogeneous graph structure, this paper can more precisely model the heterogeneous state information and multidimensional interactive dependencies in the human-robot collaborative scheduling problem, comprehensively characterizing the synergies and resource conflicts between different participants. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 A flowchart of human-machine collaborative workshop scheduling based on reinforcement learning provided by an example of the present invention;

[0106] Figure 2 This is a diagram of the human-machine collaborative workshop scheduling structure based on reinforcement learning provided by an example of the present invention;

[0107] Figure 3 A diagram of the reinforcement learning model architecture provided by an example of the present invention;

[0108] Figure 4 This is a reinforcement learning flowchart based on tightening screws of new energy battery packs provided by the example of the present invention. DETAILED DESCRIPTION

[0109] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0110] Reinforcement learning, an emerging machine learning technology, offers new insights into complex decision-making problems. Through interactive learning between an agent and its environment, reinforcement learning can automatically find the optimal task allocation strategy to maximize long-term cumulative rewards. It eliminates the need for pre-defined complex rules and conditions, instead learning through trial and error, automatically adjusting strategies based on environmental feedback. This adaptive learning capability makes reinforcement learning particularly well-suited for complex and volatile human-machine collaborative environments.

[0111] Compared to traditional rule-based approaches, reinforcement learning is more adaptable to dynamically changing production environments. By rationally designing the state space, action space, and reward function, reinforcement learning transforms the task allocation problem into a sequential decision-making problem and learns the optimal strategy. Furthermore, reinforcement learning possesses the ability to make real-time decisions, enabling rapid response to environmental changes and emergencies, adjusting task allocation plans promptly to ensure smooth production. For example, in actual production, if a robot malfunctions or a worker is unable to complete a task in a timely manner, the reinforcement learning model can quickly react and reallocate the task to other available resources, thereby avoiding production interruptions and improving the system's robustness and adaptability.

[0112] In human-machine collaborative task allocation, real-time task allocation is key to improving production efficiency and system responsiveness. Traditional task allocation methods are typically completed during the production planning phase and lack the ability to make real-time adjustments. Reinforcement learning, on the other hand, monitors production status in real time and makes task allocation decisions based on this real-time information. Through real-time interaction with the environment, reinforcement learning models can continuously learn and optimize task allocation strategies to adapt to changing production needs.

[0113] This example proposes a reinforcement learning-based method for human-robot collaborative task allocation. This method leverages reinforcement learning to address the flexible job shop scheduling problem, using the tightening of screws in new energy vehicle battery packs as an assembly task, and allocates tasks between humans and robots. By rationally allocating tasks, this method leverages the respective strengths of both workers and robots, improving production efficiency and assembly quality while also enhancing the system's flexibility and adaptability to dynamically changing production environments and emergencies.

[0114] A human-machine collaborative task allocation method based on reinforcement learning, such as Figure 1 、 Figure 2 、 Figure 3 As shown, the following describes the implementation of the present invention in detail in conjunction with a specific new energy battery pack screw tightening task scenario.

[0115] Scenario: Suppose M workers and N (collaborative) robots need to complete the task of tightening K screws on a battery pack. Each screw tightening operation can be performed by a worker with specific skills, a specific robot model, or human-robot collaboration. There may be order constraints between tasks. The specific steps are described below.

[0116] S1. Construct a human-machine collaborative workshop environment model for the screw tightening task of new energy vehicle battery packs, including task definition, resource definition, state space, action set, and reward function design.

[0117] (1) Task definition: The screw tightening task of the new energy vehicle battery pack is decomposed into multiple subtasks, each of which has a clear location, tightening torque requirement and time limit. A battery pack contains 10 screws, and the tightening task of each screw is a process. The complexity of the process is classified according to the tightening torque requirement and the accessibility of the position, and is divided into simple tasks (suitable for robots to perform) and complex tasks (suitable for workers to perform). A single screw tightening operation is recorded as op i,j , represents the jth operation in the i-th job; the processing time matrix P = {p i,j,k} represents the task op i,j The time required for operator k to execute; some screws must be tightened before other screws can be operated, forming a dependency matrix D = {d i,j,p,q}, where d i,j,p,q =1 indicates task op i,j Must be in op p,q You can start only after you finish it.

[0118] (2) Resource definition: Human operators and collaborative robots in the workshop are defined as resources. Each operator has different capabilities and limitations. For example, human operators are suitable for handling complex and flexible tasks, while robot operators are suitable for handling repetitive and high-precision tasks. The worker set W = {w1, w2, ... w m}, where m is the number of workers, and the robot set R = {r1, r2, ...r n}, where n is the number of robots, and the collaborative unit set C = {c i,j |c i,j =(w i ,r j )}, indicating worker w i With robot r j The collaborative unit composed of i,j represents the efficiency of the collaborative unit, defined as in and Represents workers h iand robot r j is the efficiency parameter, and α is the collaboration weight coefficient.

[0119] (3) State space: Based on the actual situation of the workshop, define the state space S of the reinforcement learning environment, including: basic time state t current Indicates the current time. Task state vector TS = {ts1, ts2, ... ts K}, where K is the total number of tasks, ts i Represents the state of the i-th task (0: not started, 1: in progress, 2: completed), the operator state matrix: Worker state vector: WS = {ws1, ws2, ..., ws m}, ws i Represents the state of the i-th worker, the robot state vector: RS = {rs1,rs2,...rs n},rs j Represents the state of the j-th robot, the collaborative unit state matrix: CS = {cs i,j}, represents worker w i With robot r j The status of the collaborative unit and the operator's busy time: represents the estimated busy end time of each worker, represents the expected busy end time of each robot, the task completion time vector: FT = {ft1, ft2, ..., ft K}, where ft i represents the completion time of task i (0 or preset maximum if completed), worker utilization rate Robot utilization in and Represents workers w i With robot r j Busy time, T total Indicates the total production time.

[0120] (4) Action set: Actions include assigning a task to a worker or robot, or adjusting the execution order of tasks. The action space A is defined as the task assignment decision, including: assigning a task op i,j Assigned to worker w k , and set the task op i,j Assigned to robot r l , and set the task op i,j Assigned to collaborative unit c k,l , waiting for action (no work is assigned), the action is represented by a vector a = (op idx ,executor idx ,executor type), where op idx Indicates the index of the task to be assigned, executor idx The index of the executor (worker, robot or collaborative unit), executor type Indicates the executor type (0: worker, 1: robot, 2: collaborative unit).

[0121] (5) Reward function design: The reward function evaluates the quality of the current action based on the time, quality, and resource utilization of the task completion. For example, a positive reward is obtained for successfully completing a task with qualified quality, while a negative reward is given for the task timeout or quality problems. The reward function R(s,a,s′) represents the reward obtained by executing action a from state s to state s′. The time point (makespan) at which the last task is completed in each scheduling scheme is calculated, and then the negative value is set as the reward function. Through the negative value conversion, the reinforcement learning algorithm will learn to maximize this negative makespan value, that is, minimize the actual makespan (the total time for all tasks to be completed); the reward function is:

[0122] R(s,a,s′)=-λ1·Makespan+λ2·U

[0123] Where R(s,a,s′) represents the reward value obtained by transferring from state space s to state space s′ through action a; Makespan represents the time point when the last task is completed in each scheduling scheme; λ1 and λ2 are weighting coefficients; and U is the resource utilization index:

[0124] U=β·U W +(1-β)·U R

[0125] in, is the average utilization rate of all workers, is the average utilization of all robots, and Represents workers w i With robot r j Busy time, T total represents the total production time, and β is the weight coefficient.

[0126] S2. Build a reinforcement learning model using heterogeneous graph neural network (HetGNN) and proximal policy optimization (PPO) reinforcement learning algorithm.

[0127] (1) The state space S of the human-machine collaborative workshop is modeled as a heterogeneous graph G = (V, E), where the node set is V = V T ∪V W ∪V R ,include:

[0128] Task Node Set Each node represents a specific task op i,j , the task node characteristics include the current task status ts i , estimated processing time vector {p i,j,k}, represents the time required for execution by various operators and task dependency information, according to the task dependency matrix D = {d i,i,p,q}Get, task completion time ft i ;

[0129] Worker node set Each node represents a worker w i , the worker node features include the worker’s current state ws i , Workers' busy time ends Worker efficiency parameters

[0130] Robot Node Set Each node represents a robot r i , the robot node features include the robot’s current state rs j , Robot busy end time Robot efficiency parameters

[0131] The edge set is E=E T-T ∪E T-W ∪E T-R ∪E W-R ,include:

[0132] Inter-task dependency edge E T-T , represents the dependency relationship between tasks, if d i,j,p,q =1, that is, task op i,j Must be in op p,q Only after completion can it start, establishing directional edges between task nodes. The characteristics of inter-task dependency edges include dependency strength or priority;

[0133] Task-Worker Side E T-W , indicating that the task can be executed by workers, if the worker w k Can execute mission op i,j , then the task node and the worker node are connected, and the task-worker edge features include the worker processing time p i,j,k ;

[0134] Mission-Machine Edge E T-R , indicating that the task can be performed by the robot. If the robot r l Can execute mission op i,j , then the task node and the robot node are connected, and the task-machine edge features include the robot processing time pi,j,l ;

[0135] Worker-Robot Edge E W-R , indicating a relationship that can form a collaborative unit, if worker w i With robot r j Can form collaborative units cs i,j , then connect the worker node and the robot node, the worker-robot edge features include the collaboration efficiency coefficient η i,j .

[0136] (2) Using a heterogeneous graph neural network encoder, node features and edge features are input into the network, and the message passing function between different types of nodes is defined as in and They are node v i ,v j In the feature representation of the l-1 layer, is the edge feature connecting them, φ (l) is the message function of layer l, which can be a multi-layer perceptron. i , collects messages from all its neighboring nodes and performs weighted aggregation based on node type, Among them, N(v i ) is the node v i The neighbor set of is the attention weight, defined as Where W and a are learnable parameter matrices and vectors. Update node feature representation where σ is the activation function, and Is for node type tgpe(v i )’s weight matrix and bias vector. After L layers of message passing, the final node representation is obtained. The global graph representation can be obtained through pooling operations: Pool can be average pooling, maximum pooling or attention pooling.

[0137] The present invention models the state space in human-machine collaborative scheduling as a heterogeneous graph and uses a heterogeneous graph neural network for processing in order to overcome the core problems of the existing technology, such as insufficient state representation capabilities, rough relationship modeling, and difficulty in fine generalization of scheduling strategies. In actual human-machine collaborative production scenarios, there are many types of entities, such as workers, robots, tasks, workstations, etc. There are diverse relationships between these entities, such as task allocation, sequential dependencies, collaborative pairing, resource conflicts, etc. Traditional methods often use fixed-dimensional vector forms or isomorphic graphs for modeling. Although this method is unified in form, it cannot effectively express the type differences and relationship semantics between nodes, resulting in limited expression capabilities of scheduling strategies in the face of entity heterogeneity and complex task dependencies, and reduced scheduling quality.

[0138] The present invention constructs different types of entities as nodes of different categories and explicitly distinguishes various types of relationship edges to form a heterogeneous graph structure, thereby achieving a more discriminative and expressive state modeling method. Compared with isomorphic graphs or homogeneous graph structures of specific channels, heterogeneous graphs can accurately capture multi-semantic information such as worker fatigue level, robot load status, task priority and skill requirement matching relationship, avoiding confusion of different node and relationship semantics during information dissemination. Through heterogeneous graph neural networks, node type perception and relationship type perception mechanisms are introduced in the information dissemination and feature aggregation stages, so that each type of node can obtain an exclusive feature update path and execute customized information transfer functions according to different types of edges, so that the state representation has semantic fidelity, structural resolution and behavior guidance.

[0139] In scheduling decisions, heterogeneous graph neural networks provide a more detailed and dynamic state representation, enabling the reinforcement learning policy network to identify key constraints such as the adaptability of different tasks to the executor, the rationality of the human-machine collaboration sequence, and the balance of resource loads. They can also flexibly adjust scheduling strategies in dynamic environments, enhancing system robustness and generalization capabilities. Compared with the three-channel graph neural network scheduling method disclosed in the prior art, this method can improve expressiveness through parallel feature extraction, but it still uses a homogeneous processing approach to the graph structure itself, unable to explicitly handle node categories and edge semantics, making it difficult to adapt to the complex semantics of human-machine collaboration tasks.

[0140] Therefore, the present invention solves the problem that the existing scheduling model cannot recognize node semantic differences and multiple relationship semantics in state modeling by introducing heterogeneous graph modeling and heterogeneous graph neural network processing methods, improves the scheduling strategy's ability to understand and respond to the state, and realizes more accurate, dynamic and robust task allocation and scheduling optimization, ultimately achieving faster task response, higher resource utilization, and more reasonable human-machine cooperation in actual production.

[0141] (3) Build a reinforcement learning training framework based on the proximal policy optimization (PPO) algorithm, policy network: π θ (a|s)=SoftmaxLayer(MLP(h G )), where h G is the graph representation obtained by the HetGNN encoder, MLP is a multi-layer perceptron, θ is the policy network parameter; value network: V φ (s) = MLP′(h G ), where MLP′ is another multi-layer perceptron and φ is the value network policy. PPO objective function: in is the importance weight, Is the advantage function estimate, ∈ is the clipping function. Value function loss: Where V t target is the estimated value of the return, obtained according to the reward function R(s,a,s′). Entropy regularization term: S[π θ |(s t )]=-∑ a π θ (a|s t )logπ θ (a|s t ). Total loss function: L(θ|φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ], where c1 and c2 are hyperparameters that control the value loss and entropy regularization strength. Backpropagation and gradient updates are performed through the total loss function to optimize the policy network and value network parameters to obtain the trained reinforcement learning model.

[0142] The reinforcement learning method for human-machine collaborative task allocation based on heterogeneous graph neural networks and proximal policy optimization first models people, robots, and tasks as nodes of a heterogeneous graph, and realizes information transmission and fusion between nodes through the HetGNN multi-layer structure to generate a state representation that takes into account complex relationships; then, using the PPO algorithm framework, the policy network outputs the action distribution based on the encoded state, and the value network evaluates the state value. By collecting interaction data and calculating the clipping objective function, the policy optimization is achieved, while the update step size is controlled to maintain training stability; the entire process alternately executes state encoding, action selection, environment interaction, and parameter update in each iteration, so that the model can continuously adapt to the dynamically changing human-machine collaborative environment, and ultimately achieve real-time online task allocation and scheduling.

[0143] S3. Use the workstation vision sensor to identify the screw position information and the torque sensor to measure the screw tightening torque and quality. Use the monitoring system to collect workshop production data in real time, including the current location of the task, completed tasks, unfinished tasks, the worker's working status, the robot's working status, etc. The robot status monitoring module obtains data such as the robot's power and working status, and updates it to the state space of the reinforcement learning model.

[0144] Workstation vision sensors are installed at each workstation to identify the location of screws in real time. These sensors can accurately detect the two-dimensional and three-dimensional coordinates of the screws on the battery pack, ensuring that the task allocation system can accurately assign the screw tightening task to the appropriate operator (worker or robot). The vision sensor has an acquisition frequency of 10Hz and can update the screw position information in real time to ensure the accuracy of task allocation. Torque sensors are installed on the tightening tool to measure the torque value during the screw tightening process. These sensors can monitor in real time whether the torque of the screw tightening meets the preset quality standards. The torque sensor has an acquisition frequency of 100Hz and can accurately record the torque curve of each tightening operation to evaluate the tightening quality. If the torque value exceeds the preset range, the system will immediately issue an alarm and mark the task as incomplete for reassignment or quality inspection. The monitoring system integrates data from multiple sensors to collect workshop production data in real time. The core of the monitoring system is a central data processing server, which is responsible for collecting, processing and storing data from various sensors. The data collected by the monitoring system includes but is not limited to the following:

[0145] Current location of the task: The real-time location information of each task (screw tightening operation) is obtained through the workstation vision sensor.

[0146] Completed tasks: The torque sensor and vision sensor work together to determine whether the task is completed. If the torque value meets the standard and the vision sensor confirms the screw position is correct, the task is marked as completed.

[0147] Unfinished tasks: Tasks that do not meet quality standards or have not been started are marked as unfinished and stored in the task queue waiting to be reassigned.

[0148] Worker work status: Use wearable devices and workstation sensors to obtain workers' real-time status, including whether they are idle, busy, or unavailable.

[0149] Robot Operating Status: The robot status monitoring module monitors the robot's real-time status, including whether it is idle, busy, or faulty. It also records the robot's battery level, tool end status, and maintenance needs.

[0150] S4. Input the real-time updated state into the trained reinforcement learning model. The model quickly outputs the optimal scheduling action by evaluating the value function or policy network of the current state and assigns the task to the appropriate worker or robot, such as Figure 3 shown.

[0151] Collect current workshop status t , use the HetGNN encoder to convert the state into a graph representation hG, and calculate the action probability distribution π output by the policy network θ (a|s t ), generate action mask M a (s t ), select actions from the masked probability distribution Use the value network to estimate the current state value V φ (s t ) is used for decision evaluation and uncertainty processing. When some sensor data is missing or abnormal, historical data or similar state data are used to supplement it.

[0152] The collected data is updated into the state space of the reinforcement learning model through the following steps to remove noise, outliers, and redundant data. Incorrect torque values ​​or position information caused by sensor failures are eliminated. Data from different sources is unified and converted into state vectors and tensors. Sparse events (such as sudden failures) are sampled or simulated to enhance the model's ability to cope with rare situations. Key features are extracted from the raw data, reducing the dimensionality of the state space. The monitoring system feeds collected data back to the reinforcement learning model in real time to ensure that the model can allocate tasks based on the latest shop floor status. The state space of the reinforcement learning model is continuously updated using real-time data, ensuring that the model can dynamically adjust task allocation strategies. For example, if a worker becomes increasingly fatigued, the model automatically adjusts task allocation to avoid assigning difficult tasks. When an abnormal situation occurs (such as a sensor failure or robot malfunction), the system immediately initiates the exception handling mechanism. For example, if a torque sensor detects substandard tightening quality, the system reassigns the task and notifies maintenance personnel to inspect the tool.

[0153] Execute task assignment and issue instructions. Task assignment instruction format: CMD = {op id ,executor id ,start time ,priority,procedure params}, where op id Is the task ID, executor id Is the executor ID, start time is the start time, priority is the priority, procedure paramsThese are task execution parameters. Task instructions are sent to workers via workstation displays, mobile devices, or AR glasses. Instructions are sent directly to collaborative robots via the control system API, or coordinated instructions are sent to both workers and robots simultaneously. Ordinary tasks are executed in the order output by the model. For high-priority tasks, currently executing low-priority tasks can be interrupted. When urgent tasks are interrupted, the execution plan is immediately adjusted, tasks are reallocated, and resources are prioritized for handling urgent tasks.

[0154] S5. Based on the model’s decision-making results, the task allocation instructions are sent to the corresponding execution units through the workshop’s communication network. The execution of the tasks is monitored in real time, and the execution results are fed back to the reinforcement learning model so that the model can further adjust and optimize the task allocation strategy.

[0155] Record the start and end times of new energy battery pack screw tightening tasks, track progress in real time, and evaluate task completion quality. By detecting discrepancies between actual execution time and estimated time, and between execution results and expected quality standards, feedback is provided on task completion status, time, and quality data, along with any anomalies, interruptions, or failures during execution, including subjective worker evaluations and suggestions, and robot performance metrics. Continuously update the model using practical execution data: Where α is the learning rate, is the gradient calculated based on the new data.

[0156] Extract valuable patterns from historical task execution, identify high-efficiency and high-quality task allocation patterns, analyze the factors leading to execution failure or delay, integrate human expert experience and heuristic rules into the model, and improve the adaptability of the reinforcement learning model to different tasks.

[0157] The reinforcement learning-based human-machine collaborative task allocation method proposed in this invention has been effectively applied in the screw tightening task of new energy battery packs. The specific effects are as follows:

[0158] Improved production efficiency: Compared with traditional rule-based scheduling, the production cycle time is shortened by 20%-30%;

[0159] Improved resource utilization: The average utilization of workers and robots increased by 15%, reducing idle waiting time;

[0160] Enhanced adaptability: Ability to complete rescheduling within 5 seconds after an unexpected abnormality (equipment failure, material delay, etc.) occurs;

[0161] Improved collaboration efficiency: The efficiency of human-machine collaborative tasks is 25% higher than that of individual tasks.

[0162] Improved decision quality: The rationality score of task allocation decisions increased by 35%, reducing unnecessary resource movements and task switching;

[0163] Enhanced system robustness: Under various interference conditions, system stability is improved by 40%, which can effectively maintain the production rhythm.

[0164] The reinforcement learning-based human-machine collaborative task allocation method proposed in the present invention achieves efficient allocation and real-time online scheduling of new energy battery pack screw tightening tasks by constructing a reinforcement learning environment model that conforms to the actual production environment, using heterogeneous graph neural networks for state encoding, and adopting the PPO algorithm training strategy and value network. This method has strong adaptability and robustness, can effectively respond to emergencies in the production process, optimize human-machine collaboration efficiency, and improve production efficiency and resource utilization. The technical solution of the present invention can be extended to other types of human-machine collaborative manufacturing scenarios, providing a new solution for the field of intelligent manufacturing.

[0165] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A human-machine collaborative task allocation method based on reinforcement learning, characterized in that: The following steps are involved: Build a reinforcement learning model for human-machine collaborative workshop reinforcement learning, including task definition, resource definition, state space, action set, and reward function; A heterogeneous graph neural network is used to model the collaborative relationship of the state space of the reinforcement learning model, and the reinforcement learning model is trained with a proximal policy optimization algorithm to obtain a trained reinforcement learning model based on the state space input and output task allocation strategy. The sensors and monitoring system in the workshop collect real-time workshop production data, including the current location of tasks, completed tasks, unfinished tasks, workers' working status, and robots' working status. The state space of the reinforcement learning model is updated based on the collected workshop production data. The updated state space is input into the trained reinforcement learning model, which outputs the optimal task allocation action based on the current state space through the policy network. According to the optimal task allocation action, the task instructions are sent to the corresponding workers or robots and other execution units through the workshop communication network to complete the task allocation and scheduling; The task execution results are monitored in real time, and the actual execution effects are fed back to the reinforcement learning model as environmental interaction feedback to further optimize and adjust the reinforcement learning model.

2. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The task definition includes: Task sequence op i,j , represents the jth operation in the i-th job; Processing time matrix P = {p i,j,k }, indicating the task op i,j The time required to be executed by operator k, including workers and robots; Task dependency matrix D = {d i,j,p,q }, where d i,j,p,q =1 indicates task op i,j Must be in op p,q You can start only after you finish it.

3. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The resource definition includes: Worker set W = {w1,w2,...w m }, where m is the number of workers; Robot set R = {r1, r2, ...r n }, where n is the number of robots; Collaboration unit set C = {c i,j |c i,j =(w i ,r j )}, indicating worker w i With robot r j The collaborative units are composed of i,j Including the collaborative efficiency coefficient η i,j , defined as: Among them, α is the collaboration weight coefficient, Represents workers w i and robot r j efficiency parameters.

4. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The state space S includes: Basic time state t current , indicating the current time; Task state vector TS = {ts1, ts2, ... ts K }, where K is the total number of tasks, ts i Indicates the status of the i-th task. The status values ​​include: 0 for not started, 1 for in progress, and 2 for completed; Worker state vector WS = {ws1, ws2, ..., ws m }, ws i represents the state of the i-th worker; Robot state vector RS = {rs1,rs2,...rs n },rs j represents the state of the j-th robot; Collaboration unit state matrix CS = {cs i,j }, indicating worker w i With robot r j The status of the composed collaborative units; Worker busy end time vector Indicates the expected busy end time of each worker; Robot busy end time vector Indicates the estimated busy end time of each robot; Task completion time vector FT = {ft1, ft2, ..., ft K }, where ft i Indicates the completion time of task i. If the task is not completed, the value is 0 or the preset maximum value; Worker utilization rate Robot utilization in and Represents workers w i With robot r j Busy time, T total Indicates the total production time.

5. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The action set includes a task assignment action and a waiting action, wherein the task assignment action includes the following forms: The task op i,j Assigned to worker w k , put the task op i,j Assigned to robot r l , put the task op i,j Assigned to collaborative unit c k,l ; The waiting action indicates that no task is currently assigned, that is, no task assignment operation is performed; The action is expressed in vector form as: a=(op idx ,executor idx ,executor type ) Among them, op idx Indicates the index of the task to be assigned, executor idx Indicates the index of the executor, executor type Indicates the executor type. The value is: 0 for worker; 1 for robot; 2 for collaborative unit.

6. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The reward function is: R(s,a,s′)=-λ1·Makespan+λ2·U Among them, R(s,a,s ′ ) represents the transition from state space s to state space s via action a ′ The reward value obtained; Makespan represents the time point when the last task is completed in each scheduling scheme; λ1 and λ2 are weighted coefficients; U is the resource utilization index: U=β·U W +(1-β)·U R in, is the average utilization rate of all workers, is the average utilization of all robots, and Represents workers w i With robot r j Busy time, T total represents the total production time, and β is the weight coefficient.

7. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The use of heterogeneous graph neural networks to model collaborative relationships in the state space of the reinforcement learning model specifically includes: According to the task definition and resource definition, the state space S of the human-machine collaborative workshop is modeled as a heterogeneous graph G = (V, E); A heterogeneous graph neural network is used to model the heterogeneous graph G = (V, E), where each layer of the neural network performs message passing and feature update through the following steps: For each pair of connected nodes v i ←v j , define the message passing function according to its node type and edge type: in, and They are node v i ,v j In the feature representation of the l-1 layer, is the edge feature connecting them, φ (l) is a learnable message function such as a multilayer perceptron at layer l; For node v i , collects messages from all its neighboring nodes and performs weighted aggregation based on node type: in, Indicates that in the l-th layer network, all neighbor nodes are aggregated to node v i The message result, N(v i ) is the node v i The neighbor set of is the attention weight, defined as Among them, W and a are the learnable parameter matrix and vector; According to the aggregated message results Update each node feature: Where σ is the activation function, and Is for node type tgpe(v i )’s weight matrix and bias vector; After L layers of message passing, the final node representation is obtained The global graph representation is obtained through pooling operation: Pool can be average pooling, maximum pooling or attention pooling.

8. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 7, characterized in that: The heterogeneous graph G = (V, E), where: The node set is V = V T ∪V W ∪V R ,include: Task Node Set Each node represents a specific task op i,j , the task node characteristics include the current task status ts i , estimated processing time vector {p i,j,k }, represents the time required for execution by various operators and task dependency information, according to the task dependency matrix D = {d i,j,p,q }Get, task completion time ft i ; Worker node set Each node represents a worker w i , the worker node features include the worker’s current state ws i , Workers' busy time ends Worker efficiency parameters Robot Node Set Each node represents a robot r i , the robot node features include the robot’s current state rs j , Robot busy end time Robot efficiency parameters The edge set is E=E T-T ∪E T-W ∪E T-R ∪E W-R ,include: Inter-task dependency edge E T-T , represents the dependency relationship between tasks, if d i,j,p,q =1, that is, task op i,j Must be in op p,q Only after completion can it start, establishing directional edges between task nodes. The characteristics of inter-task dependency edges include dependency strength or priority; Task-Worker Side E T-W , indicating that the task can be executed by workers, if the worker w k Can execute mission op i,j , then the task node and the worker node are connected, and the task-worker edge features include the worker processing time p i,j,k ; Mission-Machine Edge E T-R , indicating that the task can be performed by the robot. If the robot r l Can execute mission op i,j , then the task node and the robot node are connected, and the task-machine edge features include the robot processing time p i,j,l ; Worker-Robot Edge E W-R , indicating a relationship that can form a collaborative unit, if worker w i With robot r j Can form collaborative units cs i,j , then connect the worker node and the robot node, the worker-robot edge features include the collaboration efficiency coefficient η i,j .

9. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning model includes a policy network and a value network; The method of training the reinforcement learning model by combining the proximal strategy optimization algorithm specifically includes: Obtain the global graph representation h after the state space is processed by heterogeneous graph neural network modeling G , input to the policy network and value network to generate policy distribution π θ (a|s) and state value V φ (s); The policy network is: π θ (a|s)=SoftmaxLayer(MLP(h G )) Where MLP is the first multi-layer perceptron, θ is the policy network parameter, π θ (a|s) is the probability of selecting action a from the action set in state space s; The value network is: V φ (s)=MLP′(h G ) Among them, MLP′ is the second multi-layer perceptron, φ is the value network strategy, V φ (s) is the value estimate of the state space s; The proximal policy optimization algorithm is used to train the policy network, and the objective function is: in, is the importance weight, is the advantage function estimate, ∈ is the clipping factor, clip is the clipping function, is the expected estimate at time step t; The mean square error is used to train the value network, and the value function loss is: in, is the estimated value of the return, according to the reward function R(s,a,s ′ )get, Introduce the policy entropy regularization term: Among them, S[π θ |(s t )] is the strategy regardless of the state s t Entropy under Construct a joint loss function based on the objective function, value function loss and policy entropy regularization term: L(θ|φ)=L CLIP (θ)-c1L VF (φ)+c2S[π θ ] Among them, c1 and c2 are hyperparameters that control the value loss and entropy regularization strength, and L(θ|φ) is the total loss function; Backpropagation and gradient update are performed through the total loss function to optimize the policy network and value network parameters to obtain the trained reinforcement learning model.

10. The method for human-machine collaborative task allocation based on reinforcement learning according to claim 1, characterized in that: The updated state space is input into the trained reinforcement learning model. The reinforcement learning model outputs the optimal task allocation action based on the current state space through the policy network, specifically including: The updated state space is processed by heterogeneous graph neural network modeling to obtain the global graph representation h G , the global graph is represented by h G Input into the policy network of the trained reinforcement learning model to obtain the optimal task allocation action.

Images