Man-machine cooperation assembly sequence planning method based on double-agent DQN

Abstract

The invention discloses a man-machine cooperation assembly sequence planning method based on a double-agent DQN, and the method comprises the steps: constructing a reinforcement learning environment based on a process-level assembly knowledge graph, and training a double-agent DQN system representing an artificial Agent-H and a robot Agent-R in the environment; vectorizing the assembly state based on the knowledge graph and generating an action mask to shield illegal actions; an event-triggered human-based priority joint decision-making mechanism is adopted to realize asynchronous collaboration; designing a multi-target reward function including a nonlinear accumulation-recovery fatigue model to suppress artificial fatigue and balance man-machine load; and each agent maintains the online network and the target network and performs asynchronous updating by adopting experience playback. According to the method, an executable and human-friendly cooperative assembly sequence can be generated under the constraint of a complex process, the assembly efficiency is improved, and the artificial fatigue peak value is reduced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing and collaborative robot technology, and relates to a reinforcement learning method for human-machine collaborative assembly sequence planning and task allocation at the process level, specifically a human-machine collaborative assembly sequence planning method based on dual-agent DQN. Background Technology

[0002] Human-Robot Collaboration (HRC) is an important form of intelligent manufacturing. Assembly Sequence Planning (ASP) is a key technology within it. Based on structured assembly information, ASP generates an executable sequence of processes for human-machine collaborative assembly tasks, serving as the direct basis for subsequent scheduling and execution. As product complexity increases and process granularity decreases to the "process level," the number of parts, process relationships, and constraint types that sequence planning needs to handle increase dramatically, leading to an exponential growth in the feasible region. Traditional methods face significant bottlenecks in computational efficiency and solution quality.

[0003] Existing technologies generally follow three paths: First, rule-based or scoring-based sequence evaluation methods. These methods have the advantages of simple implementation and good interpretability. However, when the size of parts and the number of process levels increase rapidly, the search space expands dramatically, local heuristics are prone to getting trapped in suboptimal solutions, and it is difficult to maintain stability under dynamic perturbations. Second, intelligent optimization decision-making methods. Typical approaches employ multi-objective optimization while simultaneously balancing assembly time, energy consumption, and other indicators; genetic algorithms (GA) have been repeatedly verified, as they can balance task allocation and sequential coupling; discrete Bees series methods can improve search capabilities under complex constraints and uncertain working conditions. The above two methods have advantages in multi-objective trade-offs and large-scale search, but they generally suffer from problems such as insufficient modeling of human factors and insufficient constraints on the reachability of robot actuators. Third, deep reinforcement learning and multi-agent reinforcement learning. It exhibits higher robustness and efficiency in continuous control and practical tasks; considering that human-machine collaboration involves at least two agents, "one person and one machine", MARL becomes a natural paradigm for task allocation and sequential decision-making, and existing studies have explicitly injected perturbations during the training phase to enhance adaptability to human uncertainty.

[0004] Despite this, existing HRC-ASP technologies still suffer from the following common shortcomings: 1. Many multi-agent reinforcement learning frameworks abstract "human / machine" into homogeneous agents or isomorphic task executors, failing to fully reflect human-machine differences and policy preferences, making it difficult to achieve true human-centered collaboration. 2. Weak human-centered interaction and asynchronous collaboration mechanisms. Industrial sites commonly involve event triggering and asynchronous parallelism, generally following a "human-first" collaboration rule. However, existing methods often employ synchronous rounds or fixed scheduling, making it difficult to meet on-site cycle time and safety requirements without increasing overhead. 3. Inadequate trade-offs between multiple objectives and reliability. The trade-offs and unified measurement among multiple indicators such as time, fixture / tool ​​replacement, energy consumption, ergonomics, and solution confidence and stability are still insufficient, resulting in insufficient robustness and reproducibility under interference, noise, and uncertain operating conditions. 4. Insufficient online update and migration capabilities. Faced with product variations, batch differences, and real-time changes on-site, existing solutions offer limited support in knowledge updates, policy migration, and incremental learning, affecting large-scale application. Summary of the Invention

[0005] Objective: This invention proposes a human-machine collaborative assembly sequence planning method based on dual-agent DQN. It expresses and updates assembly relationships and constraints online within a unified knowledge carrier, namely a knowledge graph; it distinguishes between the heterogeneous characteristics of humans and machines and reflects human-centric priorities and fatigue factors; it supports event-triggered asynchronous collaboration and robust multi-objective trade-offs; and it improves interpretability, transferability, and decision-making efficiency in complex assembly scenarios. The effective solution to the above problems will directly affect the feasibility and operational quality of human-machine collaborative assembly systems in real industrial environments.

[0006] Technical solution: The present invention provides a human-machine collaborative assembly sequence planning method based on dual-agent DQN, the implementation process of which is as follows:

[0007] (1) Construct a learning environment for the human-centered interactive dual-agent DQN reinforcement learning algorithm, namely, construct an assembly knowledge graph based on the process-level assembly process; the knowledge graph takes process-level assembly operations as nodes, and establishes priority relationship, non-simultaneous relationship and strong association relationship between nodes, and binds attributes on the nodes;

[0008] (2) Vectorize the knowledge graph into an environmental state representation of the agent, and define the respective state spaces for Agent-H, representing human beings, and Agent-R, representing robots;

[0009] (3) Generate action masks based on the relationships and attributes between nodes in the knowledge graph to shield candidate actions that violate priority relationships or cause resource conflicts, and mark them as illegal actions with the mask; design differentiated action selection strategies for different agents to form a real-time action space for each agent.

[0010] (4) An event-triggered human-centric joint decision-making mechanism is adopted. When Agent-H and Agent-R are ready at the same time, Agent-H selects the process first, and then Agent-R selects the compatible process in the updated state.

[0011] (5) Formulate multi-objective reward rules for fatigue perception that are in line with human-machine collaborative assembly scenarios, taking into account four types of objectives: completion time, human-machine task matching degree, human fatigue factors, and aggregation of strongly related processes, so as to promote the balance of human and machine load.

[0012] (6) Establish online Q network and target Q network for Agent-H and Agent-R respectively and maintain experience replay buffers respectively. During the training phase, periodically sample and update the online network parameters from the replay buffer in small batches. The target network is synchronously or softly updated according to a predetermined strategy.

[0013] (7) The knowledge graph is locally incrementally updated based on the training and execution results to maintain the timeliness of the environment.

[0014] Furthermore, the attributes mentioned in step (1) include the working time, difficulty factor, and fatigue coefficient of the process when it is executed independently by a human, independently by a robot, or in collaboration with a human.

[0015] Furthermore, the state spaces of Agent-H and Agent-R in step (2) both include the global assembly state, the list of executable processes and their attributes, and the agent's observable state; the state space of Agent-H also includes the accumulated fatigue level of Agent-H; and the state space of Agent-R also includes the collaboration request identifier.

[0016] Furthermore, the implementation process of step (3) is as follows:

[0017] The Agent-H is constructed to select the set of actions that can be selected for independent execution of the process. Based on the conditional filtering and random sampling of the process attributes, the action selection strategy is formulated. The three inherent attributes of different processes when they are completed manually are filtered and then randomly sampled to obtain a specific process to be executed.

[0018] Based on the degree of dependence of process relationships, other action selection strategies are set. In the assembly sequence planning process, the evaluation index needs to consider the influence of the relationship between the sequences to be planned, and take into account the process relationships when making action selections; select the process with the fewest "priority over" relationships, the process with the most "priority over" relationships, and the process with the most "not simultaneous" relationships.

[0019] The Agent-H selects a set of actions that can be chosen for collaborative execution processes. This is similar to the Agent-H selects a set of actions that can be chosen independently. The difference is that the process attributes considered are three attributes for human-machine collaboration: collaboration time, collaboration difficulty coefficient, and collaboration fatigue factor.

[0020] Constructing the action space A of Agent-R R Agent-R's task is to autonomously execute tasks or work collaboratively with Agent-H. It only needs to consider the case where the robot performs a certain process alone, including: independent execution with short time consumption, low difficulty, small dependencies, many conflict relationships, and responding to various collaborative requests.

[0021] Furthermore, the implementation process of step (4) is as follows:

[0022] When Agent-H completes a process but Agent-R is not ready, only Agent-H is triggered to make a decision. Agent-H chooses to execute the process independently or to initiate cooperation. If it chooses to initiate cooperation, Agent-H immediately enters a resting state and waits until Agent-R is idle and responds to the cooperation request.

[0023] When Agent-R completes a process but Agent-H is not ready, only Agent-R's decision-making is triggered. First, it checks whether there is a collaboration request initiated by Agent-H in the environment. If there is, it responds first; otherwise, it selects an action to execute within its own action space.

[0024] When Agent-H and Agent-R complete the process simultaneously, Agent-H is triggered to make a decision first, and the knowledge graph environment is updated based on the action selection result. Then Agent-R is triggered to make a decision.

[0025] Furthermore, the process for achieving the completion time target in step (5) is as follows:

[0026] To meet the efficiency target, a reward factor that minimizes the completion time is designed, expressed as follows:

[0027]

[0028] in, These represent the time required to complete process k in three modes: alone, alone (by a robot), and in a human-robot collaboration mode. During training, a reward is given for intensive work hours. The training direction can only be correct within a local field of view. To minimize the total training time, a final time reward needs to be introduced. :

[0029]

[0030] in, These represent the time required to complete all processes individually, individually by robots, and in a human-robot collaboration scenario, respectively. This indicates the required working time for the algorithm planning scheme. This reward ensures that a reasonable algorithm planning scheme is rewarded and an unreasonable scheme is penalized. At the same time, the shorter the total working time of the plan, the greater the reward. It is a coefficient that adjusts the sensitivity of rewards.

[0031] Furthermore, the process for achieving the human-machine task matching target in step (5) is as follows:

[0032] Design a reward factor for human-machine task matching to ensure that task allocation aligns with the respective strengths and limitations of both humans and robots, avoiding the assignment of tasks beyond their capabilities or excessively difficult tasks to a single executor.

[0033]

[0034] in, These represent the difficulty coefficients corresponding to the three modes of completing process k: alone by a human, alone by a robot, and in collaboration between humans and robots. This is the difficulty reward coefficient, ensuring that the difficulty reward matches the other rewards in terms of magnitude.

[0035] Furthermore, the process for achieving the target of human fatigue factors in step (5) is as follows:

[0036] Design a nonlinear fatigue accumulation-recovery reward factor, introduce accumulation-recovery coupling on the Agent-H side, and design the fatigue reward as follows:

[0037] Design a nonlinear fatigue accumulation model. When Agent-H selects to execute a fatigue-inducing process k at time t, the independent processes... or collaborative process Its fatigue increment The definition is as follows:

[0038]

[0039] in, β is the basic fatigue coefficient for process k, and β is the fatigue sensitivity coefficient, which controls the nonlinearity of fatigue accumulation. The larger β is, the stronger the amplification effect of existing fatigue on newly generated fatigue. After executing a new process, the new accumulated fatigue level is updated as follows:

[0040]

[0041] A fatigue recovery model is designed, employing an exponential decay model to describe the AI ​​agent's recovery process, which is triggered after Agent-H selects a collaborative step k; it is assumed that Agent-R still has [time] days to complete its current task. Time, accumulated fatigue Attenuation occurs:

[0042]

[0043] in, The fatigue recovery rate determines the recovery speed; after Agent-H selects the collaborative process k, it triggers a process of first recovering fatigue and then accumulating fatigue. The final accumulated fatigue after completing the entire "wait-collaboration" process is:

[0044]

[0045] By integrating the fatigue reward factor from the fatigue accumulation-recovery model, the dynamic change process of the aforementioned fatigue level is transformed into an effective reward signal, and the negative value of accumulated fatigue is regarded as the potential energy of the worker's "fatigue state":

[0046]

[0047] Here, C is a positive constant, and the goal of Agent-H is to maintain a high potential energy state; fatigue reward It is directly defined as the negative value of the cumulative change in fatigue level:

[0048]

[0049] in, This is a fatigue reward coefficient, designed to ensure that fatigue rewards match the other rewards in terms of magnitude. It is a basic reward that ensures the reward will not always be negative, thus preventing the agent from losing its way in decision-making.

[0050] Furthermore, the process for achieving the aggregation target of the strongly correlated process in step (5) is as follows:

[0051] Design an aggregated reward factor based on strong correlations. For Agent-H's current decision and the process selected from the previous decision, another... It is a Boolean function that iterates through and queries the assembly knowledge graph. and If a "strong correlation" exists, return 1; otherwise, return 0. Then, after the agent completes its process selection at time t, the aggregate reward is calculated as follows:

[0052]

[0053] The reward value set here is the unit reward, meaning it only confirms whether it's based on a reward; the specific reward value is determined by the reward weighting during aggregation. adjust.

[0054] Furthermore, the implementation process of step (6) is as follows:

[0055] A multilayer perceptron (MLP) is used to encode the state vector and output the Q-values ​​of the corresponding action set. For Agent-H, the encoded Q-values ​​of the action space are output. ,in For Agent-H output ,in ;

[0056] The two agents each use independent experience replay buffers, B H and B R ,according to Storage; during the training phase, the network is periodically updated by sampling in small batches from its respective replay library, without being forcibly bound to the event triggering time; when the Q value is updated, for Agent-H:

[0057]

[0058] in, These are the parameters of agent H. It is a discount factor. It is the Q-value function of agent H. It is the experience replay buffer for agent H. It's a reward. and These are the current state and the state at the next moment, respectively. and These are the current action and the next action, respectively.

[0059] Agent-R's update rules are as follows:

[0060]

[0061] During the exploration phase of training, the agent follows an ε-greedy scheduling strategy to ensure sufficient exploration; during the testing and application phases, it follows a greedy strategy to ensure the selection of the most valuable action.

[0062] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0063] 1. This invention is based on the explicit modeling of three relationships between processes using an assembly knowledge graph, and uses this graph to generate action masks during the reinforcement learning stage to mask any conflicting candidate actions in real time, fundamentally avoiding the disconnect problem of "sequences are feasible but actions are not"; this structured constraint mechanism can effectively reduce the search space, enabling the algorithm to maintain a high convergence speed and solution executability even in complex assembly scenarios.

[0064] 2. This invention introduces an event-triggered human-centric collaborative mechanism into a multi-agent reinforcement learning framework for the first time, and clarifies human-machine collaborative decision-making strategies in three different scenarios. These strategies are more in line with the human-centric collaboration logic in industrial settings. This asynchronous event-triggered mechanism effectively avoids the ineffective decision-making overhead caused by periodic polling, and achieves higher decision-making real-time performance and natural interaction.

[0065] 3. This invention introduces a nonlinear cumulative fatigue model into the reward function, simultaneously incorporating process difficulty, execution time, and fatigue growth rate into the multi-objective optimization process. This enables the learning strategy to actively suppress the accumulation of human fatigue and balance the human-machine load while maintaining overall assembly efficiency. Experiments show that this mechanism can effectively reduce the peak value of human fatigue and improve the stability of the collaborative cycle time, which is of great significance for ensuring the long-term operational quality of human-machine collaboration. Attached Figure Description

[0066] Figure 1 This is a flowchart of the present invention;

[0067] Figure 2 This is a flowchart of the intelligent agent decision-making process under the human-centered collaborative mechanism triggered by the event of this invention;

[0068] Figure 3 This is a schematic diagram of a dual-agent DQN network structure. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings.

[0070] like Figure 1 As shown, this invention provides a human-machine collaborative assembly sequence planning method based on dual-agent DQN, specifically including the following steps:

[0071] Step 1: Construct the learning environment for the human-centered interactive dual-agent DQN reinforcement learning algorithm, namely the assembly process knowledge graph.

[0072] The knowledge graph uses assembly operations at the process level as nodes. Edges between nodes contain at least three types of relationships: priority relationships, non-simultaneous relationships (constraints preventing parallel execution of two processes within a given time window), and strong association relationships (process groups that can be merged in the same execution context and typically share tools or adjacent operation surfaces). Additionally, each process node is bound to an attribute vector, with attributes including at least three types of indicators: time taken for independent manual execution, independent robot execution, and human-robot collaborative execution of the process; difficulty factor; and fatigue coefficient. This graph provides environmental support for subsequent algorithm training.

[0073] The constructed knowledge graph mainly contains three types of elements: First, it uses all the assembly steps required to assemble a product as elements, creating nodes representing all the necessary assembly steps within the Neo4j knowledge graph. Second, it creates edges connecting these nodes based on the relationships between assembly steps, including three types of relationships: "priority relationship" describing the necessary sequence of assembly steps, "non-simultaneous relationship" restricting time and space conflicts between assembly steps, and "strong association relationship" describing the similarity and correlation between assembly steps. Third, it deduces and organizes nine state attributes corresponding to different steps based on their differences and characteristics, including assembly time for independent manual execution, robot independent execution, and human-machine collaborative execution, as well as operation difficulty factors and personnel fatigue coefficients, totaling nine attributes. These attributes are then bound as node labels to each step node.

[0074] Step 2: Vectorize the knowledge graph reinforcement learning environment constructed in Step 1 into the agent environment state. At the same time, clarify the other state space variables of the two agents in the human-centered interactive dual-agent DQN algorithm and their meanings, including at least the global assembly state, the list of executable processes and attributes, and the agent's observable state.

[0075] The process nodes, three types of relation edges, and process attribute labels are read from the knowledge graph. After vectorization, the state space S of the artificial intelligence agent (Agent-H) is defined. H Specifically, it includes:

[0076] 1) Global Assembly Status: Completed / Remaining Operations Status, and a list of currently executable operations obtained based on the "priority over relation" of the assembly knowledge graph (KG). KG refers to a data structure that uses operations as nodes, expresses the priority, non-simultaneity, and strong association relationships between operations in the form of triples (node-relationship-node), and binds attributes such as time, difficulty, and fatigue to the nodes.

[0077] 2) List of executable processes and their attributes: Time T in both independent and collaborative execution modes. k Difficulty D k Fatigue factor F k .

[0078] 3) Observable status of Agent-R: Idle / Working, Current Process and Estimated Remaining Time.

[0079] 4) Cumulative fatigue level of Agent-H .

[0080] The process nodes, three types of relation edges, and process attribute labels are read from the knowledge graph. After vectorization, the state space S of the robot agent (Agent-R) is defined. R Specifically, it includes:

[0081] 1) Global Assembly Status: The global assembly status after Agent-H's latest action update;

[0082] 2) List of executable operations and their attributes: Operations not selected by Agent-H, or specifically enabled due to Agent-H's decision, and their corresponding robot time / difficulty and other attributes;

[0083] 3) Observable status of Agent-H: Idle / Working, Current Process and Estimated Remaining Time;

[0084] 4) Collaboration Request Identifier: A specific signal triggered by Agent-H initiating a collaboration request, containing information such as the collaboration process ID. The constructed agent state space is summarized in Table 1:

[0085] Table 1. Agent State Space (S) H S R )

[0086]

[0087] Step 3: In the knowledge graph reinforcement learning environment constructed in Step 1, action masks are generated based on the relationships between nodes and node attributes. Candidate actions that violate the "priority over relationships" rule or conflict with resources are marked as illegal actions using the masks. Furthermore, differentiated action selection strategies are designed for different agents (Agent-H / Agent-R) to form a real-time action space for each agent.

[0088] Constructing the action space A of the artificial intelligence agent (Agent-H) H To ensure the dominant role of human labor in assembly production, Agent-H must have the decision-making power to initiate collaborative tasks. Therefore, Agent-H's action space can be divided into two parts for discussion, specifically including:

[0089] (3.1) Constructing the action set that Agent-H can choose to independently execute processes: First, an action selection strategy is formulated based on conditional filtering and random sampling of process attributes. Conditional filtering is performed on three inherent attributes of different processes when completed manually: time, difficulty coefficient, and fatigue factor. Then, a specific process is randomly sampled to obtain the action. For example, Top-k sorting is used for a certain attribute, meaning a process is randomly selected from the top k processes with the highest time ranking; or Bottom-k sorting is used for the difficulty coefficient, meaning a process is randomly selected from the bottom k processes with the lowest difficulty coefficient. Therefore, two conditional sampling methods are used for each of the three attributes to provide six selectable actions for Agent-H's action space. This process selection method conforms to the priority strategy for a specific attribute in actual production processes. At the same time, the combination of conditional filtering and random sampling enhances the model's generalization learning ability. In addition, other action selection strategies are set based on the dependency of process relationships. In the assembly sequence planning process, the evaluation index not only needs to consider the various attributes of the processes but also the influence of the relationships between the sequences to be planned. This strategy primarily considers the relationships between processes when making action selections, providing three optional actions in the action space: 1. Select the process with the fewest "priority over" relationships. Fewer "priority over" relationships indicate that the process is relatively independent and does not provide support for other processes. It is often the last process in the assembly of a component. Prioritizing the completion of this process ensures that the part is assembled into a component first, which helps reduce the complexity of the assembly space. 2. Select the process with the most "priority over" relationships. More "priority over" relationships indicate that this process is a prerequisite for many other processes. Prioritizing the execution of the current process expands the selectable options in the process array, which is beneficial for the operation of the entire assembly process and algorithm training. 3. Select the process with the most "dissimilarity" relationships. More "dissimilarity" relationships indicate that the parts involved in this process provide assembly support for many other parts, or are prone to spatial interference with other parts during assembly. Prioritizing the execution of this process helps eliminate these conflicts and facilitates the smooth execution of the overall assembly plan.

[0090] (3.2) Construct the set of actions that Agent-H can select for collaborative execution procedures. This set of actions is similar to the set of actions that can be selected for independent execution procedures. The only difference is that the procedure attributes considered are three attributes for human-machine collaboration: collaboration time, collaboration difficulty coefficient, and collaboration fatigue factor. Therefore, Agent-H also has 9 actions to choose from for collaborative execution procedures.

[0091] Constructing the action space A of the robotic intelligent agent (Agent-R) RAgent-R's task is to autonomously execute tasks or collaborate with Agent-H, and all its actions are predicated on supporting and cooperating with human-led decision-making. Therefore, Agent-R's action space is relatively simple, requiring only consideration of the case where the robot executes a single process independently. Similar to the previous case of a human executing a single process independently, there are two different strategies: a conditional filtering-random sampling strategy based on process attributes and a dependency assessment strategy based on process relationships. These will not be elaborated upon here; the detailed agent action space is shown in Table 2.

[0092] Table 2 Agent Action Space (A) H A R )

[0093]

[0094] Step 4: For human-machine collaborative assembly scenarios, design as follows: Figure 2 The event-triggered, human-centric joint decision-making mechanism shown operates as follows: Upon completion of any action, a new round of decision-making is triggered. Event types include "Agent-H completes the action," "Agent-R completes the action," and "both complete simultaneously." Preferably, when both are ready simultaneously, following the human-centric principle, Agent-H first selects the action based on the current shared state and action mask, and then Agent-R selects a compatible action in the updated state to achieve parallel or collaborative execution. This mechanism replaces fixed-sequence rounds with event-triggered events, reducing ineffective decision-making overhead and improving real-time performance.

[0095] When Agent-H completes a task while Agent-R is not ready, the agent interaction and decision-making mechanism will only trigger Agent-H's decision-making. Agent-H can choose to execute the task independently (selecting any action from the set designed in step 311) or choose to initiate collaboration (selecting any action from the set designed in step 312). If it chooses to initiate collaboration, Agent-H will immediately enter a resting state and wait until Agent-R becomes available and responds to the collaboration request.

[0096] When Agent-R completes a process but Agent-H is not ready, the agent interaction and decision-making mechanism will only trigger Agent-R's decision-making. First, it will check whether there is a collaboration request initiated by Agent-H in the environment. If so, it will respond first. Otherwise, it can choose any action to execute from its own action space, i.e., the action set designed in step 320.

[0097] When Agent-H and Agent-R complete the process simultaneously, the agent interaction and decision-making mechanism is designed so that Agent-H is triggered to make a decision first, and the knowledge graph environment is updated based on the action selection result. Then Agent-R is triggered to make a decision.

[0098] Step 5: Develop multi-objective reward rules for fatigue perception that are suitable for human-machine collaborative assembly scenarios. The reward function plays the role of transforming high-level planning objectives into learnable scalar signals. It needs to consider four types of objectives: efficiency (completion time), suitability (human / machine-task matching), human-centered fatigue suppression, and aggregation of strongly related processes, to promote a balance between human and machine workload. Specifically, this includes:

[0099] (5.1) To meet the efficiency objective, design a reward factor that minimizes the completion time, as shown in the following expression.

[0100]

[0101] in, These represent the time required to complete process k in three modes: manually, robotically, and through human-robot collaboration. During training, intensive time rewards are applied. The training direction can only be correct within a local field of view. To minimize the total training time, a final time reward needs to be introduced. :

[0102]

[0103] in, These represent the time required to complete all processes entirely by human workers, robots alone, and human-robot collaboration, respectively. This indicates the required working time for the algorithm planning scheme. This reward ensures that a reasonable algorithm planning scheme is rewarded and an unreasonable scheme is penalized. At the same time, the shorter the total working time of the plan, the greater the reward. It is a coefficient that adjusts the sensitivity of rewards.

[0104] (5.2) To meet the competency objective, a human / machine-task matching reward factor is designed. The core of the competency objective is to ensure that task allocation conforms to the respective strengths and capability limitations of humans and robots, avoiding the assignment of tasks that exceed their capabilities or are too difficult to a particular executor. Similar to the above-mentioned time-intensive reward design principle, the difference lies in the difficulty coefficient having a range of 0 to 1. Therefore, the difficulty reward design is as follows:

[0105]

[0106] in, These represent the difficulty coefficients corresponding to the three modes of completing process k: alone by a human, alone by a robot, and in collaboration between humans and robots. This is the difficulty reward coefficient, ensuring that the difficulty reward matches the other rewards in terms of magnitude.

[0107] (5.3) To meet human-centered goals, a nonlinear fatigue accumulation-recovery reward factor is designed. In actual production scenarios, fatigue not only affects work efficiency but is also a key factor leading to operational errors and impacting product quality. Traditional human fatigue models, with their simple linear representation of the fatigue coefficient, fail to capture the dynamic characteristics of fatigue: the fatigue state itself affects the subsequent fatigue accumulation rate, and rest can quickly restore fatigue. To reflect the true rhythm of "execution-rest-re-execution," accumulation-recovery coupling is introduced on the Agent-H side, and fatigue rewards are designed based on this, specifically including:

[0108] 1) Design a nonlinear fatigue accumulation model. In actual production, when an individual is already fatigued, performing the same task becomes more strenuous, and fatigue increases more rapidly. Therefore, it is proposed that the increment of fatigue depends not only on the fatigue coefficient of the task itself but also on the amplification effect of the current accumulated fatigue level. When Agent-H selects to execute a fatigue-inducing process k (an independent process) at time t... or collaborative process ), its fatigue increment The definition is as follows:

[0109]

[0110] in, β is the basic fatigue coefficient for process k. β is the fatigue sensitivity coefficient. This coefficient controls the nonlinearity of fatigue accumulation: the larger β is, the stronger the amplification effect of existing fatigue on newly generated fatigue. This model makes the agent more cautious in selecting high-fatigue tasks when the fatigue coefficient is high. Therefore, after executing a new process, the new accumulated fatigue level... Updated to:

[0111]

[0112] 2) Design a fatigue recovery model. In the algorithm framework of this study, Agent-H will have a rest period after initiating a collaboration request. Numerous ergonomic studies have confirmed that even short rests can significantly reduce muscle fatigue and are crucial for maintaining work capacity. Therefore, an exponential decay model is adopted to describe this recovery process, which can more realistically simulate the physiological recovery law.

[0113] This recovery model is triggered after Agent-H selects collaborative step k. Assume that Agent-R still has [time / day] to complete its current task. This period of time constitutes Agent-H's effective rest period. During this time, accumulated fatigue... Attenuation occurs:

[0114]

[0115] in, It is the fatigue recovery rate. The value determines the recovery speed and can be set by combining ergonomic data and expert experience.

[0116] Considering that Agent-H will immediately execute the collaborative process k after resting, and this process will also generate fatigue, Agent-H will definitely trigger a process of fatigue recovery followed by fatigue accumulation after selecting collaborative process k. Therefore, the final accumulated fatigue after completing the entire "wait-collaboration" process will be... for:

[0117]

[0118] 3) Integrating the fatigue reward factor from the fatigue accumulation-recovery model, and drawing on the core idea of ​​Potential-Based Reward Shaping (PBRS), the dynamic change process of the aforementioned fatigue level is transformed into an effective reward signal. This paper considers the negative value of accumulated fatigue as the potential energy of the worker's "fatigue state":

[0119]

[0120] Here, C is a positive constant, and the goal of Agent-H is to maintain a high potential energy state. Based on this, this paper will discuss fatigue reward. It is directly defined as the negative value of the cumulative change in fatigue level:

[0121]

[0122] in, This is the fatigue reward coefficient, to ensure that fatigue rewards match the other rewards in terms of magnitude. This is a basic reward function that ensures the reward doesn't remain negative, preventing the agent from losing its decision-making direction. This reward function ensures that Agent-H is penalized when task execution leads to a significant increase in fatigue, with the penalty proportional to the magnitude of the fatigue increase. Therefore, performing tasks under high fatigue conditions will result in more severe penalties. Similarly, Agent-H receives a larger reward for resting and recovering from fatigue. This directly incentivizes Agent-H to proactively choose collaborative tasks that trigger rest and waiting when its fatigue level is high, thus achieving an intelligent, self-regulating fatigue management strategy.

[0123] (5.4) To satisfy the aggregation goal of strongly correlated processes, an aggregation reward factor based on strong correlation is designed. In actual assembly scenarios, continuous use of the same tool or operation of parts in adjacent areas can usually form a more efficient and smooth operation sequence. Aggregation reward is a positive and immediate incentive. An agent will receive this reward if and only if there is a strong correlation between the new process selected by the agent and the previous process it performed. To achieve this, the agent needs to track the process performed in the previous decision from the state space. Assume... For Agent-H's current decision and the process selected from the previous decision, another... Let be a Boolean function that iterates through and queries KG. and Does a "strong association" exist? Return 1 if it exists, otherwise return 0.

[0124] Then, when the agent completes its process selection at time t, the aggregate reward is calculated as follows:

[0125]

[0126] The reward value set here is the unit reward, meaning it only confirms whether it's based on a reward; the specific reward value is determined by the reward weighting during aggregation. adjust.

[0127] (5.5) Integrate all reward factors, design a comprehensive reward function and training score. Based on the above complex reward system design, the instantaneous rewards R that the two intelligent agents can obtain at decision time step t are respectively t The whole can be represented as:

[0128]

[0129] in, These represent four non-negative weight coefficients, and have... For Agent-R, there is It is always 0.

[0130] At the end of training, each agent receives the following total training score:

[0131]

[0132] This scoring function is also the final evaluation metric during the agent training process.

[0133] Step 6: As Figure 3As shown, a dual-agent DQN network structure is established. The artificial intelligence agent (Agent H) and the robotic intelligence agent (Agent R) each maintain independent online Q-networks and target Q-networks, and perform asynchronous interaction and experience replay updates in a shared state. Preferably, the network architecture and hyperparameters of the two agents are configurable and adjustable to adapt to heterogeneous action spaces and reward sensitivity.

[0134] The network structure is designed such that the two agents involved in the algorithm, Agent H and Agent R, exist in the same knowledge graph environment but do not share parameters or observations. Simultaneously, each agent maintains an "online network". "and target network" The state vector is encoded using a multilayer perceptron (MLP), and the Q-values ​​of the corresponding action sets are output. For Agent-H, the encoded Q-values ​​of the action space are output. ,in Similarly, there is also output for Agent-H. ,in .

[0135] The algorithm learning process is clearly defined. To balance the asynchronicity of engineering timing with the efficiency of learning samples, the learning process decouples "event-triggered decision-making" from "mini-batch updates." Each of the two agents uses an independent experience replay buffer. H and B R ,according to Storage. During the training phase, the network is periodically updated by sampling in small batches from its respective replay library, without being forcibly bound to event triggering times. When the Q-value is updated, for Agent-H:

[0136]

[0137] in, These are the parameters of agent H. It is a discount factor. It is the Q-value function of agent H. It is the experience replay buffer for agent H. It's a reward. and These are the current state and the state at the next moment, respectively. and These are the current action and the next action, respectively.

[0138] Similarly, the update rules for Agent-R also include:

[0139]

[0140] During the exploration phase of training, the agent follows an ε-greedy scheduling strategy to ensure sufficient exploration. During the testing and application phases, a greedy strategy is followed to ensure the selection of the most valuable action.

[0141] Step 7: Based on the decision and execution results of Step 6, perform local incremental updates on the knowledge graph: refresh the status of completed nodes, update the affected nodes, edges and related node attributes to ensure the consistency and timeliness of the status in subsequent decisions.

[0142] Example: Based on the assembly process and manual for a three-shaft manual transmission, all process-level operation steps required to assemble the product are listed. Three types of relationships between assembly processes are summarized: priority relationship, non-simultaneous relationship, and strong association relationship. Nine attributes of each assembly process are compiled, including the time, difficulty factor, and fatigue coefficient for independent manual execution, independent robot execution, and human-machine collaborative execution. Then, based on all the necessary data, a knowledge graph for the assembly of a three-shaft manual transmission is constructed using the JanusGraph database, with all process-level operation steps as nodes, inter-process relationships as connecting edges between nodes, and process attributes as node labels. This knowledge graph serves as the learning environment for subsequent reinforcement learning algorithms.

[0143] All nodes, relationships, and labels in the constructed assembly knowledge graph are vectorized into environmental states that can be observed by the agents. Simultaneously, other state-space variables observable by the two agents in the proposed human-centered interactive dual-agent DQN reinforcement learning algorithm are defined, including: global assembly state, product assembly progress, observable states of each agent, list of executable processes, and their attributes.

[0144] To determine the action space for each agent in a multi-agent reinforcement learning algorithm, action masks are generated and illegal actions are masked. Before selecting an action, the Q-value of illegal actions is set to a minimum, and these actions are then ignored during the ε-greedy selection phase. To reflect the dominant role of humans in actual assembly production scenarios, Agent-H, representing humans, has priority decision-making and collaboration initiation authority, while Agent-R, representing robots, prioritizes responding to human requests and has autonomous decision-making capabilities in idle states. Based on these principles, differentiated action spaces for the two agents are designed. The action space A of Agent-H is designed based on two action selection strategies: process attribute condition filtering-random sampling and process relationship dependency. H The 18 selectable actions are similarly designed for Agent-R, which has an action space A. R The eight optional actions;

[0145] For the specific scenario of human-machine collaborative assembly, a human-centric joint decision-making mechanism based on event triggering is designed for agent interaction in multi-agent reinforcement learning algorithms to align with actual human-machine collaborative assembly scenarios. Under this mechanism, the completion of any process will trigger a new round of decision-making. According to the state of each agent at the time of triggering, the triggering scenarios can be subdivided into: Agent-H is ready while Agent-R is not ready, Agent-R is ready while Agent-H is not ready, and both Agent-H and Agent-R are ready.

[0146] Develop multi-objective reward rules for fatigue perception that are suitable for human-machine collaborative assembly scenarios, while taking into account four types of objectives: completion time of the solution, human / machine task matching degree, human fatigue factors, and aggregation of strongly related processes.

[0147] In this embodiment, an algorithm framework is established to realize the learning and decision-making process of the human-centered interactive dual-agent DQN reinforcement learning algorithm. This framework consists of Agent H, representing the artificial intelligence agent, and Agent R, representing the robot. Each agent independently maintains a pair of Q-networks: an online network and a target network. The parameters of both networks are periodically updated synchronously, thus forming a "dual Q-structure." The two agents interact asynchronously in a shared environment and update their policies independently. The input state vector of each agent includes the state encoding of executable process nodes in the current knowledge graph, the observable states and resource occupancy flags of each agent, the action mask, the current time step, and the cumulative reward statistics.

[0148] The network structure is configured as follows: the input layer is 256-dimensional, and each process node is represented by a fixed-length feature vector. The top-20 candidate processes are selected and concatenated in a fixed order, with zeros added if necessary; the first hidden layer is a fully connected layer with 256 neurons and ReLU activation; the second hidden layer is a fully connected layer with 128 neurons and ReLU activation; the third hidden layer is a fully connected layer with 64 neurons and ReLU activation; and the output layer is a fully connected layer corresponding to the number of actions that can be executed in the action space of each agent.

[0149] Hyperparameter settings: Learning Rate: 1×10⁻ 4The system employs the Adam optimizer, with a discount factor (γ) of 0.95, a replay buffer size of 100,000 samples, a batch size of 64, a target update frequency of 500 steps, an ε-greedy exploration strategy (initial ε = 1.0, linearly decaying to 0.05 with 50,000 decay steps), a reward normalization range of [-1, 1], a gradient clipping threshold of 5.0, and a maximum training episode count of 5,000. Each episode contains a complete assembly task sequence.

[0150] Under the above parameter configuration, the present invention converged after approximately 3000 training iterations, with the average comprehensive reward per task episode stabilizing within ±0.5. Compared to ordinary DQN without action masking, the training time was reduced by approximately 37%, human-machine collaboration efficiency improved by approximately 12%, and peak human fatigue decreased by approximately 20%. Experiments show that the algorithm can stably output executable assembly sequences with balanced human and machine workloads under this parameter configuration, verifying the feasibility and effectiveness of the method of the present invention. Based on the action selection and state updates during the training process, the constructed knowledge graph of automotive three-axle manual transmission assembly is dynamically and incrementally updated locally to ensure node state refresh, providing a stable and reliable training environment for the reinforcement learning algorithm.

[0151] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A dual-agent DQN based human-robot collaborative assembly sequence planning method, characterized in that, Comprise the following steps: (1) Construct human-based interactive dual-agent DQN reinforcement learning algorithm learning environment, that is, construct assembly knowledge graph based on process level assembly process; The knowledge graph takes process level assembly operation as a node, and establishes priority relationship, different time relationship and strong correlation relationship between nodes, and binds attributes on the nodes; (2) Vectorize the knowledge graph into the environment state representation of the agent, and define the state space of the agent Agent-H representing human and the agent Agent-R representing robot respectively; (3) Generate action mask based on the relationship between nodes of the knowledge graph and node attributes to mask candidate processes that violate priority relationship or cause resource conflict, and mark them as illegal actions with the mask; Design differentiated action selection strategies for different agents to form real-time actionable action space for each agent; (4) Adopt event-triggered human-based priority joint decision mechanism, when Agent-H and Agent-R are ready at the same time, Agent-H selects the process first, and then Agent-R selects compatible process under the updated state; (5) Formulate fatigue perception multi-objective reward rules in line with human-robot collaborative assembly scenarios, taking into account four types of targets including completion time, human-robot task matching degree, human fatigue factor and strong correlation process aggregation, to promote human-robot load balance; (6) Establish online Q network and target Q network for Agent-H and Agent-R respectively and maintain experience replay buffer respectively, periodically sample and update online network parameters from replay buffer in small batches during training stage, and target network is updated synchronously or softly according to predetermined strategy; (7) According to the training and execution results, the knowledge graph is locally updated to keep the environment up-to-date.

2. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The attributes in step (1) include the working hours, difficulty factor and fatigue coefficient of human independent execution, robot independent execution and human-robot collaboration respectively.

3. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The state space of Agent-H and Agent-R in step (2) includes global assembly state, executable process list and attributes, and observable state of the agent; The state space of Agent-H further includes the accumulated fatigue level of Agent-H; The state space of Agent-R further includes collaboration request identifier.

4. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The step (3) is implemented as follows: Construct the action set that Agent-H can select to execute the process independently, and develop action selection strategy based on conditional filtering-random sampling of process attributes, and after conditional screening of three inherent attributes of human alone in different processes, namely working hours, difficulty coefficient and fatigue factor, a specific process execution is obtained by random sampling; Set other action selection strategies based on the degree of dependence of process relationship, in the process of assembly sequence planning, the relationship between the to-be-planned sequences needs to be considered when evaluating the index, and the process relationship is considered when selecting actions; Select the process with the least priority relationship, the process with the most priority relationship and the process with the most different time relationship; The construction of the Agent-H selection collaborative action set for the execution of the process is similar to the construction of the independent action set, except that the process attributes considered are the three attributes for human-machine collaboration, namely, collaboration time, collaboration difficulty coefficient, and collaboration fatigue factor. Action space A of Agent-R R The task of Agent-R is to autonomously perform a task or to work in collaboration with Agent-H. In the case of a robot performing a task alone, the following actions are considered: short time-consuming, low difficulty, low dependency, high conflict, and high response to collaboration requests.

5. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The step (4) is implemented as follows: When the Agent-H completes the process and the Agent-R is not ready, only the Agent-H intelligent agent decision is triggered, and the Agent-H selects independent execution of the process or initiates collaboration; if collaboration is selected, the Agent-H immediately enters a resting state and waits until the Agent-R is idle to respond to the collaboration request; When the Agent-R completes the process and the Agent-H is not ready, only the Agent-R intelligent agent decision is triggered, and the environment state is first detected to determine whether there is a collaboration request initiated by the Agent-H intelligent agent, and if there is, the response is prioritized, otherwise, a random action is selected from the action space for execution; When the Agent-H and the Agent-R complete the process at the same time, the Agent-H intelligent agent decision is triggered first, and the knowledge graph environment is updated according to the action selection result, and then the Agent-R intelligent agent decision is triggered.

6. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The step (5) is implemented as follows: To meet the efficiency target, a minimum completion time reward factor is designed, and the expression is as follows: wherein, respectively represent the required working hours for process k to be completed by manual, robot, and human-robot collaboration, respectively. During the training process, a dense working hour reward Only limit the training direction correctly under the local field of view, and introduce a final working hour reward to compress the total working hours as much as possible : wherein, respectively represent the required working hours of all processes completed by manual alone, robot alone, and manual-robot cooperation, represents the required working hours of the algorithm planning scheme, and the reward ensures that the algorithm planning scheme is rewarded when it is reasonable and punished when it is not reasonable, and the shorter the total working hours of the planning is, the greater the reward is; is a coefficient for adjusting the sensitivity of the reward.

7. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The human-machine task matching degree target is achieved as follows: A human-machine task matching degree reward factor is designed to ensure that the task allocation is consistent with the respective advantages and capacity limitations of humans and robots, and to avoid assigning tasks that exceed their capacity or are too difficult to a certain execution party: wherein, respectively represent the difficulty coefficients corresponding to the three modes of completing the process k by human alone, robot alone, and human-robot cooperation, is the difficulty reward coefficient, to ensure that the difficulty reward matches the remaining rewards in the order of magnitude.

8. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The human fatigue factor target is achieved as follows: A non-linear fatigue accumulation-recovery reward factor is designed, and a cumulative-recovery coupling is introduced on the Agent-H side, and a fatigue reward is designed: A nonlinear fatigue accumulation model is designed. When Agent-H chooses to execute a fatigue-producing process k at time t, independent processes or cooperative processes whose fatigue increments are defined as follows: wherein, is the base fatigue coefficient of process k, β is the fatigue sensitivity coefficient, which controls the degree of nonlinearity of fatigue accumulation, the larger β is, the stronger the amplification effect of existing fatigue on newly generated fatigue; after performing a new process, the new cumulative fatigue level is updated as: A fatigue recovery model is designed, which adopts an exponential decay model to describe the recovery process of the artificial agent. It is triggered after the agent-h selects the cooperative process k. It is assumed that the agent-r still needs time to complete its current task, and the accumulated fatigue decays: wherein, is the fatigue recovery rate, which determines the recovery speed; Agent-H triggers a process of fatigue recovery before fatigue accumulation after selecting the collaborative process k, and the final accumulated fatigue after completing the entire "waiting-collaboration" process is: The fatigue reward factor of the fatigue accumulation-recovery model is integrated, and the dynamic change process of the fatigue level is converted into an effective reward signal, and the negative value of the cumulative fatigue is regarded as the potential energy of the "fatigue state" of the worker: where C is a constant, and Agent-H's goal is to maintain a high potential energy state; reward fatigue directly defined as the negative of the change in accumulated fatigue level: wherein, is a fatigue reward coefficient to ensure that the fatigue reward matches the order of magnitude of the remaining rewards, is a base reward to ensure that the reward is not always negative, which would cause the agent to lose the direction of decision-making.

9. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The strong correlation process aggregation target is achieved as follows: An aggregated reward factor based on strong correlation is designed, The procedure is selected for the current decision and the last decision of Agent-H, is a Boolean function that traverses the query from the assembly knowledge graph, and whether there is a "strong correlation", if there is 1, otherwise 0; then when the intelligent agent completes the procedure selection at time t, the aggregated reward is calculated as follows: The reward value set here is a unit reward, that is, only whether to base on the reward is confirmed, and the specific reward value is the reward weight when the reward is summarized Adjustment.

10. The dual-agent DQN-based human-robot collaborative assembly sequence planning method according to claim 1, characterized in that, The step (6) is implemented as follows: The state vector is encoded by using a multi-layer perceptron (MLP) and the Q values of the corresponding action set are output. For Agent-H, the encoded Q values of the action space are output wherein ; and the output of Agent-H is wherein ; The two agents each use independent experience replay buffers, B H and B R ,according to Storage; during the training phase, the network is periodically updated by sampling in small batches from its respective replay library, without being forcibly bound to the event triggering time; when the Q value is updated, for Agent-H: wherein, is a parameter of the agent H, is a discount factor, is a Q-value function of the agent H, is an experience replay buffer of the agent H, is a reward, and are a current state and a next time state, respectively, and are a current action and a next action, respectively; The update rule of the Agent-R is as follows: In the exploration phase of training, the intelligent agent follows the ε-greedy scheduling strategy to ensure sufficient exploration; in the test and application phase, the greedy strategy is followed to ensure the selection of the most valuable action.

Images