Cooperative control method of dual bionic dexterous hands based on multi-agent system

Abstract

This invention discloses a collaborative control method for a dual-bionic dexterous hand based on a multi-agent system. The method comprises: utilizing a causal graph attention network to analyze the dynamic influence relationships between agents in real time, and generating a computational sequence that reflects true causal logic through topological sorting; evaluating the marginal contribution of a particular agent based on this computational sequence by comparing the system value after including and excluding the actions of that agent, and calculating its advantage function value based on this; wherein all values ​​required for calculating the marginal contribution are generated in parallel by an efficient value network in a single forward propagation; and utilizing this advantage function value to update the corresponding control network. This invention improves the accuracy and computational efficiency of collaborative control.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent reinforcement learning and robot control, and in particular to a dual-bionic dexterous hand collaborative control method based on a multi-agent system. Background Art

[0002] In practice, a large number of tasks involve the collaborative manipulation of irregular, fragile, or large, complex objects, far exceeding the capabilities of traditional dedicated robotic arms. Consequently, the bionic dexterous hand, with its highly biomimetic properties and task generalization capabilities, has emerged. By mimicking the multi-joint, multi-degree-of-freedom structure of the human hand, it can flexibly utilize a vast array of tools designed for humans, breaking free from the constraints of specialized tools and demonstrating unparalleled adaptability. However, current research and applications largely focus on single-handed manipulation, which significantly limits the complexity and stability of its operations.

[0003] Currently, the mainstream approach in academia and industry for the coordinated control of multi-agent systems, such as dual bionic dexterous hands, is based on Multi-Agent Reinforcement Learning (MARL), particularly the Centralized Training with Decentralized Execution (CTDE) framework. Within this framework, several representative control methods have emerged. The first category is based on a shared critic, such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Multi-Agent Proximal Policy Optimization (MAPPO). During training, these methods utilize a centralized critic network to evaluate the joint actions of all agents (i.e., the two dexterous hands). This critic network has access to global state information and action information for all agents, providing relatively stable and comprehensive gradient guidance to the distributed actor networks. Another category of methods has explored the problem of credit assignment, focusing on more rationally decomposing the global reward signal received by the team to each individual agent. For example, based on the idea of ​​value-decomposition, network structure design is used to ensure the consistency of the sum of individual value functions with the global value function, such as the VDN and QMIX algorithms. Furthermore, some research has introduced counterfactual thinking, assessing the impact of the presence or absence of an agent on the team's total revenue to preliminarily determine its marginal contribution.

[0004] However, when applying the above-mentioned existing methods to scenarios such as dual bionic dexterous hands with high joint dimensions, tight dynamic coupling, and strong task timing logic, there are still causal ambiguity and economic issues in the distribution of contributions. In other words, it is difficult for the algorithm to distinguish which hand and which action should be attributed to the success or failure of the task. At the same time, in order to calculate this credit, very complex calculations are often required, making the whole process slow and resource-intensive, making it difficult to apply in reality. Summary of the Invention

[0005] The purpose of the invention is to provide a collaborative control method for dual bionic dexterous hands based on a multi-agent system, in order to solve at least one technical problem existing in the prior art.

[0006] The technical solution is a collaborative control method for dual bionic dexterous hands based on a multi-agent system, including:

[0007] Obtaining environmental state data from the multi-agent system and generating current environmental state data; and determining a computation sequence for multiple agents of the dual-bionic dexterous hand based on the current environmental state data;

[0008] Based on the calculation sequence and the current environment state data, the advantage function value is calculated for each agent; accordingly, the network weight of the control network corresponding to the agent is updated.

[0009] Beneficial effect: The present invention dynamically determines the calculation sequence that reflects the true causal dependency between intelligent agents, solving the causal ambiguity problem of contribution allocation caused by fixed or arbitrary calculation order in the prior art; it provides each intelligent agent with an accurate marginal contribution evaluation, while ensuring the accuracy of contribution allocation and improving the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of the steps of a collaborative control method of dual bionic dexterous hands based on a multi-agent system provided in an embodiment of the present application.

[0011] Figure 2 A flowchart of the steps for determining a calculation sequence for multiple intelligent agents of a dual bionic dexterous hand provided in an embodiment of the present application.

[0012] Figure 3 A flowchart of the steps for generating dynamic influence relationships provided in an embodiment of the present application.

[0013] Figure 4 A flowchart of the steps for constructing a calculation sequence provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products or apparatus.

[0016] Research has found that existing methods for handling interactions between agents often suffer from causally ambiguous credit assignment. Specifically, whether using a shared critic or preliminary counterfactual approaches, most implicitly assume that all agents' contributions within a time step occur simultaneously and are equal, or that the order of their interactions is fixed and predetermined. This is seriously inconsistent with the real world. For example, in a task involving two-handed coordination to pass a precision instrument, the posture adjustment and stabilization of the handing hand necessarily precedes the grasping and following actions of the receiving hand in the causal chain. Existing methods employ a fixed computational sequence that is independent of the task state and fail to capture dynamic, asymmetric causal dependencies. This prevents the algorithm from clearly identifying the key drivers of task success, potentially misallocating rewards to subsequent, non-critical actions, or penalizing necessary preparatory actions that have little short-term effect. This severely limits the accuracy of the learned strategy and the ultimate coordination effectiveness.

[0017] Secondly, existing methods face a conflict between homogenized advantages and exponentially increasing computational costs when evaluating individual contributions. On the one hand, shared critic methods estimate a global, undifferentiated generalized advantage function (GAE) for all agents. This means that even if one hand performs an extremely delicate maneuver while the other makes a mistake, the resulting advantage evaluation signals for both hands are averaged or homogenized. This prevents the algorithm from effectively incentivizing truly excellent actions or accurately penalizing poor ones. On the other hand, while counterfactuals can theoretically address this issue, their naive implementation leads to a combinatorial explosion of computational overhead in the context of ambidextrous hands (each with over 20 degrees of freedom). If each agent's marginal contribution requires a separate forward propagation through the main network, the computational burden increases exponentially with the number of agents and the dimensionality of their actions. This makes training extremely lengthy and even fails to meet the fundamental real-time decision-making requirements of real-world tasks.

[0018] like Figure 1 As shown in the figure, a collaborative control method of dual bionic dexterous hands based on a multi-agent system is proposed, which includes the following steps:

[0019] Obtain environmental state data from the multi-agent system and generate current environmental state data;

[0020] Determine the computational sequence for multiple agents of the dual bionic dexterous hand based on the current environmental state data;

[0021] Based on the calculation sequence and the current environment state data, the advantage function value is calculated for each agent;

[0022] The advantage function value calculated for each agent is used to update the network weight of the control network corresponding to the agent.

[0023] In one embodiment of the present application, in order to solve the problems of causal ambiguity and low computational efficiency in the prior art of contribution allocation, a dual-bionic dexterous hand coordinated control method is proposed. From a multi-agent system including the dual-bionic dexterous hands and their manipulated objects, raw information for subsequent decision-making and evaluation is collected. Specifically, the environmental state data s t It is collected at time t, which can be a high-dimensional vector, including the angles, angular velocities, position and posture of all joints of each dexterous hand, and the position, posture, and speed of the manipulated object. t , dynamically generates a calculation sequence that reflects the causal influence logic between each agent. Based on the calculation sequence, the quantitative index of each agent's contribution to the team is accurately calculated, that is, the advantage function value A t . Using the calculated advantage function value A of each agentt i (where i is the agent index), the control network (i.e., Actor network) corresponding to the agent is optimized. Specifically, the network weight θ can be updated by using the clipping objective function in the proximal policy optimization (PPO) algorithm. i The objective function L(θ i ) is defined as follows: L(θ i )=E t [min(r t (θ i )A t i ,clip(r t (θ i ), 1-ε, 1+ε) A t i )]; where r t (θ i ) is the ratio of the new and old strategies, r t (θ i )=πθ i (a t i ∣s t ) / πθ old (a t i ∣s t ), indicating that in state s t Next, the new strategy πθ i Select action a t i The probability of the old policy πθ old The probability ratio of t i is the advantage function value of agent i; ε is the preset clipping coefficient, which can be set to 0.2, for example, to limit the amplitude of strategy update and ensure the stability of training; E t is the expected value operator, and clip is the clipping operation function. By calculating the objective function L(θ i ) for network weight θ i The control network can be updated by taking the gradient of the function and using a gradient optimizer such as Adam for gradient ascent.

[0024] like Figure 2 As shown, according to one aspect of the present application, a calculation sequence is determined for multiple intelligent agents of a dual bionic dexterous hand, including:

[0025] Analyze the current environment state data, evaluate and generate dynamic influence relationships that characterize the interactions between various intelligent agents;

[0026] According to the dynamic influence relationship, a calculation sequence for calculating the advantage function value is constructed.

[0027] like Figure 3 As shown, according to one aspect of the present application, generating a dynamic influence relationship includes:

[0028] The current environment state data is input into the pre-configured causal graph attention network, and each bionic dexterous hand agent is regarded as a node in the graph structure;

[0029] Utilize the attention mechanism of the causal graph attention network to learn and quantify the directional influence strength between nodes under the current environment state;

[0030] The obtained directional influence intensity is output as a dynamic influence relationship.

[0031] According to one aspect of the present application, the data structure of the directional influence strength is a weighted adjacency matrix; in the weighted adjacency matrix, the row and column indices correspond to each bionic dexterous hand agent, and the matrix element value located in the i-th row and j-th column is used to quantify the directional influence strength exerted by agent i on agent j.

[0032] In one embodiment of the present application, the current environment status data s t It is converted into structured data that can quantify the strength of mutual influence between agents. Specifically, it is implemented using an independent, pre-trained Causal Graph Attention Network (CGAT). t The CGAT network treats N bionic dexterous hand agents as N nodes of a graph and learns and quantifies the current state s through its internal graph attention mechanism. t The output of the network is an N×N weighted adjacency matrix W. t Where W t is the weighted adjacency matrix generated at time t; W t (i, j) is the value of the element in the i-th row and j-th column of the matrix. This value is a scalar that quantifies the potential impact of agent i's behavior on agent j. This value, as a dynamic influence relationship, is a key intermediate result.

[0033] like Figure 4 As shown, according to one aspect of the present application, a calculation sequence is constructed, specifically:

[0034] Parse the dynamic influence relationship into a directed graph, where the nodes of the graph correspond to the agents and the directed edges are determined according to the dynamic influence relationship;

[0035] Perform topological sorting on the directed graph to generate a computational sequence that can reflect the order of influence transmission between agents.

[0036] In one embodiment of the present application, the weighted adjacency matrix W t is transformed into a linear, unique computational sequence. Specifically, the weighted adjacency matrix W t Parsed into a directed graph G t In Figure G t In, if W t If the value of (i, j) is greater than a preset threshold (e.g., 0), a directed edge from i to j is constructed between the node representing agent i and the node representing agent j. t Execute the topological sort algorithm. The topological sort algorithm outputs a linear ordering of the nodes, which ensures that for any directed edge from node i to node j in the graph, node i comes before node j. The output of this algorithm is the calculation sequence L t , L t =(i1, i2, ..., i N ); where i k is the index of the agent. t It can reflect the most likely causal influence transmission order between agents in the current state.

[0037] This embodiment introduces a causal graph attention network to analyze the current environmental state data, parses the output dynamic influence relationship into a directed graph, and then performs topological sorting. This transforms the original fixed or arbitrary contribution distribution calculation order into a dynamic calculation sequence that is tightly coupled with the task state and reflects the actual causal logic. In the specific scenario of bimanual collaborative control, this means being able to identify in real time which hand's action is the prerequisite for the other hand's successful operation. For example, in the task of delivering fragile items with both hands, it can automatically identify that the stabilizing action of the supporting hand precedes the contact action of the grasping hand in the causal chain, thereby giving priority to the contribution of the supporting hand in the subsequent advantage function calculation. This embodiment fundamentally solves the causal ambiguity problem caused by the mismatch between the calculation order and physical logic by deeply linking the abstract reinforcement learning algorithm with the physical causal laws of the operation task. This makes the contribution distribution more accurate and reasonable, and the resulting control strategy has higher logic and success rate when executing complex collaborative tasks.

[0038] According to one aspect of the present application, calculating the advantage function value for each agent includes:

[0039] For any target agent in the computation sequence, obtain the joint value of its own actions;

[0040] Get the baseline value without the target agent's own actions;

[0041] By comparing the joint value with the baseline value, the marginal contribution of the target agent is quantified and its corresponding advantage function value is generated.

[0042] According to one aspect of the present application, generating an advantage function value includes:

[0043] Calculate the temporal difference term based on the environmental rewards, joint value, and baseline value obtained from the multi-agent system;

[0044] calculating a marginal contribution term based on the joint value and the current state value corresponding to the current environmental state data;

[0045] The temporal difference term and the marginal contribution term are weightedly summed in the time dimension to generate the advantage function value corresponding to the target intelligent agent.

[0046] According to one aspect of the present application, the joint value, baseline value and current state value are generated by the same efficient value network; through one forward propagation calculation of the efficient value network, the joint value, baseline value and current state value are generated simultaneously.

[0047] According to one aspect of the present application, the internal structure of the efficient value network includes at least:

[0048] State encoder, used to process current environment state data and generate state code;

[0049] Action encoder, used to process the joint actions of each agent to generate action encoding;

[0050] The multi-head attention module is used to receive state encoding and action encoding, and synchronously generate joint value, baseline value and current state value through attention operation.

[0051] According to one aspect of the present application, after the multi-head attention module receives the state code and the action code, the processing process includes:

[0052] Performing a matrix operation on the action code and a preset mask matrix to generate a modified action code set in parallel; wherein each modified action code in the action code set corresponds to a scene in which a specific agent action is removed;

[0053] The state code and the modified action code set are fed into the attention mechanism for operation to simultaneously generate the joint value, baseline value, and current state value.

[0054] In one embodiment of the present application, all the value data required for subsequent computation is efficiently generated in one computation. Specifically, a specially designed efficient value network (i.e., a Critic network) is used. The internal structure of the network includes a state encoder, an action encoder, and a multi-head attention module.t and the joint action of all agents (a t1 ,…,a tN ) as input. The state encoder processes the current environment state data s t Processing generates state encoding. The action encoder processes the joint action to generate action encoding. Among them, in order to realize parallel counterfactual calculation, the multi-head attention module performs key masking operations when performing operations. It performs matrix operations on the action encoding and the preset mask matrix (Mask Matrix). The mask matrix is ​​constructed to be able to generate a modified set of action encodings in parallel at one time. Each encoding in the set corresponds to a scenario in which the contribution of a specific agent's action is eliminated. Through the above structure, the efficient value network can simultaneously generate and output all the values ​​required for subsequent calculations through a single forward propagation calculation: the joint value Q(s t , a t ): The value of all agent actions; the current state value V(s t ): The value of the current state without taking into account any action; Baseline value: In this embodiment, it specifically refers to the state value V(s) at the next moment t+1 ), that is, the value of the next state without taking into account any action.

[0055] Using the generated multiple values, calculate the advantage function value A for each target agent i t i . Calculate two key intermediate terms: the time difference term δ t :δ t = r t +γV(s t+1 ) - Q(s t , a t ); marginal contribution term ψ t :ψ t = Q(s t , a t ) - V(s t ); where r t is the immediate reward obtained from the environment; γ is a preset discount factor used to measure the importance of future rewards, which can be set to 0.99. t and the marginal contribution term ψ t Perform weighted summation of generalized advantage estimates in the time dimension to generate the advantage function value A corresponding to the target agent i t i :A t i =-ψ t +∑ m=0 ∞ (γλ) m (δt+m +ψ t+m ); where λ is the smoothing parameter of the GAE algorithm, which is used to balance between bias and variance and can be set to 0.95; m is the time step index.

[0056] This embodiment provides highly personalized and precise behavioral assessments for each individual agent by constructing a counterfactual advantage function calculation method based on marginal contributions. In scenarios involving bimanual coordinated control, subtle but crucial differences often exist between the movements of the two hands. By calculating the difference between the joint value (including the hand's movement) and the baseline value (excluding the hand's movement), the net impact of the hand's movement on the overall system performance (i.e., the marginal contribution) can be precisely isolated. For example, when one hand exhibits a slight tremor (negative contribution) while the other hand performs a compensatory stabilizing movement (positive contribution), the two can be assigned advantage function values ​​with distinct signs and magnitudes. This completely resolves the technical pain point of traditional shared critic methods, where all agents receive an averaged, undifferentiated, homogenized advantage signal. This allows the learning algorithm to discern right from wrong, providing strong positive incentives for truly beneficial fine movements and precise negative penalties for minor errors, thereby guiding the two hands to learn more sophisticated, coordinated, and robust coordination strategies. By constructing an efficient parallel value network with a built-in masking mechanism, the accuracy of counterfactual evaluations is ensured while addressing the significant computational overhead inherent in these evaluations at the architectural level. In high-dimensional control scenarios involving ambidextrous hands (typically with over 40 degrees of freedom), conventional methods require multiple calls to the value network to calculate the marginal contribution of each hand, resulting in computational time proportional to the number of agents, making the training process extremely lengthy. The value network of this embodiment takes state data and joint actions as input. Through a single matrix operation between its internal action encoder and a preset mask matrix, it simultaneously generates encodings for all scenarios with one missing hand. This parallel design allows all values ​​required for counterfactual evaluation to be output simultaneously in a single forward pass, reducing the computational complexity of value estimation. Combined with specific hardware computing features (such as the parallel computing capabilities of GPUs), this improves training efficiency, addresses the practical challenge of high computational cost for counterfactual methods in complex scenarios, and enables efficient training and deployment of high-dimensional, complex ambidextrous hand systems.

[0057] In another embodiment of the present application, considering that in the coordination of multiple bionic dexterous hands, the estimation of the generalized advantage function may be biased due to the influence of the actions of other dexterous hands, resulting in poor final performance, a method for estimating the generalized advantage function is proposed, specifically: the system includes a critic network that can generate value estimates when there is no agent action input and value estimates when there is agent action input; n bionic dexterous hand agents are arranged in a certain order, which can be arbitrarily specified, but the sequence number of each agent must be fixed; the generalized advantage function of the agent with sequence number n is calculated according to the following formula: δ t =γV(s t+1 )+r t - Q(sa t 0:n-1 );ψ t =Q(sa t 0:n-1 )-V(s t );A t GAE = -ψ t +∑ m=0 ∞ (γλ) m (δ t+m +ψ t+m ); where V(s t+1 ) is the value estimate of the Critic network when there is no agent action input, Q(sa t 0:n-1 ) is the value estimate calculated by the Critic network after inputting the actions of agents numbered 0 to n-1, that is, after inputting all the agent actions, the value estimate of the agent numbered n is eliminated; γ and λ are pre-set fixed parameters; r t is the immediate reward; V(s t ) is in state s t The estimated long-term return of the system without considering the agent's actions; A t GAE is the generalized advantage estimate, δ t is the time difference term, ψ t is the marginal contribution term. Calculate the generalized advantage function of the agent with sequence number i: Simply place the i-th agent at the end of the agent sequence, that is, form the order {1, 2, ..., i-1, i+1, ..., n, i} to calculate the generalized advantage function of the i-th agent. Use the obtained generalized advantage function to calculate the optimization objective function of the weight of the actor network of the corresponding agent according to the following formula: L(θ i )=Φ[min ((πθ i (a t i │o ti ) / πθ old i (a t i |o t i ))A t ,clip(πθ i (a t i │o t i ) / πθ old i (a t i |o t i ), 1-ε, 1+ε) A t )]; where Φ is the expectation operator, πθ i Adopt the current policy parameters θ for agent i i The probability distribution of a t i For agent i in state o t i The action to be performed next; t i is the state of the environment observed by agent i; πθ old i Adopt the old policy parameters θ for agent i old i The probability distribution of A t is the advantage function value. The objective function is optimized based on the weight of the actor network using the gradient descent method. The network weight of the bionic dexterous hand is adjusted to finally obtain the control algorithm of the bionic dexterous hand.

[0058] In another embodiment of the present application, the construction of an efficient value network includes: the state quantity of the intelligent agent and the executed action enter the state encoder and the action encoder (two MLP networks) for encoding respectively, and obtain the state code and action code respectively. The state code directly enters the multi-head attention block to calculate the self-attention mechanism. The action code is transposed and multiplied with a specially designed mask to obtain the action code excluding the corresponding action. The result of the state code self-attention calculation is then combined with the action code excluding the corresponding action to calculate the multi-head attention, and finally the value estimate V(s) of the Critic network without the intelligent agent action input is obtained through calculation of the feedforward network. t+1 ) and the value estimate Q(s, a) of the corresponding action of the agent with sequence number i removed t -iThe efficiently designed critic network can obtain all the required calculation values ​​for the new estimated generalized advantage function in a single forward computation, which is more efficient than the general implementation method that requires sequentially calculating the value estimate of each agent's corresponding action.

[0059] According to one aspect of the present application, a dual-bionic dexterous hand collaborative control system based on a multi-agent system includes:

[0060] An acquisition module is used to collect environmental state data from the multi-agent system to generate current environmental state data;

[0061] A sequence determination module is used to determine the calculation sequence of each bionic dexterous hand agent based on the current environmental state data generated by the acquisition module;

[0062] Advantage calculation module, which is used to combine the current environment state data with the calculation sequence to calculate the corresponding advantage function value for each agent;

[0063] And the weight update module is used to use the advantage function values ​​generated by the advantage calculation module to adjust and update the network weight of the control network corresponding to each intelligent agent.

[0064] Case 1: Taking the ShadowHandCatchOver2Underarm task in the Isaac Gym simulation environment as an example, this task requires two ShadowHand bionic dexterous hands to cooperate and throw a stone from one hand to the other. The method provided in this application is used for control. Initialize the control network for each bionic dexterous hand (ie, each intelligent agent) and build an efficient value network. Set the discount factor γ = 0.99 and the GAE parameter λ = 0.95. In the training loop, the system interacts with the environment (75 steps), collects state data, records the actions performed by each simulated dexterous hand, determines the dynamic calculation sequence, and then calculates the precise advantage function value for each hand, and finally updates the control network weights of the two hands. The entire training process iterates about 3e 7 The final control weights of the bionic dexterous hand are obtained after 10 steps of interaction with the environment, and the resulting control network is used to complete the final task. Test results show that the bi-dexterous hand using this method can stably and efficiently complete the throwing and catching tasks, with a higher success rate than the traditional shared critic method, and the training time is shortened due to the use of an efficient value network.

[0065] The dual-bionic dexterous hand coordination control method provided in the embodiments of this application utilizes a causal graph attention network and topological sorting to dynamically determine a computational sequence that reflects the true causal dependencies between agents. This addresses the causal ambiguity in contribution allocation caused by fixed or arbitrary computational sequences in the prior art. By designing an efficient value network with a built-in masking mechanism, the value data required for all counterfactual evaluations can be generated in parallel in a single forward propagation. Combined with a specific generalized advantage estimation formula, this provides an accurate marginal contribution assessment for each agent. This approach improves computational efficiency while ensuring the accuracy of contribution allocation, resolving the technical pain point of the high computational costs of existing methods.

[0066] The present invention can be applied to industrial robots in flexible production lines. In flexible production line systems with a wider range of operating scenarios and more suitable for multiple types of tasks, the two-handed dexterous hand can enable the flexible production line to adapt to new tasks by replacing the tools used. The coordination of the two hands can also make the tasks that the production line can handle more extensive and complex. It can also be used in domestic robots. Domestic service robots are often designed to have the same working mode as humans to handle daily housework. Therefore, the bionic dexterous hand with coordinated control of both hands is the optimal implementation to achieve this goal. It can also be used in special operation robots, such as rescue, bomb disposal, maintenance and other scenarios, and can support the bionic dexterous hand with two hands to complete a wider range of tasks.

[0067] This invention addresses the causal ambiguity in contribution allocation by introducing a dynamic causal sorting mechanism. Specifically, existing technologies, due to their fixed or arbitrary computational order, fail to align the algorithm's evaluation logic with the physical causal logic of the task. This application, instead of adopting a static setup, utilizes a Causal Graph Attention Network (CGAT) to input real-time collected data on the current environmental state, including the postures of the bidexterous hands and the manipulated objects. Through network learning, it outputs a weighted adjacency matrix that quantifies the strength of the directed influence between agents in this specific state. This matrix itself dynamically models the current causal relationships. By applying a topological sorting algorithm to the directed graph represented by this matrix, a unique computational sequence is generated that matches the current task state and reflects the true order of influence transmission. This ensures that the computation order of the subsequent advantage function strictly adheres to the actual causal logic of the task in the physical world. For example, in a delivery task, the delivering hand is always evaluated before the receiving hand. This fundamentally eliminates the ambiguity in contribution allocation introduced by incorrect sorting, ensuring that rewards and penalties are transmitted along the correct causal chain, thereby improving the accuracy and rationality of the learning strategy. By combining a counterfactual advantage calculation method with an efficient parallel value network architecture, the contradiction between homogeneous advantage and exponential computational cost is resolved. At the algorithmic level, this approach addresses the homogeneous advantage issue by calculating the difference between the combined value and the baseline value after including and excluding the target agent's actions. This, combined with a specific generalized advantage estimation (GAE) formula, accurately quantifies the marginal contribution of each agent. This allows each hand to receive a highly differentiated and personalized advantage function evaluation, completely avoiding the issues of blurred rewards and penalties, or confusion between superiority and inferiority, caused by sharing a global evaluation signal. At the architectural level, to mitigate the exponential computational cost associated with precise evaluation methods, an efficient value network with a built-in masking mechanism is constructed. This allows for the parallel generation of all required counterfactual scenario encodings using a single matrix operation, and simultaneously outputs all values ​​in a single forward pass. This reduces the computational complexity of counterfactual evaluation from a linear dependence on the number of agents to a constant, making precise evaluation methods, previously impractical due to their computational complexity, efficient and feasible. This enables precise evaluation of each agent's contribution without sacrificing computational efficiency, thereby resolving the aforementioned technical contradictions.

[0068] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A collaborative control method for dual bionic dexterous hands based on a multi-agent system, characterized in that: include: Obtain environmental state data from the multi-agent system and generate current environmental state data; Based on this, a computation sequence is determined for multiple agents of the dual bionic dexterous hand; wherein the environmental state data includes the angles and angular velocities of all joints of each dexterous hand, the position and posture of the end effector, and the position, posture and velocity of the manipulated object; Based on the calculation sequence and the current environment state data, the advantage function value is calculated for each agent; Based on this, the network weights of the control network corresponding to the agent are updated; Determine the computational sequence for multiple agents of the dual bionic dexterous hand, including: Analyze the current environment state data, evaluate and generate dynamic influence relationships that characterize the interactions between various intelligent agents; Based on the dynamic influence relationship, a calculation sequence for calculating the advantage function value is constructed; Generate dynamic influence relationships, including: The current environment state data is input into the pre-configured causal graph attention network, and each bionic dexterous hand agent is regarded as a node in the graph structure; Utilize the attention mechanism of the causal graph attention network to learn and quantify the directional influence strength between nodes under the current environment state; The obtained directional influence intensity is output as a dynamic influence relationship; Calculate the advantage function value for each agent, including: For any target agent in the computation sequence, obtain the joint value of its own actions; Get the baseline value without the target agent's own actions; By comparing the joint value with the baseline value, the marginal contribution of the target agent is quantified and its corresponding advantage function value is generated.

2. The method according to claim 1, characterized in that The data structure of directed influence strength is a weighted adjacency matrix; In the weighted adjacency matrix, the row and column indices correspond to each bionic dexterous hand agent, and the matrix element value at the i-th row and j-th column is used to quantify the intensity of the directional influence exerted by agent i on agent j.

3. The method according to claim 1, characterized in that Construct a calculation sequence, specifically: Parse the dynamic influence relationship into a directed graph, where the nodes of the graph correspond to the agents and the directed edges are determined according to the dynamic influence relationship; Perform topological sorting on the directed graph to generate a computational sequence that can reflect the order of influence transmission between agents.

4. The method according to claim 1, wherein Generate advantage function values, including: Calculate the temporal difference term based on the environmental rewards, joint value, and baseline value obtained from the multi-agent system; calculating a marginal contribution term based on the joint value and the current state value corresponding to the current environmental state data; The temporal difference term and the marginal contribution term are weighted and summed in the time dimension to generate the advantage function value A corresponding to the target agent. t i =-ψ t +∑ m=0 ∞ (γλ) m (δ t+m +ψ t+m ), where ψ t is the marginal contribution term, m is the time step index, γ is the preset discount factor, λ is the smoothing parameter, δ t is the time difference term.

5. The method according to claim 4, characterized in that The joint value, baseline value, and current state value are generated by the same efficient value network; Through a forward propagation calculation of the efficient value network, the joint value, baseline value and current state value are generated simultaneously.

6. The method according to claim 5, characterized in that The internal structure of an efficient value network includes at least: State encoder, used to process current environment state data and generate state code; Action encoder, used to process the joint actions of each agent to generate action encoding; The multi-head attention module is used to receive state encoding and action encoding, and synchronously generate joint value, baseline value and current state value through attention operation.

7. The method according to claim 6, characterized in that After the multi-head attention module receives the state code and action code, the processing process includes: Performing a matrix operation on the action code and a preset mask matrix to generate a modified action code set in parallel; wherein each modified action code in the action code set corresponds to a scene in which a specific agent action is removed; The state code and the modified action code set are fed into the attention mechanism for operation to simultaneously generate the joint value, baseline value, and current state value.

Images