Dexterous two-hand cooperative control method based on multi-agent reinforcement learning

Abstract

The invention relates to a dexterous two-hand cooperative control method based on multi-agent reinforcement learning, and the method comprises the steps: initializing a neural network and an environment of dexterous two hands, and collecting the data during environment interaction; based on data during environment interaction, the joints perform sequential decision actions according to a greedy strategy; evaluating the decision action through a Q-value distribution network, and calculating a reward return at the same time; on the basis of reward return, entropy regularization is introduced, a strategy gradient optimization method is used, and a joint strategy is updated; and based on a preset condition, ending updating of the joint strategy, and outputting an optimal cooperative control strategy. The method is suitable for complex scenes such as grabbing, rotating and assembling, and has high cooperation efficiency and environmental adaptability.

Description

Technical Field

[0001] The present invention relates to the fields of reinforcement learning and control technology, and in particular to a dexterous two-handed collaborative control method based on multi-agent reinforcement learning. Background Art

[0002] Dexterous manipulators play an important role in robotics and are widely used in industrial manufacturing, medical assistance, and service robots. They use multi-degree-of-freedom joints and fingers to achieve complex grasping, handling, and precision manipulation tasks. However, in the control process of dexterous manipulators, due to the complexity of high-degree-of-freedom movements and the need for dynamic coordination between joints, high requirements are placed on control accuracy and collaboration efficiency, especially in two-handed collaborative operations, which require efficient motion planning and precise collaborative control.

[0003] Current control methods are mainly divided into model-based control methods and reinforcement learning-based control methods. Model-based control methods rely on the manipulator dynamics model and achieve task objectives through trajectory optimization and model predictive control (MPC). Such methods perform well in structured environments, but in complex and unstructured scenarios, the model is difficult to accurately describe the dynamic interaction between the manipulator and the environment, resulting in insufficient adaptability and generalization capabilities.

[0004] Control methods based on reinforcement learning can avoid dependence on precise models by learning optimal strategies through interaction with the environment, and show strong task adaptability and decision-making flexibility. In the operation of dexterous manipulators, single-agent reinforcement learning has been widely used in tasks such as one-handed grasping, rotation, and throwing. However, when applied to bimanual collaborative tasks, due to the high dimensionality of the action space and the complex dynamic interactions between agents, existing methods are difficult to effectively handle conflicts and non-stationarity between agents, and the collaborative efficiency and learning stability are low. Although the multi-agent reinforcement learning (MARL) method can model multi-agent systems, it still faces problems such as unreasonable reward distribution and slow strategy convergence in bimanual dexterous operations. Summary of the invention

[0005] The purpose of the present invention is to provide a collaborative control method for dexterous two-handed manipulator based on multi-agent reinforcement learning. By modeling each joint and finger of the dexterous manipulator as an independent agent, each agent makes decisions in sequence based on the advantage function decomposition lemma, and assumes that the subsequent agent takes the optimal action, thereby improving the decision-making performance of the current agent. At the same time, the present invention designs a Q-value distribution network to reduce the over-estimation problem of the traditional network for the agent and enhance the collaborative effect. Finally, by performing a sampling evaluation strategy on the Q-value distribution network, the strategy instability in the high-dimensional continuous action space is reduced, and the operation accuracy and collaborative efficiency of the dexterous manipulator in complex task scenarios are further improved.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A dexterous two-handed collaborative control method based on multi-agent reinforcement learning, comprising:

[0008] Initialize the neural network and environment of the dexterous hands and collect data when interacting with the environment;

[0009] Based on the data from the interaction with the environment, the joints make sequential decisions according to the greedy strategy;

[0010] Use the Q-value distribution network to evaluate the decision action and calculate the reward return;

[0011] Based on the reward return, entropy regularization is introduced, and the joint strategy is updated using the policy gradient optimization method;

[0012] Based on the preset conditions, the update of the joint strategy is terminated and the optimal collaborative control strategy is output.

[0013] Optionally, the neural network and environment for dexterous hands include: a dexterous hands policy network, an evaluation network and a simulation environment;

[0014] The strategy network includes: a strategy network for each joint, the input of which is environment information and its own state, and the output is joint action;

[0015] The evaluation network includes: a Q-value distribution network, the input of which is environmental information, its own state and joint action, and the output of which is the evaluation of the joint action;

[0016] The simulation environment includes: starting a preset simulator and a hypothetical scenario.

[0017] Optionally, the neural network and environment of the dexterous hands further include constraints; the constraints include:

[0018] Torque constraint: The torque applied by each joint should meet the limitations of hardware capabilities;

[0019] Motion rate constraint: The movement amplitude of the joint must not exceed the preset threshold in each time step to avoid system instability caused by high-frequency changes;

[0020] Coordination constraints of joint actions: The combination of actions of all joints must meet the coordination requirements of dynamic interaction between dexterous hands and objects.

[0021] Optionally, the data collected during the environment interaction includes:

[0022] In each time step, the system state and action of the dexterous hands collect data through interaction with the environment, including: the current state s t 、Current action a t、Instant Rewards t , next state s t+1 and task termination flag d t ;

[0023] The collected interaction data (s t ,a t ,r t ,s t+1 ,d t ) is stored in the experience pool D; when the preset size of the experience pool is exceeded, the first-in-first-out strategy is adopted to replace the old data.

[0024] Optionally, collecting data during environmental interaction also includes: performing importance sampling correction on the data stored in the experience pool; including:

[0025] For each sample (s, a) in the experience pool, calculate the importance weight Among them, π new (a|s) is the probability of the current strategy, π old (a|s) is the probability of the old strategy at the time of sampling;

[0026] Based on the importance weight, combined with the immediate reward r t and time difference error, setting priority δ for the sample i =r i +γQ(s i+1 ,a′)-Q(s i ,a i ), the sample priority is Among them, α is the priority adjustment parameter, which is used to balance the sampling probability of high-priority samples and low-priority samples;

[0027] According to the sample priority P i Sampling is performed, and samples that contribute to the strategy optimization preset are given priority for training.

[0028] Optionally, the joint makes sequential decision actions according to the greedy strategy including:

[0029] The joints are set up in a decision order according to the task requirements: the advantage value of each joint is obtained by using the advantage function decomposition theorem, and sequential decisions are made from large to small according to the advantage value. ;

[0030] The action a of each joint k k By local strategy π k (a k |o k ) determines; the input of the local strategy is the local observation o of the joint k and the action of the preceding joint a 1:k-1; Different joints of the two hands are set as different agents, and each agent has its own strategy, which is the local strategy;

[0031] The first joint a1 directly selects action a1=π1(o1) based on local observation o1; the second joint a2 selects action a2=π2(o2,a1) based on local observation o2 and the action a1 of the first joint; this logic is followed in sequence until the last joint a 40 , we get the joint action a={a1,a2,...,a 40}; Execute joint action a to push the system from the current state s t Transition to next state s t+1 .

[0032] Optionally, for the decision action, evaluating the action through a Q-value distribution network includes:

[0033] By constructing a Q-value distribution network, fitting the Q-value distribution according to the data generated by the interaction between the environment and the agent, and obtaining the rewards of the joints, i.e., local rewards, through sampling methods, this can reduce the overestimation problem of the traditional network for the agent and enhance the collaborative effect; among them, the global reward is used to evaluate the overall performance of the dexterous hands in the process of completing the task;

[0034] Normalize and smooth the local rewards.

[0035] Optionally, calculating the reward return includes:

[0036] Based on the local reward, a local reward function of each joint is calculated; wherein the local reward function is used to characterize the contribution of the action of the current joint to the long-term reward;

[0037] Accumulate the local reward functions of all joints to obtain the global reward function;

[0038] The local reward function is:

[0039]

[0040] Among them, r k (s,a) is the local reward of joint k, γ is the discount factor, Represents the expectation for the next state s′ and action a′;

[0041] The global reward function is:

[0042]

[0043] Among them, N is the total number of joints and k is the kth joint.

[0044] Optionally, the joint strategy π(a|s) is: the local strategy π of all joints k (a k |o k ) The optimization goal of the joint strategy is to maximize the long-term return of the system under the current strategy. Optimize policy parameters using policy gradient method in, To find the gradient of the policy parameter φ, J(π φ ) is the objective function, π φ (a|s) is the probability distribution of the strategy, and Q(s,a) is the action-value function.

[0045] Optionally, updating the joint strategy includes:

[0046] Dynamically adjust the temperature parameter α to keep the entropy value of the strategy close to the target entropy value H target : Update using gradient descent Combining policy gradient and entropy regularization, the policy parameter φ is updated according to the following formula:

[0047] The beneficial effects of the present invention are:

[0048] By modeling each joint of the dexterous manipulator as an independent agent, adopting a multi-agent reinforcement learning method, combining a sequential decision-making mechanism with a reward distribution strategy based on marginal contribution, the action collaboration between multiple joints is made more efficient. Compared with traditional model-based control methods and single-agent reinforcement learning methods, the present invention can dynamically adapt to complex action spaces and joint interaction requirements during task execution, effectively reduce action conflicts, and improve the operation accuracy and stability of both hands in complex tasks such as grasping, rotating and assembling.

[0049] Through the distributed reward function estimation and importance sampling technology, the problem of slow strategy convergence in high-dimensional action space is solved. At the same time, by introducing entropy regularization and smoothing mechanism, the exploration ability and convergence efficiency of the strategy are improved. Dynamic adjustment of the weight parameters of the reward function and the strategy optimization target enables the present invention to adapt to the needs of different task stages, showing strong task versatility and environmental adaptability, especially in unstructured complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the overall architecture of an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of a sequential decision architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] like Figure 1-Figure 2 As shown, a dexterous two-handed collaborative control method based on multi-agent reinforcement learning includes:

[0057] Initialize the neural network and environment of the dexterous hands and collect data when interacting with the environment;

[0058] Based on the data from the interaction with the environment, the joints make sequential decisions according to the greedy strategy;

[0059] Use the Q-value distribution network to evaluate the decision action and calculate the reward return;

[0060] Based on the reward return, entropy regularization is introduced, and the policy gradient optimization method is used to update the joint strategy;

[0061] Based on the preset conditions, the update of the joint strategy is terminated and the optimal collaborative control strategy is output.

[0062] Specifically, in this embodiment, the neural network and environment of the dexterous hands include: a strategy network, an evaluation network and a simulation environment of the dexterous hands;

[0063] The strategy network includes: a strategy network for each joint, with environmental information and its own state as input and joint action as output;

[0064] The evaluation network includes: a Q-value distribution network, the input is environmental information, its own state and joint action, and the output is the evaluation of the joint action;

[0065] The simulation environment includes: starting the pre-set simulator and scenario;

[0066] Initializing the neural network of the dexterous hands includes: initializing its weights by generating an orthogonal matrix, that is, assigning initial weights to each layer of the network by the orthogonal matrix method, thereby providing linearly independent parameter values ​​for each layer of the neural network.

[0067] Constraints are also set in the neural network and environment of the dexterous hands; the constraints include:

[0068] Torque constraint: The torque applied by each joint should meet the limitations of hardware capabilities;

[0069] Motion rate constraint: The movement amplitude of the joint must not exceed the preset threshold in each time step to avoid system instability caused by high-frequency changes;

[0070] Coordination constraints of joint actions: The combination of actions of all joints must meet the coordination requirements of dynamic interaction between dexterous hands and objects.

[0071] The agent observes the environment and obtains information: Each joint, as an independent agent, obtains its own relevant information through the local observation space, including joint angle, angular velocity, relative position information with objects, adjacent joint status and tactile feedback. Local observation only perceives its own relevant information, reduces global dependence, and builds a decentralized multi-agent system.

[0072] Collect data and perform importance sampling: When interacting with the environment, the system collects the current state, action, immediate reward, next state, and task termination flag, and stores them in the experience pool. Through importance sampling, the sampling weight is calculated so that high-priority samples are used first for optimization strategies.

[0073] like Figure 3 As shown in the figure, the actions are decided in the order of joints / fingers: the joints decide the actions step by step according to the greedy strategy order. Each joint selects an action based on the local observation and the action of the previous joint. The first joint makes a direct decision, and the subsequent joints make decisions based on the previous action. All joint actions are combined into a joint action to promote the state transfer of the system.

[0074] Distributed estimated reward function:

[0075] By constructing a Q-value distribution network, fitting the Q-value distribution according to the data generated by the interaction between the environment and the agent, and obtaining the rewards of the joints through sampling, the overestimation problem of the traditional network for the agent is reduced and the collaborative effect is enhanced. Finally, the rewards of each joint are accumulated to obtain the global reward, which provides a basis for strategy optimization.

[0076] Update strategy and optimize value function: The joint strategy is composed of local strategies of joints, which are optimized by policy gradient method to maximize the long-term return of the strategy. Entropy regularization is introduced to encourage exploration of diversified actions and avoid local optimality. The entropy coefficient and learning rate are dynamically adjusted to balance exploration and utilization and ensure the stability of strategy optimization.

[0077] Determine whether the task termination conditions are met: The task is terminated when one of the following conditions is met: the object position and posture reach the target range, the running time reaches the maximum number of steps, or the strategy change and reward value tend to be stable. If a hardware failure or abnormal action is detected, the task is terminated immediately to ensure safety, and the optimization result is output to end the loop.

[0078] Finally, output the results: After the task is completed, output the trained model and training curve.

[0079] Furthermore, the state space includes: joint states, object states, tactile feedback information, and joint action space;

[0080] The joint state includes: the angular position and angular velocity of each joint;

[0081] The object state includes: the position and posture of the object in three-dimensional space;

[0082] Tactile feedback information, expressed by the force and torque at the contact point;

[0083] The action space includes: the control variables of all joints; the control variables include the applied torque or angle adjustment of each joint;

[0084] Initializing the state space of the dexterous hands includes:

[0085] The angular positions and angular velocities of all joints are set as the initial values ​​of the task, the position and posture of the object in the three-dimensional space are initialized to given values ​​according to the task requirements, the tactile feedback information is initialized to zero, and constraints are introduced in the action space.

[0086] Specifically, in this embodiment, the state space of the dexterous hands includes joint states, object states, and tactile feedback information, as follows:

[0087] In the joint state, the state of each joint is composed of the following parameters: joint angle θ k ∈[-π,π] (radians), represents the current rotation position of the joint; joint angular velocity Indicates the rate of change of the joint angle. For the 20 controllable joints of the two hands (20 for each hand), the total dimension of the joint state is 40×2=80.

[0088] In the object state, the state of the manipulated object includes the following parts: object position x object ∈[-10,10]cm (three-dimensional Cartesian coordinate system); object posture q object , expressed as quaternion (q w ,q x ,q y ,q z ∈[-1,1]); object speed v object ∈[-5,5]cm / s, represents the linear velocity of the object; the angular velocity of the object ω object ∈[-5,5]rad / s, represents the rotation speed of the object. The total dimension of the object state is 3 (position) + 4 (quaternion attitude) + 3 (linear velocity) + 3 (angular velocity) = 13.

[0089] In the tactile feedback information, the tactile feedback information of each finger is described by the following parameters: contact force f tactile ∈[0,5] represents the force at the contact point; contact torque τ tactile ∈[0,5] represents the torque at the contact point. For 10 fingers on both hands, the total dimension of tactile feedback is 10×(3+1)=40.

[0090] In the total state space, the joint state, object state and tactile feedback information are integrated, and the total state space dimension of the dexterous hands is: total dimension = 80 (joint state) + 13 (object state) + 40 (tactile feedback information) = 133.

[0091] Since the dexterous hands adopt a multi-agent reinforcement learning framework, each joint can only obtain local state information related to itself. The local observation space includes:

[0092] The joint state itself includes the angle θ of the current joint k and angular velocity The dimension is 2.

[0093] Adjacent joint information includes the angles and angular velocities of adjacent joints, involving at most two front and rear joints; the dimension is 2×2=4.

[0094] The relative displacement x of the relative position information between the object and the joint rel,k =x object -x k , where x k is the position of the joint end; relative direction q rel,k =q object -q k Middlek It is the direction quaternion of the joint; the dimension is 3+4=7.

[0095] The contact force f at the end of the joint for tactile feedback information tactile,k and torque τ tactile,k ; The dimension is 3+1=4.

[0096] In the total dimension of the local observation space, for each joint, the total dimension of the local observation space is: local observation space dimension = 2 (own joint state) + 4 (adjacent joint information) + 7 (relative position information) + 4 (tactile feedback information) = 17.

[0097] In the single-joint action space, the action of each joint is a continuous value, representing the applied torque τ k ∈[-2,2]. In the joint action space, for the 40 joints of the two hands, the joint action space is:

[0098]

[0099] The torque τ applied by each joint k The change rate of joint movement Δτ must be within the hardware allowable range and ensure that the load capacity of the joint is not exceeded; k It should not exceed 0.5Nm in each time step to avoid high-frequency changes causing system instability; the combined actions of all joints need to meet the dynamic interaction coordination between the dexterous hands and the object to prevent conflicting actions.

[0100] At the beginning of the task, the neural network and the environment need to be initialized. The angles θ of all joints k and angular velocity Set to the initial value of the task, usually zero or a default value. The position x of the object object and posture q object Initialized to a given value according to the task requirements. The tactile feedback information is initialized to zero, indicating that the system has not yet made contact with the object. The relative position information and direction information are calculated by the difference between the position and direction of the joint end and the object.

[0101] The action space consists of the control variables of all joints, including the applied torque or angle adjustment of each joint. The action of a single joint is represented by a continuous variable τ k , whose value range is [τ min ,τ max ], where τ min and τ max are the minimum and maximum torques allowed by the joint, respectively. The range of values ​​is determined by the hardware performance and task requirements. For all joints of the dexterous hands, the joint action space is defined as:

[0102] A=A1×A2×…×A N

[0103] Among them, A k =[τ min ,τ max ] represents the action space of k joints, and N is the total number of joints of the dexterous hands.

[0104] In order to ensure the controllability of joint movements, the present invention introduces constraints in the action space, including the following:

[0105] Torque constraint: The torque τ applied by each joint k The hardware capability limit should be met to ensure that the action does not exceed the load-bearing capacity of the mechanical structure;

[0106] Motion rate constraint: the range of motion change of the joint |Δτ k |The preset threshold must not be exceeded in each time step to avoid system instability caused by high-frequency changes;

[0107] Coordination constraints of joint actions: The action combination of all joints must meet the coordination requirements of the dynamic interaction between the dexterous hands and objects to prevent conflicting actions.

[0108] During the action execution, each joint is based on its local observation space O m and the current policy π m (a m |O m )Select action a m , where a m represents the specific control input of the mth joint. The joint action is obtained by the joint action a={a1,a2,...,a N} acts on the system, pushing the system state from the current state s to the next state s'. The transfer of system state is determined by the following physical laws:

[0109] s′=T(s,a)

[0110] Among them, T(s,a) state transfer function describes the evolution process of the system under the joint action a, s and s' are the current state and the next state respectively. In the present invention, the state transfer function T(s,a) includes two parts: joint state update and object state update. The joint state update is based on the following formula:

[0111]

[0112] Among them, Δτ k is the joint action input, f(τ k ) is the function of torque on joint angular velocity. The state update of the controlled object is described by the following formula:

[0113] x object (t+1)=x object (t)+v object ·Δt

[0114] q object (t+1)=q object (t)+ω object ·Δt

[0115] Among them, v object and ω object They represent the linear velocity and angular velocity of the object, respectively, which are determined by the joint action a and tactile feedback.

[0116] In order to balance exploration and utilization and accelerate algorithm convergence, the present invention introduces random noise in the action space. The actual movement of each joint is:

[0117] a′ m =a m +∈ m

[0118] Where σ is the standard deviation of the noise, which controls the degree of exploration. The purpose of introducing randomness is to prevent the strategy from converging to the local optimal solution too early and to improve the exploration ability of the dexterous hands in the high-dimensional action space.

[0119] The definition and constraints of space provide a theoretical basis for the motion planning of dexterous hands, ensuring that each joint can meet the physical and task requirements of the system when selecting actions, while providing direct input for subsequent reward function design and strategy optimization.

[0120] The global reward function R(s,a) is used to evaluate the overall task completion of the dexterous hands, which includes the following three items:

[0121] Rewards dexterity in moving the object to the target position, defined as r pos =-α||x object -x goal || 2 , where α = 10, x goal is the target position. The reward is that the dexterous hands rotate the object to the target posture, defined as r rot =-β||q object -q goal || 2 , where β = 5, q goal is the target posture. Penalizing joints for applying excessive torque to encourage energy-saving operation is defined as Where γ = 0.1. The global reward function is the weighted sum of the above three parts:

[0122] R(s,a)=r pos +r rot +r energy

[0123] Distribute the global reward to each joint, and the marginal contribution of each joint is φ k Represents its increment to the global reward, defined as φ k =Q(s,a 1:k )-Q(s,a 1:k-1 ); the global reward is distributed to each joint according to the marginal contribution, and the local reward r k The calculation formula is

[0124] Furthermore, the data collected during the environment interaction includes:

[0125] In each time step, the system state and action of the dexterous hands collect data through interaction with the environment, including: the current state s t 、Current action a t 、Instant Rewards t , next state s t+1 and task termination flag d t ;

[0126] The collected interaction data (s t ,a t ,r t ,s t+1 ,d t ) is stored in the experience pool D; when the preset size of the experience pool is exceeded, the first-in-first-out strategy is adopted to replace the old data.

[0127] Specifically, in this embodiment, in each time step, the system state and action of the dexterous hands collect data by interacting with the environment, specifically including the following: the current state s t Contains joint status, object status, tactile feedback information; current action a t The action generated by the joint policy of the joint; the immediate reward r t According to the global reward function R(s t ,a t ) Calculate; next state s t+1 The system is performing action a t The state after the termination; whether the termination condition is reached t Indicates whether the task is completed or failed. t ,a t ,r t ,s t+1 ,d t) are stored in the experience pool D; the size of the experience pool is set to 100,000 data. When this limit is exceeded, the first-in-first-out strategy is used to replace the old data.

[0128] Since the strategy is constantly updated during the training process, there may be deviations in the quality of different samples and learning efficiency, so it is necessary to perform importance sampling correction on the data to ensure repeated learning of high-quality sample data.

[0129] For each sample (s, a) in the experience pool, calculate its importance weight Among them, π new (a|s) is the probability of the current strategy, π old (a|s) is the probability of the old strategy at the time of sampling. Combined with the immediate reward r t and time difference error, setting priority δ for the sample i =r i +γQ(s i+1 ,a′)-Q(s i ,a i ), the sample priority is Among them, α is the priority adjustment parameter, which is used to balance the sampling probability of high-priority samples and low-priority samples. i Sampling is performed, and samples that contribute more to strategy optimization are given priority for training. The size of each batch sampling in the experience pool is 64.

[0130] Furthermore, the joints step by step select decision actions in order including:

[0131] The joints of the dexterous hands are set in the decision order of the greedy strategy according to the task requirements: the advantage value of each joint is obtained by using the advantage function decomposition theorem, and sequential decisions are made from large to small according to the advantage value.

[0132] The action a of each joint k k By its local strategy π k (a k |o k ) determines; the input of the local strategy is the local observation o of the joint k and the action of the preceding joint a 1:k-1 The joint strategy is the combination of the local strategies of all joints.

[0133] The first joint a1 directly selects action a1=π1(o1) based on its local observation o1; the second joint a2 selects action a2=π2(o2,a1) based on local observation o2 and the action a1 of the first joint; and this logic is followed in sequence until the last joint a 40 , we get the joint action a={a1,a2,...,a 40}. Execute joint action a to push the system from the current state s t Transition to next state s t+1 .

[0134] Furthermore, the decision-making actions are evaluated and rewarded, including:

[0135] By constructing a Q-value distribution network, the Q-value distribution is fitted according to the data generated by the interaction between the environment and the intelligent agent, the rewards of the joints are obtained through sampling, and the rewards are normalized and smoothed.

[0136] Furthermore, in this embodiment, in order to improve the cooperation ability of each joint, a Q value network is constructed, and the specific steps are as follows:

[0137] According to the global reward function R(s t ,a t ) calculates the reward for the current time step.

[0138] In order to learn the return distribution, we need to expand the Bellman equation to a distribution form. The traditional Bellman equation describes the recursive relationship of Q(s,a):

[0139]

[0140] Where R(s,a) is the immediate reward, s' is the next state after executing action a, and a' is the action selected in state s'. In distributed Q-learning, the Bellman equation for the reward distribution is expressed as:

[0141] Z(s,a)=R(s,a)+γZ(s′,a′)

[0142] in represents distribution equality. In other words, the reward distribution of the current state-action pair is equal to the discounted sum of the immediate reward and the future reward distribution. This form reflects the recursive nature of the reward distribution.

[0143] In order to specify the above formula, this embodiment defines a distributed Bellman operator J π :

[0144]

[0145] Among them J π Combine the immediate reward and future reward distribution to update the current reward distribution.

[0146] In distributed Q-learning, this example minimizes the predicted distribution Z θ (s,a) and the target distribution Z targetThe distance between (s,a) is used to train the neural network. The target distribution Z target (s,a) is defined as:

[0147]

[0148] Where y is the target value, calculated based on the distributed Bellman equation: y = r + γQ θ (s′,a′), where Q θ (s′, a′) is the expected value of the next state-action pair; σ target is the standard deviation of the target distribution, usually calculated from the uncertainty of Z(s′,a′). The target distribution approximates the true reward distribution of the current state-action pair.

[0149] In order to optimize the neural network, this embodiment needs to define a loss function to measure the predicted distribution Z θ (s,a) and the target distribution Z target (s, a). KL divergence is usually used as a metric:

[0150]

[0151] Assume Z θ (s,a) and Z target (s, a) are both Gaussian distributions, and their probability density functions are: Where Q θ (s,a) is the mean of the predicted distribution; σ ψ (s,a) is the standard deviation of the predicted distribution; y is the mean of the target distribution; σ target is the standard deviation of the target distribution.

[0152] According to the Q value network, the reward of the joint is estimated, and the local reward r k The calculation formula is To prevent the reward value from being too large or too small, normalization is performed to limit the reward value to the range of [-1,1]: Where ∈ is a small value (such as 10 -6 ) to prevent the denominator from being zero. To avoid drastic fluctuations in the reward distribution process, a smoothing mechanism is adopted: Among them, α∈[0,1] is the smoothing coefficient, is the reward value of the previous time step.

[0153] Furthermore, the calculation of reward return includes:

[0154] Based on the local reward, the local reward function of each joint is calculated; the local reward function is used to characterize the contribution of the action of the current joint to the long-term reward;

[0155] Accumulate the local reward functions of all joints to obtain the global reward function.

[0156] Specifically, in this embodiment, in the multi-agent system, the local reward function z of each joint is k (s,a) is used to describe the contribution of the current joint action to the long-term reward, and is specifically defined as follows:

[0157] The local reward function z for each joint k k (s,a) consists of immediate reward and future return, which is recursively calculated as:

[0158]

[0159] Among them, r k (s, a) is the local reward of joint k; γ is the discount factor (set to γ ​​= 0.99); Represents the expectation for the next state s′ and action a′.

[0160] The global reward function Z(s,a) is generated by accumulating the local reward functions of all joints:

[0161]

[0162] Wherein, N is the total number of joints (N=40).

[0163] Furthermore, the joint strategy π(a|s) is: the local strategy π of all joints k (a k |o k ) The optimization goal of the joint strategy is to maximize the long-term return of the system under the current strategy. Optimize policy parameters using policy gradient method

[0164] In order to balance exploration and exploitation, an entropy term H(π) is added to the optimization objective to encourage the randomness of the strategy: Where α is a temperature parameter that controls the trade-off between exploration and exploitation (initial setting is α = 0.2). The entropy of the strategy is defined as H(π φ (·|s))=-∫π φ (a|s)logπ φ (a|s)da; Larger entropy values ​​αH(π) encourage more extensive exploration, while smaller entropy values ​​tend to focus on high-reward actions.

[0165] The temperature parameter α is an optimizable parameter used to control the weight of the entropy term. When the optimization starts, more states need to be explored, and α is generally higher. In the later stages of the optimization, rewards are generally the main focus, and the proportion of entropy terms will decrease, which is similar to the annealing algorithm, so it is called the temperature parameter.

[0166] To improve the stability of training, the learning rate η of the policy network π Gradually decreases with training time in, is the initial learning rate, β is the learning rate decay parameter, and t is the number of training steps.

[0167] Further, updating the joint strategy includes:

[0168] Dynamically adjust the temperature parameter α to keep the entropy value of the strategy close to the target entropy value H target : Update using gradient descent Combining policy gradient and entropy regularization, the policy parameter φ is updated according to the following formula:

[0169] The final optimized joint strategy model and value function model are output to describe the action planning and control strategy of dexterous hands in complex task scenarios. The output model includes the following:

[0170] The joint strategy is composed of the local strategies of each joint and is in the form of Outputs the global value function V(s) and the local reward function z of each joint i The change curve of (s,a). Output the change curve of the reward function during training.

[0171] This embodiment proposes a modeling and calculation method for a distributed reward function, which decomposes the global reward into the local reward of each joint, estimates it recursively, and combines the upper and lower limit rewards of a dual-objective reward estimation mechanism, thereby reducing the policy instability in high-dimensional continuous action space, thereby significantly improving the task execution efficiency and the stability of policy optimization.

[0172] By sampling data with good quality based on importance priority, data that contributes more to strategy optimization are given priority for training, which significantly improves sample utilization efficiency, accelerates strategy convergence, and solves the problem of slow strategy optimization in high-dimensional action space.

[0173] In the process of policy optimization, entropy regularization is introduced to balance exploration and utilization, avoid the policy from falling into the local optimum, and dynamically adjust the reward weight parameters and entropy coefficients to adapt to the needs of different stages of the task. For example, in the grasping stage, the weight of the position error is emphasized, while in the rotation stage, the weight of the posture error is more concerned to achieve more precise task control.

[0174] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A dexterous two-handed collaborative control method based on multi-agent reinforcement learning, characterized in that: include: Initialize the neural network and environment of the dexterous hands and collect data when interacting with the environment; Based on the data from the environment interaction, the joints make sequential decisions according to the greedy strategy; Use the Q-value distribution network to evaluate the decision action and calculate the reward return; Based on the reward return, entropy regularization is introduced, and the joint strategy is updated using the policy gradient optimization method; Based on the preset conditions, the update of the joint strategy is terminated and the optimal collaborative control strategy is output.

2. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The neural network and environment of dexterous hands include: dexterous hands strategy network, evaluation network and simulation environment; The strategy network includes: a strategy network for each joint, the input of which is environment information and its own state, and the output is joint action; The evaluation network includes: a Q-value distribution network, the input of which is environmental information, its own state and joint action, and the output of which is the evaluation of the joint action; The simulation environment includes: starting a preset simulator and a scenario; Initializing the neural network of the dexterous hands includes: initializing its weights by generating an orthogonal matrix, that is, assigning initial weights to each layer of the network by the orthogonal matrix method, thereby providing linearly independent parameter values ​​for each layer of the neural network.

3. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The neural network and environment of the dexterous hands also include constraints; the constraints include: Torque constraint: The torque applied by each joint should meet the limitations of hardware capabilities; Motion rate constraint: The movement amplitude of the joint must not exceed the preset threshold in each time step to avoid system instability caused by high-frequency changes; Coordination constraints of joint actions: The combination of actions of all joints must meet the coordination requirements of dynamic interaction between dexterous hands and objects.

4. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The data collected during the environment interaction includes: In each time step, the system state and action of the dexterous hands collect data through interaction with the environment, including: the current state s t 、Current action a t 、Instant Rewards t , next state s t+1 and task termination flag d t ; The collected interaction data (s t ,a t ,r t ,s t+1 ,d t ) is stored in the experience pool D; when the preset size of the experience pool is exceeded, the first-in-first-out strategy is adopted to replace the old data.

5. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 4 is characterized in that: The data collected during the environment interaction also includes: performing importance sampling correction on the data stored in the experience pool; including: For each sample (s, a) in the experience pool, calculate the importance weight Among them, π new (a|s) is the probability of the current strategy, π old (a|s) is the probability of the old strategy at the time of sampling; Based on the importance weight, combined with the immediate reward r t and time difference error, setting priority δ for the sample i =r i +γQ(s i+1 ,a′)-Q(s i ,a i ), the sample priority is Among them, α is the priority adjustment parameter, which is used to balance the sampling probability of high-priority samples and low-priority samples; According to the sample priority P i Sampling is performed, and samples that contribute to the strategy optimization preset are given priority for training.

6. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The joints make sequential decision actions according to the greedy strategy, including: The joints are set up in a decision order according to the task requirements: the advantage value of each joint is obtained by using the advantage function decomposition theorem, and sequential decisions are made from large to small according to the advantage value; The action a of each joint k k By local strategy π k (a k |o k ) determines; the input of the local strategy is the local observation o of the joint k and the action of the preceding joint a 1:k-1 ; Different joints of the two hands are set as different agents, and each agent has its own strategy, which is the local strategy; The first joint a1 directly selects action a1=π1(o1) based on local observation o1; the second joint a2 selects action a2=π2(o2,a1) based on local observation o2 and the action a1 of the first joint; this logic is followed in sequence until the last joint a 40 , we get the joint action a={a1,a2,...,a 40 }; Execute joint action a to push the system from the current state s t Transition to next state s t+1 .

7. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: For the decision action, the Q-value distribution network is used to evaluate the action including: By constructing a Q-value distribution network, fitting the Q-value distribution according to the data generated by the interaction between the environment and the agent, and obtaining the rewards of the joints, i.e., local rewards, through sampling methods, this can reduce the overestimation problem of the traditional network for the agent and enhance the collaborative effect; among them, the global reward is used to evaluate the overall performance of the dexterous hands in the process of completing the task; Normalize and smooth the local rewards.

8. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 7 is characterized in that: Calculating the reward return includes: Based on the local reward, a local reward function of each joint is calculated; wherein the local reward function is used to characterize the contribution of the action of the current joint to the long-term reward; Accumulate the local reward functions of all joints to obtain the global reward function; The local reward function is: Among them, r k (s,a) is the local reward of joint k, γ is the discount factor, Represents the expectation for the next state s′ and action a′; The global reward function is: Among them, N is the total number of joints and k is the kth joint.

9. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The joint strategy π(a|s) is: the local strategy π of all joints k (a k |o k ) The optimization goal of the joint strategy is to maximize the long-term return of the system under the current strategy. Optimize policy parameters using policy gradient method in, For the strategy parameters φ Find the gradient, J(π φ ) is the objective function, To find the expectation, π φ (a|s) is the probability distribution of the strategy, logπ φ (a|s) is the logarithmic probability of the policy, and Q(s,a) is the action-value function.

10. The dexterous two-handed collaborative control method based on multi-agent reinforcement learning according to claim 9 is characterized in that: Updating the joint strategy includes: Dynamically adjust the temperature parameter α to keep the entropy value of the strategy close to the target entropy value H target : Update using gradient descent Combining policy gradient and entropy regularization, the policy parameter φ is updated according to the following formula: Among them, the temperature parameter is used to explore and balance the control strategy.

Images