Intelligent grasping control method and system based on multi-agent reinforcement learning

Abstract

The application provides a kind of intelligent grasping control method and system based on multi-agent reinforcement learning, each finger of anthropomorphic dexterous hand is regarded as independent agent, control action is generated based on respective strategy network, and multi-agent system is formed;Experience data generated in the interaction process of each agent in multi-agent system and environment is stored by using replay pool mechanism, the sampling priority of each agent replay pool is dynamically adjusted according to its task target, learning progress and task importance, and experience data is shared between replay pools;Phase reward mechanism is adopted, individual reward and global reward are used to guide each agent in multi-agent system to optimize position, joint angle and contact force in different stages of grasping task respectively;Multi-agent deep deterministic policy gradient algorithm is used for training, and the trained multi-agent system is used for intelligent grasping control.The application significantly improves the accuracy and stability of the grasping process.

Description

Technical Field

[0001] This invention belongs to the field of intelligent grasping control, specifically relating to an intelligent grasping control method and system based on multi-agent reinforcement learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Robotics technology is advancing rapidly, and its applications have expanded from traditional industrial production to everyday life. Humanoid dexterous hands, as robotic hands capable of mimicking human hand movements and possessing multiple degrees of freedom and high-precision control, offer greater flexibility and adaptability compared to traditional industrial robot grasping tools. They can handle objects with complex shapes and uneven weights, thus holding significant potential in various application scenarios. Humanoid dexterous hands have a five-finger structure similar to human hands, and their grasping control not only enhances robots' operational capabilities in complex environments but also significantly improves their adaptability and efficiency in practical applications. They have broad application prospects in industrial production, healthcare, service robots, and logistics.

[0004] However, in practical applications, the design of gripping controllers for humanoid dexterous hands remains challenging, especially in uncertain environments. When the shape, size, weight, etc. of the object being grasped are unknown, the grasping accuracy (such as grasping success rate, grasping posture stability, etc.) and precision (grasping force accuracy, running trajectory accuracy, etc.) are significantly insufficient, which seriously affects the grasping efficiency and widespread application of multi-fingered humanoid dexterous hands.

[0005] Traditional humanoid dexterous hand grasping control methods are mostly based on rules and experience, relying on manual parameter adjustment, and are generally unable to effectively cope with complex unstructured environments. These methods often struggle to adapt to the object's physical properties (such as material, shape, surface smoothness, etc.), initial state (such as the object's placement posture, initial velocity, etc.), motion state (such as velocity, acceleration, angular acceleration, etc.), and task objectives (such as stable grasping, rapid lifting, or moving the object). Therefore, these control methods often suffer from poor flexibility, stability, and adaptability, especially when facing unknown or dynamic environments, easily leading to grasping failures.

[0006] In recent years, with the rapid development of artificial intelligence technologies such as deep learning and reinforcement learning, data-driven dexterous hand grasping control methods have gradually attracted attention. These methods can make dexterous hands adaptively adjust strategies during grasping through large amounts of data training, thereby improving grasping accuracy and precision. Especially in the application of reinforcement learning, dexterous hands can gradually optimize grasping strategies through interaction with the environment to cope with complex grasping tasks. However, these methods still have some problems, especially in the training process, which requires a large amount of data, accurate labeling, high training time, and poor learning generalization ability, online control performance, control precision, stability, and adaptability. These shortcomings limit the universality and efficiency of reinforcement learning in practical applications. On the other hand, traditional methods usually face the problem of requiring a large amount of data and high training time, while lacking sufficient learning generalization ability and online control performance, which is particularly evident in complex environmental grasping tasks. SUMMARY

[0007] To solve the above problems, the present application proposes an intelligent grasping control method and system based on multi-agent reinforcement learning. By regarding each finger as an independent agent, the present application utilizes the cooperation and information sharing of the multi-agent system to enable each finger to adaptively adjust the grasping strategy according to real-time feedback, handle different physical properties of objects, and cope with dynamic changes in grasping tasks, thereby significantly improving the accuracy and stability of the grasping process. This method avoids object sliding or instability during object control and grasping, reduces the risk of grasping failure, and is particularly suitable for uncertain environmental object grasping.

[0008] According to some embodiments, the present application adopts the following technical solutions:

[0009] An intelligent grasping control method based on multi-agent reinforcement learning, comprising the following steps:

[0010] Each finger of the dexterous hand is regarded as an independent agent, the local state information of each finger is obtained from the global state, and the control action is generated based on the respective policy network to form a multi-agent system;

[0011] Experience data generated during the interaction between each agent in the multi-agent system and the environment is stored using a replay pool mechanism, the sampling priority of each agent's replay pool is dynamically adjusted according to its task objective, learning progress, and task importance, and experience data is shared between replay pools;

[0012] A phased reward mechanism is adopted, individual rewards and global rewards are used to guide each agent in the multi-agent system to optimize position, joint angle, and contact force in different stages of the grasping task;

[0013] The multi-agent deep deterministic policy gradient algorithm is used for training, and the trained multi-agent system is used for intelligent grabbing control.

[0014] As an alternative embodiment, the local state information includes joint angles, finger positions, contact forces, and characteristics of target objects, which are used to guide the grasping actions of the corresponding fingers.

[0015] As an alternative embodiment, the replay pool mechanism is used to store experience data generated during the interaction of each agent in the multi-agent system with the environment, and the sampling priority of each agent's replay pool is dynamically adjusted according to its task objective, learning progress, and task importance. The process includes: each agent's replay pool preferentially selects relevant experience data according to its task objective, and the task objective priority is determined according to the ratio of the experience score related to the current task objective to all task objective-related experience scores.

[0016] The replay pool preferentially selects experience data with a temporal difference error greater than a set value in the early stage, and preferentially samples experience data with diversity in the later stage.

[0017] In a multi-task learning environment, the replay pool preferentially samples experience data of more important tasks.

[0018] Further, the dynamic adjustment process includes: in the early stage of training, each agent relies more on experience with a temporal difference error greater than a set value; in the later stage of training, the proportion of diversity sampling is gradually increased, and the dynamic sampling weight is:

[0019]

[0020] wherein, is the quality score of experience , which is calculated from the temporal difference error and experience diversity, represents the relevance of experience to the current task objective, is the total number of samples in the experience pool.

[0021] As an alternative embodiment, the process of sharing experience data between replay pools includes: when some experience is lacking in the replay pool of some agents, relevant experience data is obtained from the replay pool of other agents through an experience sharing mechanism. The experience sharing mechanism allows each agent to accelerate learning by sharing data in the shared experience pool without direct interaction.

[0022] As an alternative implementation, a stage reward mechanism is adopted to guide the agents in the multi-agent system to optimize the position, joint angle and contact force in different stages of the grasping task respectively, including: the stage reward mechanism includes two stages, the first stage is the joint position reward and finger position reward stage, and the second stage is the contact force reward stage, during the grasping process, the agent automatically switches the reward calculation mode according to the task target of the current stage, and each stage is guided by individual reward and global reward.

[0023] As a further defined implementation, the individual reward is determined by comprehensively considering the joint angle, finger position and contact force to ensure that each finger can perform the optimal behavior in the grasping task, specifically:

[0024]

[0025] wherein, is the current position of the i-th finger, is the target position, is the joint angle of the i-th finger, is the initial angle of the joint, is the contact force of the i-th finger, is the desired contact force, is the corresponding weight coefficient for controlling the influence of different factors. As a further defined implementation, the global reward is the sum of the individual rewards of all agents, representing the task completion of the entire multi-agent system, and the global reward is used to reflect the overall effect of all fingers in completing the task together, and the global reward is:

[0026]

[0027] ;

[0028] wherein, is the individual reward of the i-th finger, indicates the number of fingers in the system.

[0029] As a further defined implementation, in the first stage, when the finger is not in contact with the object, the reward is determined by the difference between the finger position and the joint angle, the closer the finger position is to the target position, the greater the position reward; the greater the joint angle, the greater the joint reward; in the second stage, when the finger is in contact with the object, the reward is only determined by the contact force, the closer the contact force is to the desired contact force, the greater the reward.

[0030] ​​​​As a further defined embodiment, the reward stage is determined according to whether the contact force of the finger exceeds a set threshold value, if the contact force of the finger is less than or equal to the set threshold value, it is in the first stage, otherwise it is in the second stage.

[0031] As a further defined embodiment, the overall reward considers the task completion of each finger and the grasping effect of the whole system, and adjusts the priority of different task objectives through a weight coefficient, specifically:

[0032] ;

[0033] Wherein, is the individual reward of the i-th finger at time step t, is the weight coefficient, which controls the contribution proportion of the individual reward and the global reward in the overall reward, at the beginning of training, the individual reward occupies a larger proportion, and with the progress of training, the weight of the global reward is gradually increased.

[0034] As an optional embodiment, the process of training using the multi-agent deep deterministic policy gradient algorithm includes:

[0035] Set the number of agents, initialize the environment state, reward function and experience replay pool;

[0036] At each time step, each agent selects an action according to the local state through the policy network, the actions of all agents are combined into a joint action, the joint action is executed, the new state and reward are obtained, and the current experience is stored in the experience replay pool of each agent;

[0037] Each agent samples based on the importance and diversity of experience, preferentially samples the experience most helpful to the current task, optimizes the policy network through the policy gradient algorithm, updates the value network through the temporal difference error, and evaluates the value of the current action;

[0038] Determine the current stage, if the finger contacts the object, switch to the second stage, the reward is calculated only according to the contact force, otherwise, it is in the first stage, the reward is calculated according to the finger position and joint angle;

[0039] Determine whether to switch stages according to the contact force, if the contact force exceeds the threshold value, enter the second stage;

[0040] At the end of each round of training, output the total reward, and determine whether the convergence condition is reached, if so, stop training.

[0041] As a further defined embodiment, the process of optimizing the policy network through the policy gradient algorithm and updating the value network through the temporal difference error includes:

[0042] ​The policy network update process includes:

[0043] ;

[0044] in, It is the first Policy network parameters of each agent It is the first The policy of an agent, given a state Choose action The probability, It is a value network function, representing the agent's state. Take action below Expected returns This represents the maximum expected reward for the i-th agent. Indicating in strategy The following expectations;

[0045] Value network updates are performed using temporal difference errors:

[0046] ;

[0047] in, It is a discount factor, indicating the importance of future rewards. r i Indicates the first i Each time step represents the immediate reward the agent receives from the environment.

[0048] An intelligent grasping control system based on multi-agent reinforcement learning includes:

[0049] The multi-agent system building module is configured to treat each finger of the humanoid dexterous hand as an independent agent, obtain its own local state information from the global state, and generate control actions based on its respective policy network to form a multi-agent system.

[0050] The replay pool establishment module is configured to use the replay pool mechanism to store the experience data generated by each agent in the multi-agent system during the interaction with the environment. The sampling priority of each agent's replay pool is dynamically adjusted according to its task objective, learning progress and task importance. Experience data is shared between replay pools.

[0051] The reward mechanism optimization module is configured to adopt a phased reward mechanism, which uses individual rewards and global rewards to guide each agent in the multi-agent system to optimize position, joint angle and contact force at different stages of the grasping task.

[0052] The intelligent grasping control module is configured to be trained using a multi-agent deep deterministic policy gradient algorithm, and to perform intelligent grasping control using the trained multi-agent system.

[0053] The application relates to a humanoid dexterous hand, and the method or the system is used.

[0054] Compared with the prior art, the application has the beneficial effects that:

[0055] The application regards each finger as an independent intelligent agent, utilizes cooperation and information sharing of a multi-agent system, enables each finger to adaptively adjust a grasping strategy according to real-time feedback, processes different physical characteristics of an object and responds to a dynamically changing grasping task, can realize a grasping behavior with humanization characteristics, has autonomous learning and dynamic adaptation capability, significantly improves accuracy and stability of a grasping process, avoids object sliding or instability in object control and grasping, and reduces the risk of grasping failure.

[0056] The application optimizes task allocation and enhances robustness and fault tolerance of the system through cooperative work of the multi-agent system, other intelligent agents can timely adjust actions when a single finger fails, and the grasping task is ensured to be completed, the mechanism significantly improves adaptability and fault tolerance of the system, and can still maintain a high grasping success rate in a dynamically changing environment.

[0057] The application adopts a parallel learning mode, greatly speeds up optimization of the grasping strategy, significantly improves learning efficiency and system adaptability, breaks through the limitation of traditional methods in grasping control in an uncertain environment by adapting to a dynamic state of an object in real time, provides a new idea for grasping control of the five-finger humanoid dexterous hand, has strong practical application potential and technical value.

[0058] The application can effectively manage experience data of each intelligent agent by introducing an adaptive experience replay pool and a personalized replay pool mechanism, dynamically adjusts an experience sampling strategy according to a task target, learning progress and task importance, thereby speeds up a learning process and improves stability and efficiency of training; through joint action generation and intelligent agent cooperation, multiple fingers (intelligent agents) can cooperatively complete a grasping task, and the personalized experience replay pool is used to promote experience sharing between intelligent agents, and cooperation efficiency between the multiple intelligent agents is improved.

[0059] The application can avoid overfitting of the intelligent agent by introducing a diversity sampling replay pool mechanism, enhances generalization capability of the system, and enables the intelligent agent to maintain good adaptability and stability in a more complex and dynamic environment.

[0060] The application adopts a phased reward mechanism, respectively optimizes finger positions, joint angles and contact forces in different stages of the grasping task, enables the intelligent agent to adaptively adjust a strategy under different task targets, thereby improves grasping precision and stability, and finally realizes a higher grasping success rate.

[0061] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0062] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, the schematic embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application.

[0063] Figure 1 A flowchart of a method for an embodiment;

[0064] Figure 2 A schematic diagram of an individualized playback pool mechanism of an agent for an embodiment;

[0065] Figure 3 A schematic diagram of a hardware system integration for an embodiment, wherein 1-5 respectively represent the thumb, index finger, middle finger, ring finger, and little finger (i.e., five agents), 6 represents a mechanical arm, 7 represents a humanoid dexterous hand body, 8 represents a robot upper planning controller, and 9 represents 23 irregular objects to be grasped.

[0066] Figure 4 A flowchart of a humanoid dexterous hand application step for an embodiment. DETAILED DESCRIPTION

[0067] The present application will be further described below in conjunction with the drawings and embodiments.

[0068] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0069] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0070] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0071] Embodiment one

[0072] A smart grasping control method based on multi-agent reinforcement learning, the present embodiment takes a five-finger humanoid dexterous hand as an example for illustration, which includes the following key points:

[0073] The five fingers are regarded as independent agents, which obtain their local state information from the global state and generate control actions based on their respective policy networks. These actions are integrated through a joint coordination module to form the overall grasping operation on the target object.

[0074] The agent individualized replay pool mechanism is used to store the state, action, reward, and next state information generated during the interaction of the agent with the environment. This mechanism can dynamically adjust the storage and sampling strategies of the replay pool according to the learning progress, task objectives, and importance of samples for each agent, thereby improving the efficiency of experience utilization, accelerating the training process, and improving the stability of the model.

[0075] In a multi-process parallel training environment, the strategy is based on centralized training and distributed execution, and the multi-agent reinforcement learning optimizes the coordination strategy of the five fingers through a global reward function, enabling them to adapt flexibly to the shape, position, and dynamic changes of the object in a complex environment.

[0076] The following will be described in detail:

[0077] Step 1: Constructing a five-fingered dexterous hand multi-agent system

[0078] Each finger of the five-fingered dexterous hand is regarded as an independent agent (thumb, index finger, middle finger, ring finger, and little finger), and each agent obtains its local state information from the global state in the environment. These local state information includes joint angles, finger positions, contact forces, and characteristics of the target object, etc., which are used to guide the grasping actions of the fingers.

[0079] Step 2: Agent individualized replay pool mechanism

[0080] In reinforcement learning, the experience replay pool is an important component, and reasonable storage of experience can accelerate the learning process and improve the stability of the model. In the dexterous hand multi-agent system, multiple fingers interact with the environment, generating a large amount of experience data, which will be stored in the experience replay pool and provide feedback during the training process. Each experience record is defined as:

[0081]

[0082] where, represents the stored experience at time step t , represents the global state of the agent at time step t , represents the action selected and executed at state , represents the reward obtained after executing action , represents the agent performing The environment then transitions to the next global state. These data are stored in the replay pool, and as training progresses, new experiences will gradually replace old ones. The key to an adaptive experience replay pool is to evaluate the importance and diversity of each experience and adjust the storage and sampling strategies accordingly to ensure the efficiency and stability of training.

[0083] In a humanoid dexterous hand system or a multi-task learning environment, the task goals, learning progress, and task priorities of each finger can be different, so each finger needs a personalized replay pool that can dynamically adjust the sampling strategy according to its current task requirements and learning status. To achieve this, as shown in Figure 2 the embodiment proposes a multi-agent individualized replay pool mechanism, the core goal of which is to ensure that each agent prioritizes the selection of experiences that are most helpful for its learning from the replay pool in an adaptive manner.

[0084] ① Personalized replay pool design

[0085] The replay pool of each agent is dynamically adjusted according to factors such as its task goal, learning progress, and task importance. Specifically, the sampling strategy of the replay pool will prioritize the selection of experience data that is most relevant to the current task goal and learning stage of the finger.

[0086] a. Task goal relevance: Each finger's replay pool will prioritize relevant experiences based on its task goal. For example, if a finger's task is grasp stability, the replay pool will prioritize storing experience data related to grasping object stability. In this way, the experiences in the replay pool can always be highly relevant to the task goal.

[0087] Task goal priority adjustment formula:

[0088]

[0089] wherein, is the experience score related to the current task goal, is the priority sampling probability of the experience

[0090] ​b. The impact of finger learning progress: The learning progress of a finger directly affects the update strategy of its experience pool. In the early stage of learning, the finger usually needs more importance-weighted experience to quickly correct the strategy, so the replay pool will prefer to select those experience data with larger (Temporal Difference Error, TD) error. In the later stage of training, the strategy of the finger has stabilized, and at this time the replay pool samples more diverse experiences to enhance the adaptability of the finger in different tasks and environments. The learning progress formula is defined as follows:

[0091]

[0092] wherein, and are weight coefficients, respectively controlling the influence of TD error and task relevance in the learning process, TD error, is the score of the current experience and task target relevance.

[0093] c. Task importance and replay pool: In a multi-task learning environment, the sampling priority of the replay pool will be dynamically adjusted according to the importance and difficulty of the task. For example, for more important tasks, the replay pool will prefer to sample the experience of that task, while for less important tasks, the sampling frequency of their experience will be reduced. This ensures that in a multi-task learning environment, the finger can prioritize learning the experience of important tasks to improve overall performance. Its definition is as follows:

[0094]

[0095] wherein, is the priority of the task, is the difficulty of the task, is the learning progress of the finger on the task.

[0096] ② Experience sharing between tasks

[0097] In a multi-agent system, multiple agents usually cooperate to complete common tasks, so their experience pools can share with each other. In the individualized replay pool of the finger, when some fingers lack some key experiences in their replay pool, they can obtain relevant experience data from the replay pool of other fingers through the experience sharing mechanism. Through experience sharing, the finger can more quickly supplement and perfect its knowledge base, promoting the collaboration and learning efficiency of the entire system. The experience sharing mechanism allows fingers to accelerate learning by sharing data in the experience pool without direct interaction. Experience sharing between fingers not only helps to reduce unnecessary repeated learning, but also improves the overall collaborative effect of the system. The experience sharing formula is defined as follows:

[0098]

[0099] where, represents the agent whether to share experience , represents the other set of fingers that collaborate with the agent .

[0100] ③Dynamic priority sampling of the replay pool

[0101] As the fingers learn, the sampling priority in the replay pool will be dynamically adjusted according to the task goals of the fingers, learning progress, and task importance. In the early stages of training, the fingers will rely more on experiences with large TD errors; while in the later stages of training, the proportion of diversity sampling will gradually increase to avoid overfitting and improve the generalization ability of the agent. The dynamic sampling weight formula is defined as follows:

[0102]

[0103] where, is the quality score of the experience , calculated by TD error and experience diversity, represents the relevance of the experience to the current task goal, is the total number of samples in the experience pool.

[0104] ④Independent management of the task pool

[0105] In a multi-task learning or multi-finger environment, the replay pool of each task can be independently managed. Each task replay pool will adjust the sampling strategy according to the progress, difficulty, and importance of the task, ensuring that the agent can focus on the experience of the current task while not ignoring the learning needs of other tasks. The formula for independent task pool is defined as follows:

[0106]

[0107] where, is the priority of the task, is the learning progress of the finger on this task, is the difficulty of the task.

[0108] Step 3: Phase reward mechanism

[0109] The reward mechanism proposed in this invention aims to optimize multi-agent grasping tasks by guiding the agent (finger) to gradually adjust its behavior during the grasping process through individual and global rewards. The reward mechanism is designed in two phases: a first phase (joint position reward and finger position reward) and a second phase (contact force reward). During the grasping process, the agent automatically switches the reward calculation method according to the task objective of the current phase.

[0110] ① Individual rewards

[0111] Individual rewards are designed by comprehensively considering joint angles, finger positions, and contact force to ensure that each finger exhibits optimal behavior in grasping tasks. The simplified formula for individual rewards is as follows:

[0112] (8)

[0113] in, It is the first The current position of each finger. It is the target location. It is the first The angle of each finger joint It is the initial angle of the joint. It is the first The contact force of one finger, It is the expected contact force. These are the corresponding weighting coefficients, which control the influence of different factors.

[0114] It's important to note that in the first stage (before contact with the object), the reward is determined by the difference in finger position and joint angle. The closer the finger position is to the target position, the greater the positional reward. The larger the joint angle (before contact with the object), the greater the joint reward. In the second stage (when the finger contacts the object), the reward is determined solely by the contact force. The closer the contact force is to the desired contact force, the greater the reward.

[0115] ②Global Rewards

[0116] The global reward is the sum of the individual rewards of all fingers, representing the overall task completion status of the multi-agent system. The global reward reflects the overall effectiveness of all fingers working together to complete the task. The formula for calculating the global reward is as follows:

[0117]

[0118] in, It is the first Individual reward for each finger. This indicates the number of fingers in the system.

[0119] ③ Phase switching mechanism

[0120] The reward mechanism dynamically switches the calculation method of the reward based on whether the finger is in contact with the object. If the contact force of the finger exceeds the set threshold , it indicates that the finger has contacted the object, and at this time, the contact force reward phase is switched to.

[0121]

[0122] where, is an exponential function that indicates whether to enter the second phase. If the contact force exceeds the threshold, it returns 1, indicating that it enters the second phase; otherwise, it returns 0, still in the first phase.

[0123] ④ Reward optimization and balance

[0124] Combine individual rewards and global rewards into a comprehensive overall reward This reward is a quantitative feedback of the overall performance of the multi-agent system (finger). The overall reward considers the task completion of each finger and the grasping effect of the entire system, and adjusts the priority of different task objectives through a weight coefficient.

[0125]

[0126] where, is the individual reward of the th finger at time step t, is the weight coefficient that controls the contribution proportion of individual rewards and global rewards in the overall reward. These two variables will be adjusted gradually as the training progresses. In the early stage of training, the agent needs to pay more attention to joint positions and finger positions, so individual rewards occupy a larger proportion. As the training progresses, the system gradually shifts to optimizing contact force and grasping stability, at which time the weight of global rewards increases.

[0127] Step 4: Training of MADDPG algorithm

[0128] This step describes the training process based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, focusing on policy network updates and value network updates. Through these updates, the agent can optimize its behavior strategy based on the rewards it obtains, achieving collaboration and task completion in multi-agent grasping tasks.

[0129] ① Policy network

[0130] In MADDPG, each agent selects actions through a deterministic policy. The policy network generates corresponding actions based on the current state, with the goal of maximizing the long-term return of each agent. The policy update of each agent is performed through the following formula:​

[0131]

[0132] in, It is the first Policy network parameters of each agent It is the first The policy of an agent, given a state Choose action The probability, It is a value network function, representing the agent's state. Take action below Expected returns This represents the maximum expected reward for the i-th agent. Indicating in strategy The expectations below.

[0133] ② Value Network

[0134] Each agent's value network is used to evaluate the value of the current state and actions, guiding the agent to choose the optimal action. The value network is updated using TD error.

[0135]

[0136] in, This is a discount factor, representing the importance of future rewards. Through TD error, the value network is continuously updated to ensure it accurately assesses the value of each state-action pair. Optimizing the value network helps the agent better estimate future rewards, thereby improving the performance of the policy network.

[0137] In summary, the training process of the intelligent grasping controller based on multi-agent reinforcement learning is as follows:

[0138] 1. Set the number of agents, initialize the environment state, reward function, and experience pool.

[0139] 2. In each time step, perform the following operations:

[0140] a. Each agent selects an action based on its local state through a policy network.

[0141] b. Joint action generation: The actions of all agents are combined into a joint action.

[0142] c. Perform combined actions to obtain new states and rewards.

[0143] d. Store the current experience in the experience replay pool of each agent.

[0144] 3. Experience sampling and replay pool update:

[0145] a. Each agent samples based on the importance and diversity of experience, and preferentially samples the experience that is most helpful to the current task.

[0146] b. Update the policy network: optimize the policy network through the policy gradient algorithm, and increase the long-term return.

[0147] c. Update the value network: update the value network through the TD error, and evaluate the value of the current action.

[0148] 4. Stage reward mechanism:

[0149] Determine the current stage (joint position stage or contact force stage): if the finger contacts the object, switch to the contact force stage, and the reward is calculated only according to the contact force. Otherwise, it is in the joint position stage, and the reward is calculated according to the finger position and joint angle.

[0150] 5. Stage switching:

[0151] Determine whether to switch stages according to the contact force. If the contact force exceeds the threshold, enter the second stage (contact force reward).

[0152] 6. End condition:

[0153] At the end of each training round, output the total reward. Determine whether the convergence condition (reward stability or maximum step number reached) is met. If so, stop training.

[0154] As a typical embodiment, as shown in Figure 3 , it is applied to a five-fingered dexterous hand. The five-fingered dexterous hand body 7 is connected to the mechanical arm 6, and the robot upper planning controller 8 is used to control the action of the five-fingered dexterous hand body 7, which can control the thumb 1, index finger 2, middle finger 3, ring finger 4 and little finger 5 to cooperate in grasping various objects.

[0155] The robot upper planning controller 8 is equipped with a storage medium, and the storage medium stores the method provided in embodiment one or the trained multi-agent system (model).

[0156] In a specific grasping task, as shown in Figure 4 , the following steps are included:

[0157] Set the number of agents and set the reward function;

[0158] Initialize the experience replay pool;

[0159] Get the global state, which includes the actions, position states and immediate rewards of all agents, and store the experience data in the agent experience replay pool;

[0160] Select the action of each finger / agent;

[0161] The actions of each agent are combined to obtain a joint action and executed;

[0162] It is judged whether the current reward stage needs to be switched, if yes, the second stage is switched, otherwise the first stage is maintained;

[0163] The policy network and the value network are updated;

[0164] It is judged whether the grasping is successful, if yes, the grasping task is ended, otherwise the step of selecting the action of each finger / agent is returned.

[0165] Embodiment two

[0166] An intelligent grasping control system based on multi-agent reinforcement learning, comprising:

[0167] A multi-agent system construction module is configured to regard each finger of the anthropomorphic dexterous hand as an independent agent, obtain local state information of itself from the global state, and generate a control action based on the respective policy network to form a multi-agent system;

[0168] A replay pool establishment module is configured to store experience data generated in the interaction process between each agent in the multi-agent system and the environment by using a replay pool mechanism, and the sampling priority of each agent replay pool is dynamically adjusted according to the task target, learning progress and task importance, and the experience data is shared between the replay pools;

[0169] A reward mechanism optimization module is configured to adopt a phased reward mechanism, and guide each agent in the multi-agent system to optimize the position, joint angle and contact force in different stages of the grasping task through individual rewards and global rewards;

[0170] An intelligent grasping control module is configured to train by using a multi-agent deep deterministic policy gradient algorithm, and perform intelligent grasping control by using the trained multi-agent system.

[0171] Embodiment three

[0172] An anthropomorphic dexterous hand applies the method of embodiment one or comprises the system of embodiment two.

[0173] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, etc.) containing computer-usable program code. CD - ROM ​

[0174] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0175] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0176] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0177] The preferred embodiments of the present application have been described above with the purpose to enable not to limit the application to a certain form and details of implementation, and it is understood by those skilled in the art that many changes and modifications in form and details can be made without departing from the spirit and scope of the application. Any modifications, equivalent replacements, improvements, etc. made by those skilled in the art without creative work shall fall within the scope of the present application.

Claims

1. A multi-agent reinforcement learning-based intelligent grasping control method, characterized in that, The method comprises the following steps: Each finger of the anthropomorphic dexterous hand is regarded as an independent agent, the local state information of each finger is obtained from the global state, and the control action is generated based on the respective policy network to form a multi-agent system; Experience data generated in the interaction process between each agent in the multi-agent system and the environment is stored by using a replay pool mechanism, the sampling priority of each agent's replay pool is dynamically adjusted according to the task target, learning progress and task importance, and the experience data is shared between the replay pools; A phased reward mechanism is adopted, and individual rewards and global rewards are used to guide each agent in the multi-agent system to optimize the position, joint angle and contact force in different stages of the grasping task; The multi-agent system is trained by using a multi-agent deep deterministic policy gradient algorithm, and intelligent grasping control is performed by using the trained multi-agent system; The process of adopting a phased reward mechanism to guide each agent in the multi-agent system to optimize the position, joint angle and contact force in different stages of the grasping task comprises that: the phased reward mechanism comprises two stages, the first stage is a joint position reward and finger position reward stage, and the second stage is a contact force reward stage; during the grasping process, the agent automatically switches the reward calculation mode according to the task target of the current stage, and the individual rewards and global rewards are used to guide each stage; The individual reward is determined by comprehensively considering the joint angle, finger position and contact force to ensure that each finger can exhibit the optimal behavior in the grasping task, and specifically: wherein, is the current position of the th finger, is the target position, is the joint angle of the th finger, is the initial angle of the joint, is the contact force of the th finger, is the desired contact force, is the corresponding weight coefficient for controlling the influence of different factors; The global reward is the sum of the individual rewards of all agents, representing the task completion of the entire multi-agent system, and is used to reflect the overall effect of the task completed by all fingers in cooperation, and the global reward is: ； wherein, is the individual reward for the th finger, denotes the number of fingers in the system; In the first stage, when the finger is not in contact with the object, the reward is determined by the difference between the finger position and the joint angle, the closer the finger position is to the target position, the greater the position reward is; the greater the joint angle is, the greater the joint reward is; in the second stage, when the finger is in contact with the object, the reward is only determined by the contact force, the closer the contact force is to the expected contact force, the greater the reward is; according to whether the contact force of the finger exceeds the set threshold value, the reward stage is determined, if the contact force of the finger is less than or equal to the set threshold value, the first stage is entered, otherwise the second stage is entered; The overall reward considers the task completion of each finger and the grasping effect of the entire system, and adjusts the priority of different task targets through a weight coefficient, and specifically: ； wherein, is the individual reward of the i-th hand at time step t, is the global reward of the i-th hand at time step t, is a weight coefficient, controlling the contribution proportion of the individual reward and the global reward in the total reward. At the beginning of training, the individual reward occupies a larger proportion, and with the progress of training, the weight of the global reward is gradually increased.

2. The intelligent grasping control method based on multi-agent reinforcement learning according to claim 1, characterized in that, Experience data generated in the interaction process between each agent in the multi-agent system and the environment is stored by using a replay pool mechanism, the sampling priority of each agent's replay pool is dynamically adjusted according to the task target, learning progress and task importance, and the experience data is shared between the replay pools; The replay pool preferentially selects experience data with a time difference error greater than a set value in the early stage, and preferentially samples experience data with diversity in the later stage; In a multi-task learning environment, the replay pool preferentially samples experience data of more important tasks; Or further, the dynamic adjustment process includes: in the early stage of training, each agent relies more on experience with a time difference error greater than a set value; in the later stage of training, gradually increase the proportion of diversity sampling, and the dynamic sampling weight is: wherein, is an empirical quality score, computed from the temporal difference error and the empirical diversity, represents the relevance of the experience to the current task goal, is the total number of samples in the experience pool.

3. The intelligent grasping control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of sharing experience data between replay pools includes: when some experience is lacking in the replay pool of part of the agents, relevant experience data is obtained from the replay pool of other agents through an experience sharing mechanism, which allows agents to accelerate learning by sharing data in the experience sharing pool without direct interaction.

4. The intelligent grasping control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of training using the multi-agent deep deterministic policy gradient algorithm includes: Set the number of agents, initialize the environment state, reward function, and experience replay pool; At each time step, each agent selects an action based on the local state through the policy network, the combination of all agent actions forms a joint action, the joint action is executed, the new state and reward are obtained, and the current experience is stored in each agent's experience replay pool; Each agent samples based on the importance and diversity of experience, prioritizes sampling the experience most helpful to the current task, optimizes the policy network through the policy gradient algorithm, updates the value network through the time difference error, and evaluates the value of the current action; Determine the current stage, if the finger contacts the object, switch to the second stage, the reward is calculated only according to the contact force, otherwise, stay in the first stage, the reward is calculated according to the finger position and joint angle; Determine whether to switch stages according to the contact force, if the contact force exceeds the threshold, enter the second stage; At the end of each round of training, output the total reward and determine whether the convergence condition is met, if so, stop training.

5. The intelligent grasping control method based on multi-agent reinforcement learning according to claim 4, characterized in that, The process of optimizing the policy network through the policy gradient algorithm and updating the value network through the time difference error includes: The process of updating the policy network includes: ； in, It is the first Policy network parameters of each agent It is the first The policy of an agent, given a state Choose action The probability, It is a value network function, representing the agent's state. Take action below Expected returns This represents the maximum expected reward for the i-th agent. Indicating in strategy The following expectations; The value network is updated through the time difference error: ； where, is a discount factor representing the importance of future rewards, r i represents the immediate reward obtained by the agent from the environment at the i th time step.

6. An intelligent grasping control system based on multi-agent reinforcement learning, characterized by comprising: A multi-agent system construction module configured to treat each finger of the anthropomorphic dexterous hand as an independent agent, obtain its own local state information from the global state, and generate control actions based on its own policy network to form a multi-agent system; A replay pool establishment module configured to store experience data generated during the interaction between each agent in the multi-agent system and the environment using the replay pool mechanism, and dynamically adjust the sampling priority of each agent's replay pool according to its task objective, learning progress, and task importance, and share experience data between replay pools; A reward mechanism optimization module configured to use a phased reward mechanism to guide each agent in the multi-agent system to optimize position, joint angle, and contact force in different stages of the grasping task through individual rewards and global rewards; An intelligent grasping control module configured to train using the multi-agent deep deterministic policy gradient algorithm and use the trained multi-agent system for intelligent grasping control; The phase reward mechanism is used to guide the agents in the multi-agent system to optimize the position, joint angle and contact force in different stages of the grasping task, and the process includes: the phase reward mechanism includes two stages, the first stage is the joint position reward and finger position reward stage, and the second stage is the contact force reward stage; during the grasping process, the agent automatically switches the reward calculation mode according to the task target of the current stage, and the individual reward and the global reward are used to guide each stage; The individual reward is determined by comprehensively considering the joint angle, finger position and contact force to ensure that each finger can perform the optimal behavior in the grasping task, and the specific process is: wherein, is the current position of the th finger, is the target position, is the joint angle of the th finger, is the initial angle of the joint, is the contact force of the th finger, is the desired contact force, is the corresponding weight coefficient for controlling the influence of different factors; The global reward is the sum of the individual rewards of all agents, representing the task completion of the whole multi-agent system, and is used to reflect the overall effect of all fingers in completing the task, and the global reward is: ； wherein, is the individual reward for the th finger, denotes the number of fingers in the system; In the first stage, when the fingers are not in contact with the object, the reward is determined by the difference between the finger position and the joint angle, the closer the finger position is to the target position, the greater the position reward is; the greater the joint angle, the greater the joint reward is; in the second stage, when the fingers are in contact with the object, the reward is only determined by the contact force, the closer the contact force is to the expected contact force, the greater the reward is; according to whether the contact force of the finger exceeds the set threshold, the reward stage is determined, if the contact force of the finger is less than or equal to the set threshold, it is in the first stage, otherwise it is in the second stage; The overall reward considers the task completion of each finger and the grasping effect of the whole system, and adjusts the priority of different task targets through a weight coefficient, and the specific process is: ； wherein, is the individual reward of the th finger at time step t, is the weight coefficient of the corresponding item, which controls the contribution ratio of the individual reward and the global reward in the total reward. At the beginning of training, the individual reward occupies a larger proportion, and with the progress of training, the weight of the global reward is gradually increased.

7. A humanoid dexterous hand characterized by, The method of any one of claims 1-5 or the system of claim 6 is applied.

Images