Mechanical arm grabbing method based on human experience transmission

Abstract

The invention provides a mechanical arm object grabbing method combining imitation learning, reinforcement learning and human intervention, and aims to efficiently and safely migrate human expert experience to a robot and solve the problem that a traditional robot is low in learning efficiency from zero. Rapidly forming an initial strategy model through imitation learning; s2, performance optimization: the robot continuously adjusts and improves the strategy performance through interaction with the environment, task reward signals and reinforcement learning; and S3, performing safety intervention, wherein the robot evaluates the confidence coefficient of self decision in real time. And when the confidence coefficient is lower than a threshold value, human intervention is actively requested, and safe and reliable operation is ensured. S4, performing continuous evolution, regarding human intervention behaviors as high-quality new experiences, and storing the high-quality new experiences into an experience library. And retraining is carried out in combination with imitation learning and reinforcement learning, so that dynamic accumulation and continuous evolution of the strategy are realized. According to the method, the efficiency and robustness of the robot in the grabbing task are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, robotics, deep learning, and human-computer interaction, and more specifically to a human-computer hybrid intelligent decision-making and continuous evolution technology that integrates imitation learning and reinforcement learning. Background Technology

[0002] With the development of artificial intelligence technology, robots are increasingly being used in complex mission scenarios such as military, rescue, and industry. However, current intelligent decision-making technology for robots still faces significant bottlenecks. On the one hand, while deep learning-based models perform well in specific tasks, their generalization ability and adaptability to unknown environments are insufficient, and they are susceptible to performance degradation due to data distribution shifts. On the other hand, although reinforcement learning (RL) can achieve self-optimization through interaction with the environment, it suffers from slow convergence speed, low sample efficiency, and high exploration risk. Especially in high-risk tasks such as hazardous material disposal, a large amount of random trial and error is unacceptable.

[0003] Human experts, with their rich task knowledge and experience, are able to make flexible and reasonable judgments in uncertain and changing environments. Therefore, how to efficiently map human experience into robotic intelligent systems has become crucial to improving the autonomous decision-making capabilities of robots. Existing methods of experience transfer, such as rule-based design or offline demonstration dataset training, suffer from limitations such as being static, unable to adaptively update, having low experience utilization, and lacking continuous learning capabilities.

[0004] In conclusion, there is an urgent need for a new approach that can safely and efficiently utilize human experience while endowing robots with the ability to self-optimize and continuously evolve to adapt to the demands of complex and dynamic real-world tasks. Summary of the Invention

[0005] This invention aims to provide a robotic arm grasping method based on the transfer of human experience, addressing the shortcomings of existing technologies such as low robot learning efficiency, high risk, poor adaptability, and difficulty in efficiently utilizing human experience. This method aims to integrate the rapid experience transfer capability of imitation learning with the adaptive optimization capability of reinforcement learning, and achieve continuous evolution of robot capabilities through a human-machine collaborative closed loop. To this end, this invention adopts the following technical solution.

[0006] A robotic arm grasping method based on the transfer of human experience, the method comprising the following steps: S1: Based on a human expert demonstration dataset, the robot's policy model is initialized using an imitation learning algorithm to quickly acquire basic task capabilities; S2: Based on task-driven reward signals, the policy model is adaptively optimized using reinforcement learning algorithms to improve the policy's adaptability in changing environments; S3: During task execution, the decision reliability of the strategy model is monitored through a confidence assessment mechanism, and a human intervention request is triggered when the decision confidence is lower than a preset threshold. S4: The demonstration data generated during human intervention is dynamically accumulated as new experience and used to retrain the strategy model to achieve continuous evolution of the strategy and realize the mapping of human experience to robotic arm movements.

[0007] The embodiments of this invention bring the following beneficial effects: This application provides a robotic arm grasping method based on the transfer of human experience. On the one hand, it initializes through imitation learning and optimizes itself using reinforcement learning, greatly shortening the robot's learning cycle and enabling the robot to adapt to more varied and complex task environments, thus improving the generalization ability of the strategy. On the other hand, the confidence-based human-machine collaborative closed-loop mechanism allows the robot to dynamically learn from human experience, continuously accumulate new knowledge, and correct its own strategy deficiencies, achieving continuous evolution of capabilities. The robot only requests human assistance when needed, reducing the monitoring burden on human operators and achieving more intelligent and efficient human-machine collaboration.

[0008] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. The drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 The following is a flowchart illustrating the specific operation of the method provided in this embodiment of the invention. Figure 2 The present invention provides a structural diagram of the steps of the method. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer and easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0012] The flowchart and structural diagram of this invention are as follows: Figure 1 , Figure 2 As shown, a robotic arm object grasping method based on human experience transfer first initializes the robot's strategy using expert demonstration data through an imitation learning module. Then, the strategy is optimized by a reinforcement learning module based on reward signals during interaction with the environment. Simultaneously, a confidence assessment and intervention module monitors the reliability of the decisions in real time and requests human intervention when necessary. Finally, new experiences generated by human intervention are dynamically accumulated for retraining and continuous evolution of the strategy. The method includes the following steps: S1: Initialization of the behavior cloning strategy based on human demonstrations. The starting point for robot intelligent policy learning is the collection of demonstration data from human experts. Faced with various complex situations, such as objects being occluded or covered by other objects, the robotic arm cannot complete the grasping task. To address these complex situations, experienced operators first complete the target task via remote control devices (such as control handles or visualization software platforms), collecting operational data from human experts during task execution to construct a state-action pair demonstration dataset. Using behavior cloning, an initial policy network is trained through supervised learning, enabling the robot to possess near-expert-level operational capabilities from the task initiation stage, thus avoiding the inefficiency and risks of random exploration from scratch. The specific implementation formulas for frame sampling and preprocessing are as follows: ; ; Current status of the robotic arm; ; Given input state s, output the probability distribution for all 'a's. A collection of data gathered from human experts; Standard cross-entropy loss.

[0013] S2: Reward-driven adaptive policy optimization relies on the limited generalization ability of policies acquired through imitation learning, which is insufficient to handle new situations not covered by the demonstration data. Therefore, this step introduces reinforcement learning to adaptively optimize the initial policy. Based on the initial policy, the robot performs tasks by interacting with real or simulated environments. A composite reward function is designed based on task completion performance (e.g., success rate, path smoothness, safety). The specific reward model can be designed as follows: in: It rewards safe, smooth, and efficient robotic arm paths. It rewards the correct choice of gripping, laser, or intervention. It rewards those who proactively request human intervention in uncertain situations. Weighting. It can be dynamically adjusted according to task priority or actual needs. Furthermore, reinforcement learning algorithms are used to continuously optimize the policy network, enabling it to adapt to new situations beyond the demonstration data and discover better policies that surpass expert experience.

[0014] S3: Combining confidence feedback and human intervention, the robot does not always blindly execute decisions. At each step of task execution, it assesses the confidence level of its own decisions in real time, which is typically quantified by the action probability output by the policy network. When the confidence level falls below a preset threshold, it indicates that the robot has entered an unknown or uncertain state. At this point, the system will proactively pause and request intervention from a human operator or provide a new demonstration. The confidence feedback formula is as follows: ; Given input state s, output the probability distribution for all 'a's. Confidence feedback score; S4: Dynamic Experience Accumulation and Continuous Evolution. Human intervention behaviors are considered high-quality new demonstration data. These new state-action pairs are recorded and added to a dynamically expanding experience base. To efficiently utilize this new data, we employ a prioritized experience replay mechanism. Human demonstration samples triggered by low confidence are given higher sampling priority. This means that these samples will be selected more frequently in subsequent policy retraining, thereby helping the robot quickly fill knowledge gaps. The specific formulas for the dynamically expanding experience base and the prioritized experience replay mechanism are as follows: ； ; New human experience incorporated during the execution process; Indicates the first Priority of each example; Controlling the priority affects the intensity; human demonstrations triggered by low confidence are given higher priority. D dynamically adds human experience datasets; The sampling probability of prioritizing replay; The robot utilizes an updated experience base and retrains its policy network using a hybrid optimization algorithm that combines imitation learning and reinforcement learning objectives. This closed-loop process of "execution-evaluation-help-learning" is repeated continuously, allowing the robot's capabilities to evolve continuously. Through this joint optimization, the robot not only retains existing human knowledge but also continuously expands its policy space during task execution, avoiding getting stuck in fixed behavioral patterns. This ensures high adaptability and robustness in long-term tasks, ultimately achieving the goal of efficient and intelligent mapping of human experience to robotic arm movements. The specific formula for the hybrid reward optimization algorithm is as follows: ; The sample mean in the expert dataset; Given input state s, output the probability distribution for all 'a's. It is used to balance the importance of the two objectives: imitation learning loss and reinforcement learning reward; This represents the expectation of the trajectory generated by the robot's own strategy and its interaction with the environment. The discount factor determines the importance of future rewards; In state Execute The immediate reward provided by the environment afterwards; A hybrid optimization reward function is used to maximize the reward. Finally, the optimal operation mode for the robot under the current environmental conditions is determined through an optimization function, and then the robotic arm completes the grasping task according to the corresponding trajectory. This process enables the robot to cope with more complex and diverse environments and complete related tasks, achieving more intelligent and efficient human-machine intelligent hybrid decision-making.

[0015] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions of the same type within the technical scope disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot arm object grasping method based on human experience transfer mechanism, characterized in that, The method comprises the following steps: S1: based on the human expert demonstration data set, the policy model of the robot is initialized by the imitation learning algorithm to quickly obtain the basic task ability; S2: based on the task-driven reward signal, the policy model is adaptively optimized by the reinforcement learning algorithm to improve the adaptability of the strategy in the changing environment; S3: in the task execution process, the decision reliability of the policy model is monitored through the confidence evaluation mechanism, and when the decision confidence is lower than the preset threshold, the human intervention request is triggered; S4: the demonstration data generated in the human intervention process is accumulated as new experience dynamically, and is used for retraining the policy model to realize the continuous evolution of the strategy.

2. The end-to-end object grasping method for a robot arm based on human experience transfer mechanism according to claim 1, wherein, In step 1, the demonstration trajectory of human experts in complex environment is collected, which is set as experience set, wherein each trajectory is stored as state-action sequence, the difference between the action output by the policy model and the expert demonstration action is minimized by supervised learning, and finally the data is output to the imitation learning module, and the optimization goal is to maximize the log-likelihood prediction. The pre-processing is realized according to the following formula: ； ； current robot state; ； Input state s outputs a probability distribution over all a; a collection of human expert data; Standard cross-entropy loss.

3. The method of claim 1, wherein, In step 2, the reward signal is a composite reward function, which comprehensively considers the success rate of task, path smoothness, obstacle avoidance safety and operation efficiency. The reinforcement learning algorithm adopts PPO or SAC algorithm to ensure the stability and exploration efficiency of policy update.

4. The method of claim 1, wherein, In step 3, the decision confidence is dynamically added with human experience through the confidence feedback mechanism, so that the robot can continuously learn to handle complex tasks, realize the continuity and adaptability of the task, and the confidence feedback mechanism is inspired by the intervention module in human-machine cooperative task. An dynamic adjustment operator is added to realize efficient capture of uncertainty. The specific implementation formula of the confidence feedback is as follows: ； ； ； Confidence feedback score; Human experience newly added in execution; indicates the first The priority of the second The control priority influences the strength, with low-confidence triggered human demonstrations being given higher priority; D dynamically add human experience data set; Priority playback sample probability.

5. The general grasping method of a robot based on human experience transfer according to claim 1, wherein, In step 4, the dynamic experience accumulation process adopts the priority experience replay mechanism, and the demonstration data under low confidence of human intervention is given higher sampling weight to accelerate the learning of the model in the weak area. The retraining of the policy model adopts a hybrid optimization function combining imitation learning loss and reinforcement learning target. Finally, the prediction probability is converted into the grabbing path to realize the mapping of human experience to mechanical arm action.

Images