Robot grasping few-shot learning method and system for unstructured environment

Abstract

This invention discloses a few-shot learning method and system for robot grasping in unstructured environments, relating to the field of robot grasping technology. The method includes the following steps: acquiring tactile and visual information of the object being grasped by the robot; acquiring reference tactile and reference visual information of the object being grasped by a subject; setting a reward function based on the tactile information; training a joint motion model using behavior cloning technology based on the reference visual information to obtain the robot's initial strategy; and using a deep learning reinforcement model to fuse and analyze the tactile and visual information of the object being grasped by the robot, using the robot's initial strategy and the reference tactile information as references, to obtain the grasping strategy. This invention overcomes the problem of few-shot learning, significantly improving the robustness and autonomy of the robot system without human intervention, enabling it to exhibit stronger adaptability and generalization ability in complex and unknown environments.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, and in particular to a few-shot learning method and system for robot grasping in unstructured environments. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of robotics technology, service robots and industrial collaborative robots are gradually expanding their applications from structured environments to unstructured environments. Unlike traditional structured environments, unstructured environments typically exhibit high uncertainty and complexity, characterized by diverse object types, random placement, frequent dynamic changes in the environment, and susceptibility to sensor interference. In such environments, robots must possess stronger perception and understanding capabilities and adaptive control strategies to complete efficient and stable tasks, especially in complex grasping tasks, where the robustness and generalization ability of the grasping strategy are placed higher demands.

[0004] Humanoid grasping is an important direction in robotics research, aiming to endow robots with dexterity and compliance similar to human hands, enabling them to autonomously select appropriate finger postures and grasping methods based on the shape, texture, and pose of different objects. In unstructured scenarios, humans can quickly identify and grasp new objects through limited experience and a few demonstrations, while robots often rely on large amounts of data for training and lack the ability to flexibly adapt to new tasks.

[0005] For grasping tasks in environments, deep learning and reinforcement learning methods are currently widely used to enhance the robot's adaptive capabilities. While traditional deep learning and reinforcement learning methods perform well in standardized scenarios, they face two main problems: first, they are highly dependent on a large number of samples, resulting in high training costs and slow convergence; second, they lack transferability and generalization ability in new tasks or environments. Especially in real-world applications, it is impractical to conduct long-term training for every new object or grasping task. Therefore, how to achieve rapid learning and policy optimization for robots in unstructured environments under conditions of few or even single samples has become a key challenge in current intelligent grasping research.

[0006] In recent years, novel machine learning paradigms such as imitation learning have provided new solutions to the few-shot learning problem. However, its practical application in robotic grasping tasks still faces many challenges, including difficulties in fusing multimodal perceptual information and low efficiency in exploring high-dimensional action spaces. This is particularly pronounced in the control of humanoid robotic hands with multiple degrees of freedom, where the explosion of action space dimensions and the dramatic increase in policy search complexity make the few-shot learning problem even more significant.

[0007] Therefore, how to achieve efficient fusion and analysis of multimodal perception information under unstructured environments with few samples, and realize rapid learning and generalization transfer to improve the robot's operational capabilities and practicality in real and complex environments, has become a technical problem that needs to be solved by existing technologies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a few-shot learning method and system for robot grasping in unstructured environments. By using behavioral cloning technology to imitate human grasping behavior, the few-shot problem is overcome. Furthermore, by using deep reinforcement learning methods to fuse and analyze multimodal perceptual information, the robustness and autonomy of the robot system are significantly improved without human intervention, enabling it to exhibit stronger adaptability and generalization ability in complex and unknown environments.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0010] The first aspect of this invention provides a few-shot learning method for robot grasping in unstructured environments, comprising the following steps:

[0011] Acquire tactile and visual information of objects grasped by the robot, and acquire reference tactile and reference visual information of objects grasped by the subject;

[0012] Set a reward function based on tactile information;

[0013] By using behavioral cloning technology to train a joint motion model based on reference visual information, the robot's initial strategy can be obtained.

[0014] By using a deep learning reinforcement model, the tactile and visual information of the robot grasping the object is fused and analyzed with reference to the robot's initial strategy and reference tactile information, and a grasping strategy is obtained.

[0015] Furthermore, the tactile information is the fingertip force information of the robotic hand, the visual information includes the posture information and position information of the robotic hand, the reference tactile information is the fingertip force information of the subject's hand, and the reference visual information includes the posture information and position information of the subject's hand.

[0016] Furthermore, visual information is acquired through depth camera recognition, reference visual information is acquired through position sensor, and tactile information and reference tactile information are acquired through force sensor.

[0017] Furthermore, the specific steps for setting the reward function based on tactile information are as follows:

[0018] The threshold for segmented rewards is set based on reference tactile information as the expectancy.

[0019] Tactile information is segmented and labeled according to the segmented reward method, and combined with the interpolation reward of expectancy to form intrinsic reward;

[0020] External rewards are set based on the robot's successful grasping rate.

[0021] Furthermore, the specific steps for training a joint motion model based on visual information using behavioral cloning technology are as follows:

[0022] A joint motion model was constructed using a recurrent neural network.

[0023] By using behavior cloning technology, the joint position trajectory and corresponding state are taken as input, and the target joint position is taken as output. The joint motion model is trained to minimize the error between the predicted motion and the actual motion, thereby obtaining the robot's initial strategy.

[0024] Furthermore, the specific steps for fusing visual and tactile information using a deep learning reinforcement model based on the robot's initial policy and reward function are as follows:

[0025] The deep learning reinforcement model utilizes the deep deterministic policy gradient algorithm to train and iterate based on the robot's initial policy, using visual information as state input and combining it with a reward function to obtain the optimal grasping policy.

[0026] Furthermore, the MER mechanism is used to replace the original classic experience replay mechanism of the deep deterministic policy gradient algorithm. The MER mechanism is based on the meta-learning strategy. By weighting past experiences, it prioritizes sampling experiences that are more valuable to the current learning task, helping the algorithm converge to the optimal grasping strategy more quickly.

[0027] A second aspect of the present invention provides a few-shot learning system for robot grasping in unstructured environments, comprising:

[0028] The data acquisition module is configured to acquire tactile and visual information of the object grasped by the robot, and to acquire reference tactile and reference visual information of the object grasped by the subject.

[0029] The tactile information processing module is configured to set a reward function based on tactile information;

[0030] The visual information processing module is configured to use behavior cloning technology to train a joint motion model based on reference visual information to obtain the robot's initial strategy.

[0031] The vision-touch fusion module is configured to use a deep learning reinforcement model to fuse and analyze the tactile and visual information of the object grasped by the robot with reference to the robot's initial strategy and reference tactile information, so as to obtain the grasping strategy.

[0032] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing steps in the robot grasping few-shot learning method for unstructured environments as described in the first aspect of the present invention.

[0033] A fourth aspect of the present invention provides a computer device comprising:

[0034] A processor, adapted to execute computer programs;

[0035] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the few-shot learning method for robot grasping in unstructured environments as described in the first aspect of the present invention.

[0036] The above one or more technical solutions have the following beneficial effects:

[0037] This invention discloses a few-shot learning method and system for robot grasping in unstructured environments. First, using behavior cloning technology, a recurrent neural network (RNN) is trained based on human grasping demonstration data to learn an initial policy for grasping actions, accelerating the convergence speed of subsequent deep reinforcement learning policies. Then, based on this initial policy, a deep deterministic policy gradient (DDPG) algorithm is introduced to further optimize the policy through repeated trial and error in a simulated environment. The algorithm uses visual information as state input and tactile information as reward feedback, and integrates a meta-experience replay (MER) mechanism to more efficiently utilize training experience, accelerate the learning process, and improve policy quality. Compared to traditional control methods, this method exhibits stronger adaptability and generalization ability in complex and unknown environments, while significantly improving the robustness and autonomy of the robot system without human intervention.

[0038] This invention involves having subjects grasp different objects in space and move them to different positions, recording the spatial motion trajectory of the arm and the contact force information of the five fingers. Based on behavioral cloning, an initial control strategy for an extremity robot is generated. Subsequently, for multi-degree-of-freedom extremity-assisted grasping tasks, visual and tactile information are fused into a deep reinforcement learning algorithm. Visual information serves as the algorithm's state input, and force information serves as the algorithm's reward feedback, thereby optimizing the control strategy. This method achieves robot grasping control under visual-tactile fusion, providing new technical support for people with limb paralysis, significantly improving their daily living self-care abilities, and has broad application value.

[0039] This invention aims to accelerate the policy convergence speed of deep reinforcement learning by utilizing behavioral cloning to generate an initial policy. This involves learning from the operational experience of human experts in specific tasks, providing the robot with a high-performance starting policy, thereby accelerating the convergence speed of subsequent optimization processes. By mimicking the behavior of human experts, the robot can acquire prior knowledge about the task, reducing the inefficiency and instability caused by random exploration. This is particularly beneficial in complex tasks, improving overall learning efficiency and control effectiveness.

[0040] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a few-shot learning method for robot grasping in unstructured environments according to Embodiment 1 of the present invention.

[0043] Figure 2 This is a framework diagram of the DDPG deep reinforcement learning algorithm combined with the MER mechanism in Embodiment 1 of the present invention;

[0044] Figure 3 This is a framework diagram of the RNN algorithm used for behavioral cloning in Embodiment 1 of the present invention;

[0045] Figure 4 This is a flowchart of the experiment for collecting demonstration data from human experts in Embodiment 1 of the present invention;

[0046] Figure 5 This is a schematic diagram of the simulation training environment for the deep reinforcement learning algorithm in Embodiment 1 of the present invention;

[0047] Among them, 1. robotic arm, 2. robotic hand, 3. depth camera, 4. color block, 5. force sensor, and 6. object. Detailed Implementation

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] Example 1:

[0052] Embodiment 1 of the present invention provides a few-shot learning method for robot grasping in unstructured environments, such as... Figure 1 As shown, it includes the following steps:

[0053] Step 1: Obtain tactile and visual information of the object grasped by the robot, and obtain reference tactile and reference visual information of the object grasped by the subject.

[0054] In this embodiment, the tactile information is the fingertip force information of the robotic hand, the visual information includes the posture information and position information of the robotic hand, the reference tactile information is the fingertip force information of the subject's hand, and the reference visual information includes the posture information and position information of the subject's hand. The visual information is acquired through depth camera recognition, the reference visual information is acquired through position sensor, and the tactile information and reference tactile information are acquired through force sensor.

[0055] In one specific implementation, the learning target is a robot equipped with a robotic arm, which performs the task of grasping objects in cooperation with the robotic arm. Information on the robotic arm's fingertip force, posture, and position is acquired through the grasping task.

[0056] Step 1.1: Obtain tactile and visual information of the object grasped by the robot.

[0057] Step 1.1.1: Perform camera calibration and coordinate system transformation, and transfer the depth camera coordinate system and the robot arm coordinate system to the end effector coordinate system.

[0058] Specifically, camera calibration obtains the camera's intrinsic parameter matrix:

[0059]

[0060] Where K is the intrinsic parameter matrix of the camera, f x f y Let c be the focal length of the camera in the x and y directions. x c y This is the center of the camera's aperture in the x and y directions.

[0061] The homogeneous transformation matrix from the camera coordinate system to the end effector coordinate system is shown in equation (1):

[0062]

[0063] in, It is the homogeneous transformation matrix from the camera coordinate system to the end effector coordinate system. It is the homogeneous transformation matrix from the camera coordinate system to the world coordinate system. It is a homogeneous transformation matrix from the end effector coordinate system to the world coordinate system. This transforms the depth camera coordinate system and the robot arm coordinate system onto the end effector coordinate system.

[0064] Step 1.1.2: Use depth camera 3 to identify the position of the robotic fingertips in space, thereby obtaining the bending angle of the five fingers of the robotic hand. The fingertips of the robotic hand have color blocks of different colors 4. The depth camera locates the position of the robotic hand in space based on HSV(Hue,saturation,value) color tracking, and then uses inverse kinematics to calculate the bending angle of the five fingers of the robotic hand.

[0065] Specifically, when tracking the position of a mechanical fingertip, taking the position Pi of the index fingertip as an example, the coordinates (x, y, y) of the color patch at the index fingertip on the two-dimensional depth image are first obtained. i ,y i Then, the three-dimensional coordinates of the color patch in the camera coordinate system are obtained through equation (1). Equation (2) is shown below:

[0066]

[0067] Where d is the depth value of the depth camera, and s is the scaling factor of the depth map.

[0068] Then, the homogeneous transformation matrix from the camera coordinate system to the end effector coordinate system obtained in step 1.1.1. The three-dimensional coordinates of the mechanical fingertip in the end effector coordinate system can be calculated. As shown in equation (3):

[0069]

[0070] Finally, the finger bending angle of the robotic hand is calculated using inverse kinematics, as shown in equation (4):

[0071]

[0072] Where L1 is the phalanx length from the metacarpophalangeal joint to the proximal interphalangeal joint, L2 is the phalanx length from the proximal interphalangeal joint to the fingertip, θ1 is the joint angle of the metacarpophalangeal joint, and θ2 is the joint angle of the proximal interphalangeal joint.

[0073] Step 1.1.3: Identify the grasped object using a depth camera. Number the 12 types of grasped objects. For each grasp, randomly select one object and perform object recognition based on the YOLOv8 (Youonlylookoncevision8) algorithm, returning the corresponding object number.

[0074] Specifically, 12 common items in daily life were selected as the grasping training set, such as... Figure 5 As shown, the simulation training environment includes a robotic arm 1, a robotic hand 2, a depth camera 3, color blocks 4, a force sensor 5, and objects 6. Then, based on YOLOv8, the grasping training set is classified and identified. Each object is assigned a fixed number, which serves as the state input to the deep reinforcement learning algorithm framework. This allows the agent to distinguish the physical characteristics of different objects, thereby training a grasping strategy adapted to the objects and enhancing the algorithm's adaptability.

[0075] Meanwhile, when recognizing objects, the center of the recognition box in the depth camera image is used as the spatial position of the object. After coordinate transformation, the pixel coordinates of the center point can be converted into three-dimensional spatial coordinates in the end effector coordinate system. These coordinates are used for the spatial positioning of the object when the robot grasps it.

[0076] Step 1.1.4: Obtain fingertip force information using the force sensor on the mechanical fingertip. The force sensor on the mechanical fingertip is a 5*5 array force sensor. The sum of the forces measured at each array point of the force sensor is the fingertip force.

[0077] Specifically, the mechanical fingertip is equipped with a 5x5 array of pressure sensors, such as... Figure 5 As shown, robotic arm 1 drives robotic hand 2 to perform the task of grasping item 6, and the force sensor 5 detects the fingertip force of the robotic hand grasping the item. This fingertip force serves as reward feedback within the deep reinforcement learning algorithm framework, guiding the robot to complete the grasping task with appropriate force. The magnitude f of the force at each fingertip is... i As shown in equation (5):

[0078]

[0079] Where i represents the i-th finger, This represents the magnitude of the force detected at the nth array point of the i-th finger.

[0080] Step 1.2: Obtain reference tactile and reference visual information of the subject grasping the object.

[0081] Step 1.2.1: Use a data glove with an array of force sensors on the fingertips to collect information on the fingertip force of healthy people grasping objects and the spatial movement trajectory of the human hand as expert data.

[0082] Specifically, the data gloves used have 5x5 array pressure sensors at the fingertips, just like those in robotic hands, and position sensors are also provided at the corresponding finger joints to collect information on the fingertip force of a healthy person grasping an object, as well as the curves of joint angle changes.

[0083] This embodiment utilizes a data glove with array force sensors on the fingertips to collect information on the fingertip force of healthy individuals grasping objects and the spatial motion trajectory of their hands. This information is used as expert data to guide the training process of deep reinforcement learning.

[0084] This embodiment collected exercise data from eight healthy individuals. The experimental procedure is as follows: Figure 4 As shown. First, the subjects took their positions and put on data gloves. Then, they began grasping objects. Each subject needed to grasp the objects in the training set in turn, and the fingertip force of each finger was measured and the grasping rules of each finger were collected. A complete grasping process was defined as follows: the subject should grasp the object slowly with appropriate force, lift the object and hold it for 3 seconds without slipping or falling, and then put the object back down. Each object was grasped 5 times, and then the object was changed until all 12 objects were grasped successfully.

[0085] Step 1.2.2 Use Gaussian mixture regression to process the spatial motion trajectory of the human hand, eliminate the vibration and noise of the trajectory, and make it continuous and smooth.

[0086] For each object type, the fingertip force curves of all subjects across all trials are uniformly processed. Specifically, a trainable time warp algorithm is first used to time-align multiple sets of trajectories for each finger during the grasping of the same object, extracting their temporal consistency features. The optimization objectives are as follows:

[0087]

[0088] Where, x j τ(t) represents the state of the j-th trajectory at time t, N represents the total number of trajectories; τ(t) is the time warp function to be learned; It is the average trajectory at the current moment. This process ensures the temporal alignment consistency of different trajectories and extracts stable kinematic features.

[0089] Subsequently, a Gaussian mixture regression method based on expectation maximization was used to model and fit the aligned trajectory, thereby generating a continuous and smooth five-finger fingertip force expectation trajectory while preserving the statistical characteristics of the overall grasping pattern. The joint distribution was fitted using a Gaussian mixture model:

[0090]

[0091] Where p(ξ) is the fitting function, ξ=[t,x ξ ] T It is time t and state x ξ Joint variables; π m μ is the weight of the m-th Gaussian component, where M refers to the total number of Gaussian components; m and Σ m These are the mean and covariance, respectively, and N is a normal distribution.

[0092] Based on this, state regression is performed at any time point t* to obtain a continuous and smooth reference trajectory:

[0093]

[0094] in, h represents the reference trajectory. k (t*) is the normalized posterior probability weight; It is the mean of the k-th Gaussian component in the time and state dimensions; It is a sub-block matrix of covariance.

[0095] The final reference trajectory will serve as a reference for the expected trajectory in subsequent RNN model training.

[0096] Step 2: Set the reward function based on tactile information. Define a segmented reward function based on the fingertip force information, dividing it into ranges according to a threshold, as the reward feedback for the algorithm.

[0097] Step 2.1: Set the threshold for segmented rewards as the expectation force based on the reference tactile information.

[0098] The collected fingertip force information from healthy individuals is segmented and labeled to determine the threshold for segmented rewards. Within the threshold range, the algorithm is given a positive reward, while exceeding the threshold range results in a negative penalty.

[0099] Specifically, the trajectory of the fingertip force for the i-th finger. The trajectory is divided based on the mean and standard deviation, with the mean μ being... i It can be represented as:

[0100]

[0101] Where T represents the length of the time interval. Variance It can be represented as:

[0102]

[0103] Therefore, the upper limit f of the threshold force max and lower limit f min It can be represented as:

[0104]

[0105] Step 2.2: Collect the fingertip force during the robot's grasping process in real time, compare it with the expected force to form an interpolated reward, which serves as the intrinsic reward.

[0106] Step 2.3: After a complete grasping process is completed, set external rewards based on whether the robotic arm has successfully completed the grasping task.

[0107] Step 3: Use behavior cloning technology to train the joint motion model based on reference visual information to obtain the robot's initial strategy.

[0108] Step 3.1: Construct a joint motion model using a recurrent neural network.

[0109] Step 3.2: Using behavior cloning technology, the historical joint position trajectory (a temporal sequence of states) is used as input, and the target joint position (finger bending angle) at the next moment is used as output to train the joint motion model. By minimizing the error between the model's predicted action and the actual action, the robot's initial policy is learned.

[0110] Specifically, to construct the initial strategy for the robot's grasping task, this embodiment uses a behavior cloning method to learn from human grasping demonstrations. Joint angle trajectories collected during the human demonstration are used as training data, and an RNN is trained through supervised learning to predict the action to be performed at each moment. The algorithm flow is as follows: Figure 3 As shown, an RNN consists of an input layer, a hidden layer, and an output layer.

[0111] The input to an RNN is defined as: at time step t, the input is the state s. t That is, historical trajectory information q 1:t The output is defined as: the next action predicted by the RNN. That is, the target joint position (corresponding to the desired bending angle at the next moment). The supervision signal is defined as the actual movement a from a human demonstration. t Therefore, the training data pair can be represented as (x t ,a t ), where x t =[q 1:t ].

[0112] The basic recursive formula for RNN is:

[0113]

[0114] Among them, h t Let W be the hidden state, φ(·) be the activation function, and W be the activation function. h W is the weight matrix from hidden layer to hidden layer. x W is the weight matrix input to the hidden layer. o Let b be the weight matrix from the hidden state to the output, and b be the hidden layer bias vector. o This is the output layer bias.

[0115] The training objective is to minimize the model's predicted actions. With actual action a t The error between them is therefore considered using the mean squared error loss function:

[0116]

[0117] Where L is the mean squared error loss function, and T a Let be the total number of time steps for the trajectory. By minimizing this loss function, the RNN learns the strategies humans employ in different states to perform actions, thus constructing a grasping strategy π for initializing deep reinforcement learning training. θ This strategy, as a starting point for deep reinforcement learning, helps to shorten convergence time and improve early performance, especially in high-dimensional action spaces.

[0118] Step 4: Using a deep learning reinforcement model, the robot's initial strategy and reference tactile information are used to fuse and analyze the tactile and visual information of the object being grasped, so as to obtain the grasping strategy.

[0119] The initial reference policy trained in this embodiment guides the early exploration process of deep reinforcement learning. Subsequently, it is repeatedly trained in a simulation environment using the MER-based DDPG algorithm. During training, the maximum number of steps per episode is set to 25, and each training session includes 5000 episodes. To prevent the policy from getting trapped in local optima, decaying motion noise is added during training to increase the exploration range and promote stable policy convergence. The robotic arm continuously tries and adjusts the policy parameters through interaction with the environment to maximize the cumulative reward function. This reward function comprehensively considers multiple factors such as the distribution of fingertip force, whether the grasp is successful, whether excessive force is applied, and the stability of the grasp.

[0120] In one specific implementation, a simulation environment is built, and the generated initial strategy is applied to the simulation environment to verify whether its performance can achieve basic grasping. The DDPG algorithm, which integrates visual and tactile information, is trained iteratively to obtain the optimal strategy, thereby guiding the exolimb robot to complete the grasping task and ensuring high success rate, robustness, and generalization of grasping.

[0121] Specifically, a simulation environment is built based on the MuJoCo physics engine, such as... Figure 5 As shown, the initial policy generated by the behavior clone is applied to the simulation environment to check whether it can achieve basic grasping. If it can, it is used for deep reinforcement learning training; otherwise, the RNN neural network is retrained until an initial policy that can achieve basic grasping is obtained.

[0122] In one specific implementation, the deep learning reinforcement model utilizes the Deep Deterministic Policy Gradient (DDPG) algorithm to iteratively train and obtain the optimal grasping strategy based on the robot's initial strategy, using visual information as state input and combining it with a reward function. In this embodiment, the initial strategy is only used for initializing the initial policy network of the reinforcement learning algorithm. Based on the initial strategy generated by behavior cloning, the DDPG algorithm is repeatedly trained in a simulation environment. The policy gradient method is used to update the policy network, and the MER algorithm is used to improve the utilization rate of samples. At the same time, exploration noise is dynamically adjusted to balance exploration and utilization. The strategy is gradually optimized through trial and error to compensate for the shortcomings of behavior cloning in complex environments.

[0123] During training, the DDPG algorithm takes the bending angles of the robotic hand's five fingers and the desired angles as input, and outputs the finger bending angles of the robotic hand, randomly selected within the range of 0-2. Ornstein-Uhlenbeck noise (OUnoise) is added to the action selection process to generate time-dependent exploration using the OU process, thereby improving the exploration efficiency of the control task in the inertial system and avoiding the algorithm from getting trapped in local optima.

[0124] More specifically, deep reinforcement learning algorithm frameworks such as Figure 2 As shown, the state input is S, which includes the angles s of the five finger joints of the robotic arm. o , desired joint angle s e And the corresponding number k of the item to be grabbed.

[0125] The action space A in the deep reinforcement learning algorithm framework represents the control commands of the robotic arm, namely, grasping by dynamically adjusting the angles of the finger joints. Each time step outputs an action a. t ∈A, which represents the desired joint angle position of the robotic hand's fingers. Furthermore, this embodiment introduces OU noise during the motion to increase exploratory power, as shown in formula (6):

[0126] a t =μ(s) t |θ μ )+ε t (14).

[0127] Where, μ(s) t |θ μ () is a policy network based on parameters θ μ Predicted expected action, s t It is the state input at time step t, ε t For attenuating OU noise, the formula is shown in (15):

[0128]

[0129] Where, ε t-1 τ is the noise value at time step t-1. ε It is a parameter that controls the speed at which noise reverts to the mean. It is the target mean of the noise, σ min ζ is the minimum noise standard deviation at time step t, ζ is the decay rate of the standard deviation, and η ~ N(0,1) is a random variable with a standard normal distribution, representing random disturbance.

[0130] Rewards in deep reinforcement learning algorithm frameworks The reward r is determined by each time step t. t The cumulative reward is used to evaluate the quality of the model. The deep reinforcement learning model adjusts its strategy based on this cumulative reward, optimizing the strategy towards "higher rewards", thus demonstrating the guiding role of the reward function in the learning strategy. To avoid problems such as slow training convergence, low exploration efficiency, and unstable learning caused by sparse rewards, this embodiment formulates a piecewise reward function based on the threshold force required to grasp the object, as an intrinsic reward. The piecewise reward function for each finger i is shown in equation (16):

[0131]

[0132] Among them, f min and f max These correspond to the upper and lower limits of the threshold force obtained from behavioral cloning, respectively.

[0133] Next, the external reward will be calculated based on whether the grasp is successful or not, thus obtaining the total reward value for the entire episode:

[0134]

[0135] In one specific implementation, during the training process, the MER mechanism is used to replace the original classic experience replay mechanism of the deep deterministic policy gradient algorithm. The MER mechanism is based on the meta-learning strategy. By weighting past experiences, it prioritizes sampling experiences that are more valuable to the current learning task, helping the algorithm converge to the optimal grasping strategy more quickly.

[0136] Specifically, first, the random weights θ of the actor and critic target networks are initialized. μ and θ Q The experience revisit pool R is cleared. The depth camera begins observing the bending angles of the robotic arm's five fingers, forming a state tuple s. t The actor network is based on the state tuple s t Select the action to execute at the current moment (a) t After the agent performs an action, the state tuple becomes s. t+1 The action is updated to a t+1 At the same time, return the reward r for this step. t This forms a new Markov tuple {s} t ,a t ,r t ,s t+1 ,a t+1 The actor network stores this tuple in the experience buffer, and a portion of the stored tuple serves as the training dataset for the actor network. The critic network evaluates the current policy μ.

[0137] In the DDPG algorithm, the experience replay mechanism samples historical interaction data {s} from the experience pool R. t ,a t ,r t ,s t+1 ,a t+1 The goal of the Critic network is to optimize the policy μ and the value function Q. The objective is to minimize the following mean squared error loss:

[0138] L critic =(s t ,a t ,r t ,s t+1 ,a t+1 )~D[(Q(s t ,a t |θ Q )-y t ) 2 (18).

[0139] Among them, L critic For mean squared error loss, y t This is the target value.

[0140] The target value is:

[0141] y t =r t +γQ'(s t+1 ,μ'(s t+1 |θ μ' )|θ Q' (19).

[0142] The optimization of the Actor network is based on the deterministic policy gradient formula:

[0143]

[0144] in, For the policy gradient of the optimized Actor network, Represents the gradient of the value function. This represents the gradient of the current policy μ in the policy network.

[0145] The target network is updated using a soft update method, with the following formula:

[0146] θ Q' ←τ θ θ Q +(1-τ θ )θ Q' ,

[0147] θ μ' ←τ θ θ μ +(1-τ θ )θ μ' (twenty one).

[0148] Where, θ μ' and θ Q' Represent the updated θμ and θ respectively Q , τ θ A hyperparameter used to slowly update the target network parameters.

[0149] This embodiment replaces the original random experience replay mechanism of DDPG with the MER mechanism. The MER mechanism combines the ideas of meta-learning and gradient similarity, aiming to select the most useful samples for the current task from historical experience.

[0150] The core of the MER mechanism lies in measuring the similarity between the gradient of the current sample and the gradients of historical samples in the experience pool. Given the current sample {s} t ,a t ,r t ,s t+1 ,a t+1 The gradient of the policy network parameters is: The gradient of sample i in the experience pool is Define the gradient similarity between the two as cosine similarity.

[0151] Select k samples from the experience pool that are most similar to the current sample's gradient to form a meta-batch B. Then, perform an internal update, applying standard gradient descent on the selected meta-batch B to obtain temporary parameters.

[0152]

[0153] Where α is the internal update learning rate. Next, using the current sample {s} t ,a t ,r t ,s t+1 ,a t+1} at θ μ 'Calculate the loss and perform meta-gradient updates on the original parameters:'

[0154]

[0155] Where β is the meta-update learning rate.

[0156] This embodiment utilizes the MER experience replay algorithm and the DDPG algorithm, which integrates visual and tactile information, for training iteration to obtain the optimal strategy, thereby guiding the exolimb robot to complete the grasping task and ensuring a high success rate and generalization of grasping.

[0157] The above process involves continuous training and iteration until the robot finds a strategy that can obtain consistently high rewards over the long term, meaning the robot can complete the grasping task with a high success rate.

[0158] Example 2:

[0159] Embodiment 2 of the present invention provides a few-shot learning system for robot grasping in unstructured environments, comprising:

[0160] The data acquisition module is configured to acquire tactile and visual information of the object grasped by the robot, and to acquire reference tactile and reference visual information of the object grasped by the subject.

[0161] The tactile information processing module is configured to set a reward function based on tactile information;

[0162] The visual information processing module is configured to use behavior cloning technology to train a joint motion model based on reference visual information to obtain the robot's initial strategy.

[0163] The vision-touch fusion module is configured to use a deep learning reinforcement model to fuse and analyze the tactile and visual information of the object grasped by the robot with reference to the robot's initial strategy and reference tactile information, so as to obtain the grasping strategy.

[0164] Example 3:

[0165] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps of the robot grasping few-shot learning method for unstructured environments as described in Embodiment 1 of the present invention.

[0166] Example 4:

[0167] Embodiment 4 of the present invention provides a computer device, the device comprising:

[0168] A processor, adapted to execute computer programs;

[0169] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the robot grasping few-shot learning method for unstructured environments as described in Embodiment 1 of the present invention.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0172] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A few-shot learning method for robot grasping in unstructured environments, characterized in that, Includes the following steps: Acquire tactile and visual information of objects grasped by the robot, and acquire reference tactile and reference visual information of objects grasped by the subject; Set a reward function based on tactile information; The specific steps for setting a reward function based on tactile information are as follows: The threshold for segmented rewards is set based on reference tactile information as the expectancy. Tactile information is segmented and labeled according to the segmented reward method, and combined with the interpolation reward of expectancy to form intrinsic reward; External rewards are set based on the robot arm's successful grasping rate; By using behavioral cloning technology to train a joint motion model based on reference visual information, the robot's initial strategy can be obtained. By using a deep learning reinforcement model, the tactile and visual information of the robot grasping the object is fused and analyzed with reference to the robot's initial strategy and reference tactile information, and a grasping strategy is obtained. The specific steps for using a deep learning reinforcement model to fuse and analyze visual and tactile information based on the robot's initial policy and reward function are as follows: The deep learning reinforcement model uses the deep deterministic policy gradient algorithm to train and iterate based on the robot's initial policy, with visual information as the state input and a reward function, to obtain the optimal grasping policy. The MER mechanism replaces the original classic experience replay mechanism of the deep deterministic policy gradient algorithm. The MER mechanism is based on the meta-learning strategy. By weighting past experiences, it prioritizes sampling experiences that are more valuable to the current learning task, helping the algorithm converge to the optimal grasping strategy more quickly.

2. The robot grasping few-shot learning method for unstructured environments as described in claim 1, characterized in that, Tactile information is the fingertip force information of the robotic hand, visual information includes the posture information and position information of the robotic hand, reference tactile information is the fingertip force information of the subject's hand, and reference visual information includes the posture information and position information of the subject's hand.

3. The robot grasping few-shot learning method for unstructured environments as described in claim 1, characterized in that, Visual information is acquired through depth camera recognition, reference visual information is acquired through position sensor, and tactile information and reference tactile information are acquired through force sensor.

4. The robot grasping few-shot learning method for unstructured environments as described in claim 2, characterized in that, The specific steps for training a joint motion model based on visual information using behavioral cloning technology are as follows: A joint motion model was constructed using a recurrent neural network. By using behavior cloning technology, the joint position trajectory and corresponding state are taken as input, and the target joint position is taken as output. The joint motion model is trained to minimize the error between the predicted motion and the actual motion, thereby obtaining the robot's initial strategy.

5. A few-shot learning system for robot grasping in unstructured environments, employing the few-shot learning method for robot grasping in unstructured environments as described in any one of claims 1-4, characterized in that, include: The data acquisition module is configured to acquire tactile and visual information of the object grasped by the robot, and to acquire reference tactile and reference visual information of the object grasped by the subject. The tactile information processing module is configured to set a reward function based on tactile information; The visual information processing module is configured to use behavior cloning technology to train a joint motion model based on reference visual information to obtain the robot's initial strategy. The vision-touch fusion module is configured to use a deep learning reinforcement model to fuse and analyze the tactile and visual information of the object grasped by the robot with reference to the robot's initial strategy and reference tactile information, so as to obtain the grasping strategy.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-4: a robot grasping few-shot learning method for unstructured environments.

7. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements a few-shot learning method for robot grasping in unstructured environments as described in any one of claims 1-4.

Images