Robot non-grabbing external dexterous operation method based on rigidity perception double-stage reinforcement learning

Abstract

The invention belongs to the technical field of intelligent control, and discloses a robot non-grabbing external smart operation method based on rigidity perception dual-stage reinforcement learning, and the method comprises the steps: obtaining an environment and object point cloud through a depth camera, and obtaining a serialized tactile point cloud in combination with tentative tactile contact to perceive the object rigidity; multi-source point cloud information is fused through a high-level agent, and a target pose and an interaction action type are decided; the bottom intelligent agent plans a specific contact point and a motion direction according to the motion type; and the robot updates the state after executing the action, and circulates until the target is achieved. According to the method, the object rigidity information is fused into the decision process, the self-adaptive operation from rigid objects to soft objects is supported, the strategy does not need to be pre-defined manually, and the method has good generalization ability and learning efficiency and is suitable for robot flexible operation in a complex real scene.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology, and in particular relates to a non-grasping external dexterity operation method for robots based on stiffness-aware two-stage reinforcement learning. Background Technology

[0002] In the field of robotics, mimicking the multi-fingered dexterity of humans presents challenges due to its technical complexity and high cost. Therefore, parallel grippers, with their simple structure and ease of control, are more favored in practical applications. However, the inherent low degree of freedom of parallel grippers (typically only capable of opening and closing, with a fixed opening size) makes it difficult to directly grasp certain specific objects. For example, if the graspable area of ​​the object is larger than the gripper's opening, such as a book or a thin cardboard box lying flat on a table, its length and width are both larger than the gripper's opening. The thinner, more suitable vertical direction for grasping is not directly graspable because it is pressed against the table. If the book or box is adjusted to an upright position, fully exposing the thinner direction, it can be grasped. How to adjust initially ungraspable objects into a graspable posture through non-grasping methods is the main problem solved by this invention. For this type of problem, the academic community has proposed the concept of "external dexterity," which refers to the ability of robots to extend their manipulative capabilities by utilizing the external environment (such as planes, walls, and gravity). Operations that do not involve force-closed or shape-closed grasping are called "non-grasping operations," such as planar pushing, flipping with the aid of a plane, and lifting using vertical support.

[0003] Currently, most existing non-grasping manipulation techniques rely on manually designed motion primitives or make fixed assumptions about object properties (e.g., all are rigid bodies) and contact patterns. These solutions have significant drawbacks: First, they are typically designed for rigid objects, and their strategies are ineffective for soft objects. Differences in object stiffness lead to highly nonlinear and unpredictable changes in contact dynamics; a strategy designed for rigid objects to flip them may completely fail when used with soft objects. Second, these methods require pre-defined interaction parameters and patterns, demanding strict physical parameter settings. They lack autonomy and generalization ability, making it difficult to handle complex real-world scenarios with varying object geometry, mass, and stiffness. While some research has attempted to overcome the randomness of physical interactions through reinforcement learning and trial and error, effectively incorporating object stiffness—a key physical property—into the learning framework and coupling it with the decision-making and planning process remains a significant challenge. Existing methods struggle to generalize manually designed motion primitives and cannot effectively address the stiffness coupling problem. Therefore, there is an urgent need in this field for a non-grasping manipulation method that can adaptively perceive object physical properties (especially stiffness) and autonomously make decisions and plans. Summary of the Invention

[0004] The purpose of this invention is to provide a non-grasping external dexterity manipulation method for robots based on stiffness-aware two-stage reinforcement learning, in order to solve the technical problems of existing non-grasping manipulation techniques being unable to adaptively handle objects with different stiffnesses, relying on manually predefined strategies, and having poor generalization ability.

[0005] To address the aforementioned technical problems, the present invention provides a specific technical solution for a non-grasping external dexterity manipulation method for robots based on stiffness-aware two-stage reinforcement learning, as follows:

[0006] A non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning includes the following steps:

[0007] Step S1, Environment and Object Perception: Control the robot's end effector to move to the preset observation position, and acquire the 3D point cloud data of the current scene using a depth camera. and the point cloud of the manipulated object Serialized tactile point clouds were obtained through multiple tentative fingertip contacts by the robot. In addition, point clouds of the object's graspable area Known data, which represents the suitable area for grasping each object, can be loaded and obtained from a priori database;

[0008] Step S2, High-level Interaction Decision: The high-level intelligent agent fuses and infers the multi-source point cloud data obtained in Step S1 to determine the target pose T to which the object needs to be adjusted. pose And the recommended interaction action type A type ;

[0009] Step S3, Low-level Action Planning: Through the low-level intelligent agent, based on the current environmental point cloud... Object point cloud And the action type A of high-level decision-making type The specific contact points between the robot's end effector and the object are planned. and the direction vector of the action a dir ;

[0010] Step S4, Action Execution and State Update: Control the robot's end effector to execute a non-grasping interaction action according to the contact point and direction planned in step S3; after completion, return to the observation position and update the environment and object state;

[0011] Step S5, Loop Judgment: Repeat steps S3 and S4 until the object is successfully adjusted to the target pose or the maximum number of operation steps is exceeded.

[0012] Furthermore, the serialization of the tactile point cloud described in step S1 The following method is used: the robot gripper gently touches the upper surface of the object with a standard vertical posture and a contact force of a set threshold. The offset position of the fingertip after each contact is recorded by tactile sensors or robot kinematic estimation. This position information is mapped onto the initial point cloud of the object to form a point cloud sequence containing local regional deformation. This sequence contains the stiffness properties of the object.

[0013] Furthermore, the high-level interactive decision-making in step S2 specifically includes:

[0014] Step S2-1, Multi-source information extraction and fusion: Use the multi-source information extraction and fusion module to process the environmental point cloud with common input. Tactile point cloud sequence and prior grasping point cloud Feature encoding is performed. In the multi-information extraction and fusion module, a one-dimensional segmentation mask is added to each point in the three point clouds. The segmentation mask is used to distinguish the foreground from the background using 0 or 1, thus forming a 4-dimensional input. The inputs of the three point clouds are all extracted using the PointNet++ segmentation backbone network. Each input point cloud has an independent feature extraction module, and the weights are not shared. Furthermore, the tactile point cloud sequence... After feature extraction, the encoded features are input into a Long Short-Term Memory (LSTM) network to capture the dynamic deformation features of an object under contact. Finally, the encoded features from the three sources are concatenated to form a global fusion feature, which serves as the output of the multi-information extraction and fusion module.

[0015] Step S2-2, Policy Generation and Evaluation: The fused features obtained in step S2-1 are input into the policy network of the high-level agent to generate the environmental point cloud. For each candidate point belonging to the plane, a candidate action is generated. in The orientation of the object target is represented by a continuous vector, namely T. ori ; For action type decision-making represented by scalars, the reward value for each candidate action is further calculated through a value network. Action type decision Determined based on the range of output values: For "stamp", For "flip", that is, to obtain the specific interaction type: A type =0 is a stamp, A type =1 means flip;

[0016] Step S2-3, Decision Output: Select q i The candidate plane point with the highest value and its corresponding action parameters and As the final high-level decision output, it determines the target location, target orientation, and interaction type of the object.

[0017] Furthermore, the underlying action planning described in step S3 specifically includes:

[0018] Step S3-1, Feature Extraction: The input to the underlying agent is the scene observation point cloud, containing... and and the object pose error matrix Observe each point in the point cloud and The input is connected and a one-dimensional segmentation mask is added at the end to form a 7-dimensional input; the segmentation mask is used to distinguish between object points and environment points using 0 or 1; the environment points correspond to three dimensions. All values ​​are set to 0, and PointNet++ segmentation backbone network is used to extract features from the 7-dimensional input;

[0019] Furthermore, the object pose error matrix in step S3 is obtained by the following method: the observation point cloud of the object in its current pose. Transformation to the target pose planned by the higher-level intelligent agent and By performing vector subtraction on corresponding points, the pose error, or target distance, is obtained for each point, which is then used to form the pose error matrix for all object points.

[0020] Step S3-2, Action Generation and Evaluation: Input the encoded features into the policy network of the underlying agent to generate the object point cloud. Each point in the vector generates a direction vector for the robot gripper to interact with the object. At the same time, through value networks, combined with action type A of high-level decision-making... type Calculate the direction of each candidate point. Reward value for interaction

[0021] Step S3-3, Planning Output: Select q i The candidate point with the highest value and its direction vector are used as the contact point position and movement direction of the robot gripper in this interaction with the object.

[0022] Furthermore, both the high-level and low-level agents described in steps S2 and S3 are trained using an offline policy reinforcement learning algorithm based on Q-learning, with their objective function based on the Bellman equation. The two agents share a unified actor-critic framework and a hybrid discrete-continuous action space design, specifically implemented as follows:

[0023] Step S6-1: The high-level agent aims to learn a policy π. high This strategy outputs the optimal action a based on the state s. high Its network parameter updates follow these rules:

[0024] Let ψ represent the execution network π. high Given the network parameters, execute the network's loss function. Defined as all environmental plane points The expected Q value, and its selection probability. Weighting:

[0025]

[0026] In the formula: Where λ is the temperature parameter in the Softmax operation, used to adjust the exploration strategy and affect the selection of the target location for the redirected object; N p It is a point cloud Planar region points The number represents the number of available positions to place objects;

[0027] Let φ represent the comment network Q. high The network parameters are given by the dataset sampled from the empirical replay buffer. The loss of the Q-function of the comment network is calculated based on Bellman residuals:

[0028] in, Step S6-2: The underlying agent aims to learn a policy π. low This strategy outputs the optimal action a based on the state s. low Its network parameter updates follow these rules:

[0029] Let ψ represent the execution network π. low The network parameters. Its loss function is defined as the sum of all object points. The expected Q value, and its selection probability. Weighting:

[0030]

[0031] In the formula: Where λ is the temperature parameter, N o It is an object point cloud The number of dots represents the number of touchpoints available for interaction.

[0032] Let φ represent the comment network Q. low The network parameters, and their loss function, are also calculated based on Bellman residuals:

[0033] in,

[0034] Furthermore, in step S2, the reward function r of the high-level agent... high Designed as follows:

[0035] r high =S·exp(-(φ1·D) ori +φ2·D pos ))+R col -1

[0036] Where S is based on the object stiffness coefficient k s The target pose and action type are obtained through pre-simulation and data-driven modeling to obtain a continuous success rate distribution function, wherein the stiffness coefficient k s The deformation of the object's surface is obtained by tentatively pressing down using a tactile sensor or gripper, and the equivalent stiffness is mapped through the compression amount; R col D is a penalty for collisions with the environment. ori and D pos These represent the orientation and positional distance between the planned pose and the suggested pose, respectively. The suggested pose is selected from areas with high success rates and takes into account environmental distance weights. For the "flip" action, the supporting wall closest to the initial position is selected as the suggested pose reference; for the "poke" action, the pose with the shortest movement distance is selected as the suggested pose reference.

[0037] Furthermore, the process of obtaining the continuous success rate distribution function S is as follows:

[0038] (1) Define feasible space and sampling: Define the graspable pose of the object as the orientation with the Z-axis vertically upward, and sample the feasible target pose space. Uniform sampling is performed within the plane, where x, y are the positions of the object on the plane, and θ is the position of the object. z The yaw angle of the object about the Z-axis;

[0039] (2) Pre-simulation and data collection: For each sampled target pose In various object stiffness coefficients k s Next, use the trained underlying agent and specify the interaction strategy. Conduct a large number of randomized experiments and calculate their average success rate S;

[0040] (3) Construct a discrete mapping library: Through step (2), establish a library for the two strategies of "stamping" and "flipping" based on the stiffness coefficient k. s And the mapping database from discrete poses to success rate S, i.e., M pok / M flp :(k s ,x,y,θ z → [0,1];

[0041] (4) Constructing a continuous success rate distribution function: The two discrete mappings mentioned above are fused into a function with respect to the stiffness coefficient k using the following piecewise function and smoothing technique. s A continuous success rate distribution function S:

[0042]

[0043] In the formula:

[0044] and κ is the preset stiffness transition threshold; κ is a polarization weight coefficient used to adjust the extrema of the reward function, and its calculation formula is: Among them κ base For baseline weights, κ adj To adjust the amplitude, κ max This is the upper limit of the offset;

[0045] f is a sigmoid smoothing function used to achieve a smooth transition between the success rates of two strategies. Its calculation formula is as follows: Where σ mod σ is the modulation coefficient. norm For normalization center.

[0046] Furthermore, in step S3, the reward function r of the underlying agent... low Designed as follows:

[0047]

[0048] in, For points in the current object's point cloud, For the corresponding high-level target object, the point in the point cloud.

[0049] The non-grasping external dexterity manipulation method for robots based on stiffness-aware two-stage reinforcement learning of the present invention has the following advantages:

[0050] (1) Breakthrough in stiffness limitation: For the first time, object stiffness information is seamlessly integrated into the robot operation decision loop, enabling the same system to adaptively operate various objects from rigid to soft without redesigning or manually adjusting the strategy.

[0051] (2) Fully autonomous decision-making and planning are achieved: Through a two-stage reinforcement learning framework, the robot can directly output the final end effector control command from the input of raw sensor data (point cloud), and complete the entire process of "perception-decision-planning-execution" completely autonomously without any artificially predefined action primitives.

[0052] (3) Excellent generalization ability: The trained agent exhibits zero-shot generalization ability for unknown object geometry, mass, stiffness and new environment, and is easy to deploy in practical applications.

[0053] (4) Improved learning efficiency: The high-level algorithm adopts a continuous reward representation method, which effectively solves the sparse reward problem and accelerates the training process. The unified actor-critic framework and point cloud representation make the algorithm structure concise and efficient. Attached Figure Description

[0054] Figure 1 System overall workflow diagram;

[0055] Figure 2 A diagram illustrating the system's operational capabilities;

[0056] Figure 3 Detailed diagram of high-level intelligent agent network structure;

[0057] Figure 4 Detailed diagram of the underlying intelligent agent network structure;

[0058] Figure 5 An example diagram of this robot control system in a real test scenario. Detailed Implementation

[0059] To better understand the purpose, structure, and function of this invention, the following description, in conjunction with the accompanying drawings, provides a more detailed account of a robot non-grasping external dexterity manipulation method based on stiffness-aware two-stage reinforcement learning.

[0060] The technical problem to be solved by this invention is: how to enable a robot equipped with a low-degree-of-freedom gripper to autonomously perceive (including object stiffness and environmental geometry), make intelligent decisions (select interaction mode and target pose), and plan and execute (generate specific interactive actions) a series of non-grasping operations, thereby adjusting various ungraspable objects to a graspable pose.

[0061] The interaction modes here include two action types: "poke," "flip," and "push." ​​"Poke" involves the robot pressing on the edge of an object, using a flat surface (tablet) to flip it upright. This mode is intuitively more suitable for objects with high rigidity. For soft objects, pressing causes deformation, weakening the flipping lever arm and making it impossible to stand the object upright. "Flip" involves the robot moving upwards at one end of an object. When the other end of the object moves to a wall, the object is pressed against it, thus lifting and flipping it upright. This mode is more suitable for soft objects because the lever arm of the lifting force is greater than the object's weight lever arm, making it intuitively more suitable for soft objects. For "push," both "poke" and "flip" can achieve the same pushing effect. "Poke" can continuously flip the object to the corresponding target plane position, while "flip" can slide and push the object along a plane to the target plane position. The motion planning for "push" is a transitional motion planned during posture adjustment and is included in the planning of "poke" and "flip." Therefore, the "push" motion will not be separately described later. The purpose of this method is to enable the agent to spontaneously learn these two action types and to autonomously determine the action type and subsequent actions based on information such as the current scene, the current object state, and the contact feedback obtained from pressing the object.

[0062] This invention employs a two-stage reinforcement learning framework. The core idea of ​​this technical solution is to decouple complex operational tasks into two consecutive stages: high-level interactive decision-making and low-level action planning. A flowchart of the overall technical solution is attached. Figure 1 The present invention provides a non-grasping external dexterity manipulation method for robots based on stiffness-aware two-stage reinforcement learning, comprising the following steps:

[0063] Step S1, Environment and Object Perception: Control the robot's end effector to move to the preset observation position, and acquire the 3D point cloud data of the current scene using a depth camera. and the point cloud of the manipulated object Serialized tactile point clouds were obtained through multiple tentative fingertip contacts by the robot. In addition, point clouds of the object's graspable area Known data, which represents the suitable area for grasping each object, can be loaded and obtained from a priori database;

[0064] The serialized tactile point cloud The following method is used: the robot gripper gently touches the upper surface of the object with a standard vertical posture and a contact force of a set threshold. The offset position of the fingertip after each contact is recorded by tactile sensors or robot kinematic estimation. This position information is mapped onto the initial point cloud of the object to form a point cloud sequence containing local regional deformation. This sequence contains the stiffness properties of the object.

[0065] Step S2, High-level Interaction Decision: The high-level intelligent agent fuses and infers the multi-source point cloud data obtained in Step S1 to determine the target pose T to which the object needs to be adjusted. pose (including position T) pos and towards T ori And the recommended interaction type A type ("Poke" or "flip");

[0066] The high-level interactive decision-making specifically includes:

[0067] Step S2-1, Multi-source information extraction and fusion: Use the multi-source information extraction and fusion module to process the environmental point cloud with common input. Tactile point cloud sequence and prior grasping point cloud Feature encoding is performed. In the multi-information extraction and fusion module, each point (3D) in the three point clouds is added with a one-dimensional segmentation mask. The segmentation mask is used to distinguish the foreground from the background using 0 or 1, thus forming a 4D input. The inputs of the three point clouds are all processed using the PointNet++ segmentation backbone network for point cloud feature extraction. Each input point cloud has an independent feature extraction module, and the weights are not shared. Furthermore, the tactile point cloud sequence... After feature extraction, the encoded features are input into a Long Short-Term Memory (LSTM) network to capture the dynamic deformation features of an object under contact. Finally, the encoded features from the three sources are concatenated to form a global fusion feature, which serves as the output of the multi-information extraction and fusion module.

[0068] Step S2-2, Policy Generation and Evaluation: The fused features obtained in step S2-1 are input into the policy network (MLP) of the high-level agent to generate the environmental point cloud. For each candidate point belonging to the plane, a candidate action is generated. in The orientation of the object target is represented by a continuous vector, namely T. ori ; Action type decision is represented as a scalar. Further, the reward value for each candidate action is calculated using a value network (MLP).

[0069] Action type decision Determined based on the range of output values: For "poke", For "flip", the specific interaction type is obtained: A type =0 is a stamp, A type =1 means flip.

[0070] Step S2-3, Decision Output: Select q iThe candidate plane point with the highest value and its corresponding action parameters and As the final high-level decision output, it determines the target location, target orientation, and interaction type of the object.

[0071] Step S3, Low-level Action Planning: Through the low-level intelligent agent, based on the current environmental point cloud... Object point cloud And the action type A of high-level decision-making type The specific contact points between the robot's end effector and the object are planned. (This is a point cloud of an object) (a point in the vector a) and the direction vector a dir ;

[0072] The underlying action planning specifically includes:

[0073] Step S3-1, Feature Extraction: The input to the underlying agent is the scene observation point cloud, containing... and and the object pose error matrix (Same as point cloud format). Observe each point (3D) in the point cloud and... (Three-dimensional) connections are made, and a one-dimensional segmentation mask is added at the end to form a 7-dimensional input; the segmentation mask is used to distinguish between object points and environment points using 0 or 1; the environment points correspond to three-dimensional... All values ​​are set to 0. The PointNet++ segmentation backbone network is used to extract features from the 7-dimensional input.

[0074] The object attitude error matrix is ​​obtained through the following method: observation point cloud of the object at its current attitude. Transformation to the target pose planned by the higher-level intelligent agent and By performing vector subtraction on corresponding points, the pose error, or target distance, is obtained for each point, which is then used to form the pose error matrix for all object points.

[0075] Step S3-2, Action Generation and Evaluation: Input the encoded features into the policy network (MLP) of the underlying agent to generate the object point cloud. Each point in the vector generates a direction vector for the robot gripper to interact with the object. At the same time, through value networks (MLP), combined with the action type A of high-level decision-making, type Calculate the direction of each candidate point. Reward value for interaction

[0076] Step S3-3, Planning Output: Select q iThe candidate point with the highest value and its direction vector are used as the contact point position and movement direction of the robot gripper in this interaction with the object.

[0077] Step S4, Action Execution and State Update: Control the robot's end effector to execute a non-grasping interaction action according to the contact point and direction planned in step S3; after completion, return to the observation position and update the environment and object state;

[0078] Step S5, Loop Judgment: Repeat steps S3 and S4 until the object is successfully adjusted to the target pose or the maximum number of operation steps is exceeded.

[0079] Step S6, Training Process and Parameter Settings:

[0080] In steps S2 and S3, both the high-level and low-level agents are trained using an offline policy reinforcement learning algorithm based on Q-learning, with their objective function based on the Bellman equation. The two agents share a unified actor-critic framework and a hybrid discrete-continuous action space design. The specific implementation is as follows:

[0081] Step S6-1: The high-level agent aims to learn a policy π. high The strategy outputs the optimal action a based on the state s (multi-source point cloud). high (Target pose and action type). Its network parameter updates follow these rules:

[0082] Let ψ represent the execution network (Actor) π. high The network parameters. The network loss function is executed. Defined as all environmental plane points The expected Q value, and its selection probability. Weighting:

[0083]

[0084] In the formula: Where λ is the temperature parameter in the Softmax operation, used to adjust the exploration strategy and affect the selection of the target location for the redirected object; N p It is a point cloud Planar region points The number of positions represents the number of available spaces to place an object.

[0085] Let φ represent the comment network (Critic) Q. high The network parameters are given. The dataset is sampled from the empirical replay buffer. The loss of the Q-function of the comment network is calculated based on Bellman residuals:

[0086] in,

[0087] Step S6-2: The underlying agent aims to learn a policy π. low This strategy outputs the optimal action a based on the state s (environment and object point cloud, high-level action type decision). low (Contact point and direction of movement). Its network parameter updates follow these rules:

[0088] Let ψ represent the execution network (Actor) π. low The network parameters. Its loss function is defined as the sum of all object points. The expected Q value, and its selection probability. Weighting:

[0089]

[0090] In the formula: Where λ is the temperature parameter, N o It is an object point cloud The number of dots represents the number of touchpoint locations that can be used for interaction.

[0091] Let φ represent the comment network (Critic) Q. low The network parameters. Its loss function is also based on Bellman residual calculation:

[0092] in,

[0093] The reward function r of the high-level agent in step S2 high Designed as follows:

[0094] r high =S·exp(-(φ1·D) ori +φ2·D pos ))+R col -1

[0095] Where S is based on the object stiffness coefficient k s The target pose and action type are obtained through pre-simulation and data-driven modeling to obtain a continuous success rate distribution function, wherein the stiffness coefficient k s The surface deformation of an object can be obtained by tentatively pressing down using a tactile sensor or gripper, and the equivalent stiffness can be mapped through the compression amount; R col D is a penalty for collisions with the environment. ori and D posThese represent the orientation and positional distance between the planned pose and the suggested pose, respectively. The suggested pose is selected from areas with high success rates, taking into account environmental distance weights. For the "flip" action, the supporting wall closest to the initial position is selected as the suggested pose reference; for the "poke" action, the pose with the shortest movement distance (i.e., the closest) is selected as the suggested pose reference.

[0096] Furthermore, the process of obtaining the continuous success rate distribution function S is as follows:

[0097] (1) Define feasible space and sampling: Define the graspable pose of the object as the orientation with the Z-axis vertically upward. Within the feasible target pose space... Uniform sampling is performed within the plane, where x, y are the positions of the object on the plane, and θ is the position of the object. z The yaw angle of the object about the Z-axis;

[0098] (2) Pre-simulation and data collection: For each sampled target pose In various object stiffness coefficients k s Next, use the trained underlying agent and specify the interaction strategy. ("Poke" or "flip"), conduct a large number of random experiments (try to adjust to the target pose from different initial poses), and calculate the average success rate S;

[0099] (3) Construct a discrete mapping library: Through step (2), establish a library for the two strategies of "stamping" and "flipping" based on the stiffness coefficient k. s And the mapping database from discrete poses to success rate S, i.e., M pok / M flp :(k s ,x,y,θ z → [0,1];

[0100] (4) Constructing a continuous success rate distribution function: To address the issues of reward sparsity and non-smooth policy switching encountered by discrete mappings when training high-level agents, the two discrete mappings are fused into a single distribution function with respect to the stiffness coefficient k using the following piecewise function and smoothing technique. s A continuous success rate distribution function S:

[0101]

[0102] In the formula:

[0103] and κ is the preset stiffness transition threshold; κ is a polarization weight coefficient used to adjust the extrema of the reward function, and its calculation formula is: Among them κ baseFor baseline weights, κ adj To adjust the amplitude, κ max This represents the upper limit of the offset.

[0104] f is a sigmoid smoothing function used to achieve a smooth transition between the success rates of two strategies. Its calculation formula is as follows: Where σ mod σ is the modulation coefficient. norm For normalization center.

[0105] Furthermore, in step S3, the reward function r of the underlying agent... low Designed as follows:

[0106]

[0107] in, For points in the current object's point cloud, For the corresponding high-level target object, the point in the point cloud.

[0108] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning, characterized in that, Includes the following steps: Step S1, Environment and Object Perception: Control the robot's end effector to move to the preset observation position, and acquire the 3D point cloud data of the current scene using a depth camera. and the point cloud of the manipulated object Serialized tactile point clouds were obtained through multiple tentative fingertip contacts by the robot. In addition, point clouds of the object's graspable area Known data, which represents the suitable area for grasping each object, can be loaded and obtained from a priori database; Step S2, High-level Interaction Decision: The high-level intelligent agent fuses and infers the multi-source point cloud data obtained in Step S1 to determine the target pose T to which the object needs to be adjusted. pose And the recommended interaction action type A type ; Step S3, Low-level Action Planning: Through the low-level intelligent agent, based on the current environmental point cloud... Object point cloud And the action type A of high-level decision-making type The specific contact points between the robot's end effector and the object are planned. and the direction vector of the action a dir ; Step S4, Action Execution and State Update: Control the robot's end effector to execute a non-grasping interaction action according to the contact point and direction planned in step S3; after completion, return to the observation position and update the environment and object state; Step S5, Loop Judgment: Repeat steps S3 and S4 until the object is successfully adjusted to the target pose or the maximum number of operation steps is exceeded.

2. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 1, characterized in that, Step S1 describes the serialization of tactile point clouds. The following method is used: the robot gripper gently touches the upper surface of the object with a standard vertical posture and a contact force of a set threshold. The offset position of the fingertip after each contact is recorded by tactile sensors or robot kinematic estimation. This position information is mapped onto the initial point cloud of the object to form a point cloud sequence containing local regional deformation. This sequence contains the stiffness properties of the object.

3. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 1, characterized in that, The high-level interactive decision-making in step S2 specifically includes: Step S2-1, Multi-source information extraction and fusion: Use the multi-source information extraction and fusion module to process the environmental point cloud with common input. Tactile point cloud sequence and prior grasping point cloud Feature encoding is performed. In the multi-information extraction and fusion module, a one-dimensional segmentation mask is added to each point in the three point clouds. The segmentation mask is used to distinguish the foreground from the background using 0 or 1, thus forming a 4-dimensional input. The inputs of the three point clouds are all extracted using the PointNet++ segmentation backbone network. Each input point cloud has an independent feature extraction module, and the weights are not shared. Furthermore, the tactile point cloud sequence... After feature extraction, the encoded features are input into a Long Short-Term Memory (LSTM) network to capture the dynamic deformation features of an object under contact. Finally, the encoded features from the three sources are concatenated to form a global fusion feature, which serves as the output of the multi-information extraction and fusion module. Step S2-2, Policy Generation and Evaluation: The fused features obtained in step S2-1 are input into the policy network of the high-level agent to generate the environmental point cloud. For each candidate point belonging to the plane, a candidate action is generated. in The orientation of the object target is represented by a continuous vector, namely T. ori ; For action type decision-making represented by scalars, the reward value for each candidate action is further calculated through a value network. Action type decision Determined based on the range of output values: For "stamp", For "flip", the specific interaction type is obtained: A type =0 is a stamp, A type =1 means flip; Step S2-3, Decision Output: Select q i The candidate plane point with the highest value and its corresponding action parameters and As the final high-level decision output, it determines the target location, target orientation, and interaction type of the object.

4. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 1, characterized in that, The underlying action planning described in step S3 specifically includes: Step S3-1, Feature Extraction: The input to the underlying agent is the scene observation point cloud, containing... and and the object pose error matrix Observe each point in the point cloud and The input is connected and a one-dimensional segmentation mask is added at the end to form a 7-dimensional input; the segmentation mask is used to distinguish between object points and environment points using 0 or 1; the environment points correspond to three dimensions. All values ​​are set to 0, and the PointNet++ segmentation backbone network is used to extract features from the 7-dimensional input.

5. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 4, characterized in that, The object attitude error matrix in step S3 is obtained by the following method: observation point cloud of the object in its current attitude. Transformation to the target pose planned by the higher-level intelligent agent and By performing vector subtraction on corresponding points, the pose error, or target distance, is obtained for each point, which is then used to form the pose error matrix for all object points. Step S3-2, Action Generation and Evaluation: Input the encoded features into the policy network of the underlying agent to generate the object point cloud. Each point in the vector generates a direction vector for the robot gripper to interact with the object. At the same time, through value networks, combined with action type A of high-level decision-making... type Calculate the direction of each candidate point. Reward value for interaction Step S3-3, Planning Output: Select q i The candidate point with the highest value and its direction vector are used as the contact point position and movement direction of the robot gripper in this interaction with the object.

6. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 1, characterized in that, In steps S2 and S3, both the high-level and low-level agents are trained using an offline policy reinforcement learning algorithm based on Q-learning, with their objective function based on the Bellman equation. The two agents share a unified actor-critic framework and a hybrid discrete-continuous action space design, specifically implemented as follows: Step S6-1: The high-level agent aims to learn a policy π. high This strategy outputs the optimal action a based on the state s. high Its network parameter updates follow these rules: Let ψ represent the execution network π. high Given the network parameters, execute the network's loss function. Defined as all environmental plane points The expected Q value, and its selection probability. Weighting: In the formula: Where λ is the temperature parameter in the Softmax operation, used to adjust the exploration strategy and affect the selection of the target location for the redirected object; N p It is a point cloud Planar region points The number represents the number of available positions to place objects; Let φ represent the comment network Q. high The network parameters are given by the dataset sampled from the empirical replay buffer. The loss of the Q-function of the comment network is calculated based on Bellman residuals: in, Step S6-2: The underlying agent aims to learn a policy π. low This strategy outputs the optimal action a based on the state s. low Its network parameter updates follow these rules: Let ψ represent the execution network π. low The network parameters. Its loss function is defined as the sum of all object points. The expected Q value, and its selection probability. Weighting: In the formula: Where λ is the temperature parameter, N o It is an object point cloud The number of dots represents the number of touchpoints available for interaction. Let φ represent the comment network Q. low The network parameters, and their loss function, are also calculated based on Bellman residuals: in, 7. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 6, characterized in that, The reward function r of the high-level agent in step S2 high Designed as follows: Where S is based on the object stiffness coefficient k s The target pose and action type are obtained through pre-simulation and data-driven modeling to obtain a continuous success rate distribution function, wherein the stiffness coefficient k s The deformation of the object's surface is obtained by tentatively pressing down using a tactile sensor or gripper, and the equivalent stiffness is mapped through the compression amount; R col D is a penalty for collisions with the environment. ori and D pos These represent the orientation and positional distance between the planned pose and the suggested pose, respectively. The suggested pose is selected from areas with high success rates and takes into account environmental distance weights. For the "flip" action, the supporting wall closest to the initial position is selected as the suggested pose reference; for the "poke" action, the pose with the shortest movement distance is selected as the suggested pose reference.

8. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 6, characterized in that, The process of obtaining the continuous success rate distribution function S is as follows: (1) Define feasible space and sampling: Define the graspable pose of the object as the orientation with the Z-axis vertically upward, and sample the feasible target pose space. Uniform sampling is performed within the plane, where x, y are the positions of the object on the plane, and θ is the position of the object. z The yaw angle of the object about the Z-axis; (2) Pre-simulation and data collection: For each sampled target pose In various object stiffness coefficients k s Next, use the trained underlying agent and specify the interaction strategy. Conduct a large number of randomized experiments and calculate their average success rate S; (3) Construct a discrete mapping library: Through step (2), establish a library for the two strategies of "stamping" and "flipping" based on the stiffness coefficient k. s And the mapping database from discrete poses to success rate S, i.e., M pok / M flp :(k s ,x,y,θ z → [0,1]; (4) Constructing a continuous success rate distribution function: The two discrete mappings mentioned above are fused into a function with respect to the stiffness coefficient k using the following piecewise function and smoothing technique. s A continuous success rate distribution function S: In the formula: and It is a preset stiffness transition threshold; κ is a polarization weighting coefficient used to adjust the extrema of the reward function, and its calculation formula is as follows: Among them κ base For baseline weights, κ adj To adjust the amplitude, κ max This is the upper limit of the offset; f is a sigmoid smoothing function used to achieve a smooth transition between the success rates of two strategies. Its calculation formula is as follows: Where σ mod σ is the modulation coefficient. norm For normalization center.

9. The non-grasping external dexterous manipulation method for robots based on stiffness-aware two-stage reinforcement learning according to claim 6, characterized in that, The reward function r of the underlying agent in step S3 low Designed as follows: in, For points in the current object's point cloud, For the corresponding high-level target object, the point in the point cloud.

Images