A Robot Task-Oriented Grasping Generation Method Based on Human-Guided Diffusion Model
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2026-05-29
- Publication Date
- 2026-06-30
Smart Images
Figure CN122299679A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot operation and control technology, specifically to a robot task-oriented grasping generation method based on a human-guided diffusion model, used to generate a framework for a robot's 6-DOF parallel gripper to grasp tasks. Background Technology
[0002] Task-oriented grasping is the first and crucial step in a robot's execution of maneuvers. It requires the grasping posture to remain stable while conforming to the constraints of the specific task. In daily life, humans naturally grasp objects in a task-oriented manner to facilitate subsequent maneuvers. Therefore, human grasping instinctively includes the skills required to manipulate objects (such as maintaining stability and avoiding collisions with the environment). Based on this, existing technologies typically use human grasping demonstrations as reference templates to generate task-oriented grasping for robots. Due to the significant structural differences between the human hand and a robotic gripper, current methods for transferring human grasping to a robot's 6-DOF parallel gripper mostly face problems such as complex conversion processes, time-consuming and labor-intensive processes, and difficulty in ensuring stability.
[0003] Machine learning and deep neural networks can find mapping patterns and learn autonomously through data interaction and training. Therefore, applying these algorithmic models to grasping generation can greatly improve the adaptability of robot grasping. Since the goal of task grasping is to obtain a stable posture that conforms to specific task constraints, the mechanism for converting human demonstrations into robot grasping generation plays a very important role. In early end-to-end solutions, manual rules or MLP (Multi-Layer Perceptron) networks trained on small datasets served as direct mappings. However, in current mainstream algorithms, the algorithms perform policy optimization and candidate filtering based on explicit constraints extracted from human demonstrations.
[0004] However, existing generation mechanisms face numerous limitations when transferring human grasping to a robot's 6-DOF parallel gripper. Early direct mapping methods struggle to handle the diversity of human grasping and the complexity of mapping relationships, often resulting in unstable grasps. Methods incorporating manual annotation or visual language models are also limited by high costs, weak generalization ability, and a lack of fine-grained object understanding, respectively. More critically, most mainstream algorithms currently rely on a "two-stage" sampling and filtering mechanism: first, a large number of candidates are blindly generated using a task-irrelevant sampler, and then a secondary filtering is performed using the aforementioned constraints. Due to the extremely large 6-DOF grasping space, this mechanism requires massive invalid sampling to occasionally hit a candidate that meets both requirements, leading to severe inefficiency and a high risk of sampling failure. In contrast, single-stage generation methods can directly integrate explicit task constraints into the underlying sampling and generation process, avoiding blind exploration in a vast state space. While ensuring stable grasping postures that meet task expectations, it completely eliminates cumbersome candidate filtering steps, significantly shortens inference time, and significantly improves the success rate and execution efficiency of grasping generation.
[0005] Addressing the inefficiencies and limitations in robot task-oriented grasping processes, this invention proposes a method to efficiently and stably generate a 6-DOF parallel gripper for task-oriented grasping in a single stage, overcoming the cumbersome two-stage sampling process of traditional methods. Recognizing the structural differences between human hand grasping and the difficulty in unifying the modeling of semantic constraints and physical executability between human hand grasping and robotic 6-DOF parallel grippers, this invention organically combines human grasping knowledge, task semantic information, and a diffusion generation mechanism to construct a single-stage task-oriented grasping generation framework. This framework enables robots to directly generate grasping postures that satisfy task constraints while possessing physical feasibility and stability, thereby significantly improving the success rate, efficiency, and generalization ability of task grasping generation. Summary of the Invention
[0006] The purpose of this invention is to propose a robot task-oriented grasping generation method based on a human-guided diffusion model. This method can overcome the problems in existing technologies, such as reliance on two-stage candidate sampling and filtering, low generation efficiency, many invalid samples, insufficient task adaptability, and difficulty in balancing grasping stability. It is a simple and easy-to-implement method that can improve the success rate, efficiency, and generalization ability of task grasping generation.
[0007] The technical solution of this invention: A robot task-oriented grasping generation method based on a human-guided diffusion model, characterized by comprising the following steps: (1) Obtain the observation information of the target object and the human grasping demonstration information corresponding to the current task, and establish the robot grasping state representation to establish the local geometric observation information of the target object, the human demonstration information and the robot grasping variables; The observation information of the target object in step (1) is denoted as This includes at least one information item from the target object's point cloud, depth image, local geometric features, normal vector information, and 3D mesh model; the human grasping demonstration information is denoted as... It is used to characterize the grasping behavior characteristics of a person when performing a corresponding task, including at least one of the following information items: hand contact area, approach direction, grasping orientation, functional part preference, and operation status information when performing the corresponding task. The establishment of the robot grasping state representation in step (1) specifically refers to: in order to provide a unified parameterized representation of the robot's six-degree-of-freedom parallel gripper grasping state for subsequent diffusion modeling and backsampling updates, the robot grasping state is represented as: H=(t,g,w) (1) Where t represents the spatial translation parameter of the gripper, g represents the attitude parameter of the gripper, and w represents the opening width of the gripper. This indicates the gripping state of the robot's parallel gripper.
[0008] (2) Based on the target object observation information obtained in step (1) and the robot grasping state samples in the publicly available CONG dataset, a task-independent grasping diffusion prior model is established to learn the task-independent grasping prior distribution of the robot grasping state under the given target object observation conditions. Through this step, the robot's basic grasping prior for the target object observation conditions is obtained, so that the subsequent generated results have basic geometric feasibility and grasping stability. The robot grasping state sample includes at least the spatial position, orientation, and gripper opening width information of the robot's six-degree-of-freedom parallel gripper.
[0009] In step (2), establishing a task-independent grasping diffusion prior model to learn the robot grasping distribution under given target object observation conditions specifically means: defining the robot grasping distribution under given target object observation conditions as a task-independent grasping distribution. During training, the grasping state of the robot's parallel gripper was monitored. Add Gaussian noise to obtain a noisy capture state. Its expression is: (2) in, The noise scale is represented by ε, which represents standard Gaussian noise. Represents the identity matrix; Noisy capture state based on formula (2) Construct a noise conditional fractional network This makes it approximate the noisy conditional distribution's fractional function with respect to the grasping variable, i.e. (3) in, The noise scale is represented as Distribution of noisy capture conditions at that time Represents a noise-conditional fractional network. Represents network parameters, Indicates the state of noisy capture. The gradient operator; The noise conditional score network is trained using a denoising score matching loss function: (4) in, The noise scale-related weighting coefficients are: This represents the denoising score matching loss function; After training, the task-independent grasping diffusion prior model composed of formulas (2) to (4) is used to learn and approximate the observation information of a given target object. Task-independent grasping prior distribution of robot grasping state under given conditions In the subsequent diffusion backsampling process, a noise-conditional fractional network is used. The conditional fractional function of the robot's grasping state is estimated, and the robot's grasping state is iteratively updated accordingly.
[0010] Step (2) is used to establish a task-independent grasping diffusion prior model under the observation conditions of the target object. Step (3) is used to construct a unified guiding representation corresponding to the human grasping demonstration and the local geometric information of the target object. Step (4) uses the basic prior of step (2) and the unified guiding representation of step (3) to jointly generate the task-oriented grasping result during the diffusion backsampling process.
[0011] (3) Based on the human grasping demonstration information and the local geometric observation information of the target object obtained in step (1), construct a unified guidance representation that matches the observation information of the target object obtained in step (2); To address the issues of modal heterogeneity and dimensional inconsistency between human-captured demonstration information and local geometric observations of the target object, step (3) specifically includes the following: (3-1) Extract state information from human capture demonstrations. and local geometric observation information of the target object Perform feature encoding to obtain state features and geometric features Its expression is: (5) in, and These represent the corresponding feature encoding networks. This represents the state characteristics of a human grasping demonstration. Represents the local geometric observation features of the target object; (3-2) Employing a multimodal fusion mechanism to integrate the state features With geometric features Integration into a unified guiding representation, denoted as : ① State characteristics respectively With geometric features Feature mapping and dimension alignment are performed to give both features the same or compatible feature dimensions, resulting in aligned state features. With geometric features Its expression is: (6) in, and These represent the feature mapping matrices, and These represent the bias terms; ②Then, after completing the dimension alignment, the state features are... With geometric features Joint modeling is performed, and in this implementation, the fusion module is implemented by combining feature concatenation and multilayer perceptron mapping to output a unified guided representation. Its expression is: (7) Where [·;·] represents the feature concatenation operation, ψ represents the fusion network, and the unified guidance representation This is used to characterize the correspondence between human grasping demonstrations and the local geometry of the target object, thereby providing prior information for subsequent robot task-oriented grasping generation.
[0012] (4) Based on the task-independent grasping diffusion prior model established in step (2), the unified guidance representation obtained in step (3) is applied. A diffusion backsampling process is introduced to impose task-related constraints on the grasping state in the sampling update corresponding to the prior model, thereby generating a reasonable grasping pose that meets the task requirements. This step introduces the fused human grasping demonstration features and the local geometric features of the target object into the diffusion backsampling process, so that the sampling can start from more informative priors rather than just random Gaussian noise, thereby reducing the number of invalid samples and improving the accuracy, stability and task consistency of robot task grasping generation. Step (4) specifically refers to: (4-1) The unified guiding representation obtained in step (3-2) After feature space alignment, it is embedded as conditional guiding information in the diffusion backsampling process, so that it participates in each step of the grasping state update. The step (4-1) specifically refers to: ① Feature space alignment: through a conditional coding network, the unified guided representation is aligned. ① Transform to a feature space that is the same as or compatible with the robot's grasping state representation; ② Introduce a diffusion backsampling process: In each backsampling iteration, the aligned unified guided representation is transformed... With the current grasping state and target object observation Together, these serve as conditional information for updating the grasping state, guiding the grasping position and direction during the current grasping state update process. This ensures that the grasping state update process no longer relies solely on the task-independent grasping score, but is simultaneously influenced by the observation of the target object. Demonstration of human grasping Consistent guidance information constraints enhance the observation of target objects. Human grasping demonstration The coupling between the robot's grasping posture generation and the robot's grasping posture generation.
[0013] (4-2) Using the observation information of the target object obtained in step (1) and the human grasping demonstration information as joint conditions, establish the robot task-oriented grasping target distribution: (8) in, This indicates the distribution of targets that the robot is task-oriented in grasping. This represents the task-independent grasping distribution under the conditions of target object observation. This indicates the human's grasping demonstration information and the robot's grasping state. The distribution of guiding constraints; (4-3) From the robot task-oriented grasping target distribution shown in formula (8), its logarithmic gradient expression with respect to the grasping state can be obtained as follows: (9) in, Indicates the state of the grabbing process. The gradient operator has a first term on the right that represents the task-independent capture score term and a second term on the right that represents the explicit guiding term. (4-4) Based on the logarithmic gradient expression of the grasping state obtained from formula (9), in each step of backsampling, the task-independent grasping score term and the explicit guidance term are weighted and fused as the update direction of the grasping state, and the current grasping state is iteratively updated in combination with the random perturbation term. The expression is as follows: (10) in, This represents the grasping posture at the k-th iteration. The grasping posture η represents the grasping posture at the (k-1)th iteration. k This represents the current step size parameter, where α is the guiding intensity coefficient. This represents a random perturbation term; after multiple iterations, a reasonable grasping pose that meets the task requirements is finally obtained.
[0014] (5) Perform a comprehensive quality assessment on the reasonable grasping pose generated by diffusion backsampling in step (4), and select one or more grasping poses with the highest scores as output results based on the assessment results, and send them to the robot control system, thereby realizing the robot's task-oriented grasping of the target object.
[0015] Step (5) specifically refers to: Let the task consistency scoring function be... The capture stability scoring function is: The environmental collision penalty function is The comprehensive scoring function expression is: (11) in, , and These are the weight coefficients for the task consistency item, the grasping stability item, and the collision penalty item, respectively. After comprehensively scoring the reasonable grasping pose generated in step (4) according to formula (11), one or more grasping poses with the highest scores are selected as the output results and sent to the robot control system to drive the robotic arm and gripper to complete the corresponding target grasping and subsequent operation tasks.
[0016] The working principle of this invention is as follows: First, using observational information such as point clouds, depth images, local geometric features, or 3D mesh models of the target object, a geometric representation of the target object is established. Then, based on existing robot grasping samples, a task-independent grasping diffusion prior model is trained to learn the basic distribution law of robot grasping posture under target object observation conditions. On this basis, human grasping demonstration information corresponding to the current task is further introduced. The contact area preference, approach direction preference, functional part selection preference, and operational state information exhibited by humans when performing the same or similar tasks are encoded as human guidance constraints, and these constraints are then implemented through state features. With local geometric features The invention first constructs a unified guiding representation by fusing the observation information of the target object. Then, during the diffusion backsampling process, the task-independent grasping distribution learned from the target object observation information is jointly modeled with the guiding constraints constructed from the human grasping demonstration information. This ensures that the update direction of the grasping posture is simultaneously constrained by both the geometric graspability of the object and the semantic consistency of the task, thereby gradually generating reasonable grasping poses that meet the task requirements. Finally, by comprehensively scoring the task consistency, grasping stability, and environmental collision risk of the reasonable grasping poses, one or more grasping poses with the highest scores are selected as output results and sent to the robot control system to drive the robotic arm and gripper to complete the corresponding grasping and subsequent operation tasks. Thus, this invention realizes a unified mapping process from "target object geometric observation - human task grasping demonstration - robot task-guided grasping generation," enabling the robot to efficiently generate grasping poses that balance physical executability and task adaptability in a single stage.
[0017] The advantages of this invention are: ① By establishing a task-independent grasping diffusion prior model under target object conditions and directly introducing human grasping demonstration constraints in the backsampling stage, the robot grasping generation process is transformed from the traditional "two-stage candidate sampling + posterior filtering" mode to a "single-stage guided generation" mode, reducing the number of invalid grasping candidates from the source and significantly improving grasping generation efficiency; ② Introducing human contact area preferences, approach direction preferences, and functional part selection preferences formed when performing specific tasks into the robot grasping generation process, the generated results not only meet the basic grasping stability requirements but also better conform to the semantics of specific tasks, thereby improving the consistency and success rate of task-oriented grasping; ③ By analyzing the characteristics of human grasping demonstration states and the local features of the target object... The invention integrates various features to construct a unified guiding representation, effectively mitigating the differences in structural form, motion space, and contact method between human hands and robot parallel grippers, and enhancing the feasibility of transferring human grasping knowledge to robot grasping strategies; ④ After the grasping is generated, the invention further introduces task consistency scoring, grasping stability scoring, and environmental collision penalty mechanisms, which can comprehensively score and select the best output for the generated reasonable grasping pose, thereby obtaining grasping results more suitable for real robot execution; ⑤ It is applicable to different types of target objects, different task constraints, and different parallel gripper platforms, and has strong generalization and expansion capabilities, thus having good application prospects in fields such as intelligent robot operation, service robots, autonomous assembly, and grasping in complex scenarios. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the overall process of a robot task-oriented grasping generation method based on a human-guided diffusion model, which is involved in this invention.
[0019] Figure 2This is a schematic diagram of the diffusion backsampling and optimal grasping output process in a robot task-oriented grasping generation method based on a human-guided diffusion model, which is involved in this invention.
[0020] Figure 3 is a schematic diagram illustrating the human grasping of the cup target object in an embodiment of the present invention and the distribution of reasonable grasping postures. Figure 3-a This is a diagram illustrating how a human grasps a cup when performing a drinking-related task. Figure 3-b This is a schematic diagram illustrating the reasonable grasping pose distribution generated based on local point cloud observation information of the target object. Detailed Implementation
[0021] Example: Figure 1 As shown, a robot task-oriented grasping generation method based on a human-guided diffusion model is characterized by the following steps: (1) Obtain the observation information of the target object and the human grasping demonstration information corresponding to the current task, and establish the robot grasping state representation to establish the local geometric observation information of the target object, the human demonstration information and the robot grasping variables; Among them, the observation information of the target object is denoted as This includes at least one information item from the target object's point cloud, depth image, local geometric features, normal vector information, and 3D mesh model; the human grasping demonstration information is denoted as... It is used to characterize the grasping behavior characteristics of a person when performing a corresponding task, including at least one of the following information items: hand contact area, approach direction, grasping orientation, functional part preference, and operation status information when performing the corresponding task. The process of establishing the robot grasping state representation specifically refers to: in order to provide a unified parameterized representation of the robot's six-degree-of-freedom parallel gripper grasping state for subsequent diffusion modeling and backsampling updates, the robot grasping state is represented as: H=(t,g,w) (1) Where t represents the spatial translation parameter of the gripper, g represents the attitude parameter of the gripper, and w represents the opening width of the gripper. This indicates the gripping state of the robot's parallel gripper.
[0022] (2) Based on the target object observation information obtained in step (1) and the robot grasping state samples in the publicly available CONG dataset, establish a task-independent grasping diffusion prior model consisting of formulas (2) to (4) to learn the task-independent grasping prior distribution of the robot grasping state under the given target object observation conditions; the robot grasping state samples include at least the spatial position, orientation and gripper opening width information of the robot's six-degree-of-freedom parallel gripper; The robot grasping distribution under given target object observation conditions is defined as the task-independent grasping distribution. During training, the grasping state of the robot's parallel gripper was monitored. Add Gaussian noise to obtain a noisy capture state. Its expression is: (2) in, The noise scale is represented by ε, which represents standard Gaussian noise. Represents the identity matrix; Noisy capture state based on formula (2) Construct a noise conditional fractional network This makes it approximate the noisy conditional distribution's fractional function with respect to the grasping variable, i.e. (3) in, The noise scale is represented as Distribution of noisy capture conditions at that time Represents a noise-conditional fractional network. Represents network parameters, Indicates the state of noisy capture. The gradient operator; The noise conditional score network is trained using a denoising score matching loss function: (4) in, The noise scale-related weighting coefficients are: This represents the denoising score matching loss function; After training, the task-independent grasping diffusion prior model composed of formulas (2) to (4) is used to learn and approximate the observation information of a given target object. Task-independent grasping prior distribution of robot grasping state under given conditions In the subsequent diffusion backsampling process, a noise-conditional fractional network is used. The conditional fractional function of the robot's grasping state is estimated, and the robot's grasping state is iteratively updated accordingly.
[0023] (3) Based on the human grasping demonstration information and the local geometric observation information of the target object obtained in step (1), construct a unified guidance representation that matches the observation information of the target object obtained in step (2); (3-1) Extract state information from human capture demonstrations. and local geometric observation information of the target object Perform feature encoding to obtain state features and geometric features Its expression is: (5) in, and These represent the corresponding feature encoding networks. This represents the state characteristics of a human grasping demonstration. Represents the local geometric observation features of the target object; (3-2) Employing a multimodal fusion mechanism to integrate the state features With geometric features Integration into a unified guiding representation, denoted as : ① State characteristics respectively With geometric features Feature mapping and dimension alignment are performed to give both features the same or compatible feature dimensions, resulting in aligned state features. With geometric features Its expression is: (6) in, and These represent the feature mapping matrices, and These represent the bias terms; ②Then, after completing the dimension alignment, the state features are... With geometric features Joint modeling is performed, and in this implementation, the fusion module is implemented by combining feature concatenation and multilayer perceptron mapping to output a unified guided representation. Its expression is: (7) Where [·;·] represents the feature concatenation operation, ψ represents the fusion network, and the unified guidance representation This is used to characterize the correspondence between human grasping demonstrations and the local geometry of the target object.
[0024] (4) Based on the task-independent grasping diffusion prior model established in step (2), the unified guidance representation obtained in step (3) is applied. A diffusion backsampling process is introduced, which applies task-related constraints to the grasping state in the sampling update corresponding to the prior model, thereby generating a reasonable grasping pose that meets the task requirements; such as Figure 2 As shown: (4-1) The unified guiding representation obtained in step (3-2) After feature space alignment, it is embedded as conditional guiding information in the diffusion backsampling process, so that it participates in each step of the grasping state update. ① Feature space alignment: Through conditional coding networks, the unified guided representation is aligned. Transform to a feature space that is the same as or compatible with the robot's grasping state representation; ② Introducing a diffusion backsampling process: In each backsampling iteration, the aligned unified guiding representation is... With the current grasping state and target object observation Together, these serve as conditional information for updating the grasping state, guiding the grasping position and direction during the current grasping state update process. This ensures that the grasping state update process no longer relies solely on the task-independent grasping score, but is simultaneously influenced by the observation of the target object. Demonstration of human grasping Consistent guidance information constraints enhance the observation of target objects. Human grasping demonstration The coupling between the robot's grasping posture generation and the robot's grasping posture generation.
[0025] (4-2) Using the observation information of the target object obtained in step (1) and the human grasping demonstration information as joint conditions, establish the robot task-oriented grasping target distribution: (8) in, This indicates the distribution of targets that the robot is task-oriented in grasping. This represents the task-independent grasping distribution under the conditions of target object observation. This indicates the human's grasping demonstration information and the robot's grasping state. The distribution of guiding constraints; (4-3) From the robot task-oriented grasping target distribution shown in formula (8), its logarithmic gradient expression with respect to the grasping state can be obtained as follows: (9) in, Indicates the state of the grabbing process. The gradient operator has a first term on the right that represents the task-independent capture score term and a second term on the right that represents the explicit guiding term. (4-4) Based on the logarithmic gradient expression of the grasping state obtained from formula (9), in each step of backsampling, the task-independent grasping score term and the explicit guidance term are weighted and fused as the update direction of the grasping state, and the current grasping state is iteratively updated in combination with the random perturbation term. The expression is as follows: (10) in, This represents the grasping posture at the k-th iteration. The grasping posture η represents the grasping posture at the (k-1)th iteration. kThis represents the current step size parameter, where α is the guiding intensity coefficient. This represents a random perturbation term; after multiple iterations, a reasonable grasping pose that meets the task requirements is finally obtained.
[0026] (5) Perform a comprehensive quality assessment on the reasonable grasping pose generated by diffusion backsampling in step (4), and select one or more grasping poses with the highest scores as output results based on the assessment results, and send them to the robot control system, thereby realizing the robot's task-oriented grasping of the target object.
[0027] Let the task consistency scoring function be... The capture stability scoring function is: The environmental collision penalty function is The comprehensive scoring function expression is: (11) in, , and These are the weight coefficients for the task consistency item, the grasping stability item, and the collision penalty item, respectively. After comprehensively scoring the reasonable grasping pose generated in step (4) according to formula (11), one or more grasping poses with the highest scores are selected as the output results and sent to the robot control system to drive the robotic arm and gripper to complete the corresponding target grasping and subsequent operation tasks.
[0028] The following detailed description, in conjunction with specific embodiments, illustrates the invention. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the specific steps and related content described in this invention, those skilled in the art can make various modifications or applications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0029] For example, in the process of generating and executing a task-oriented grasping strategy for a cup as a target object, the first step is to input the observation information of the cup as the target object. And human grasping demonstration information corresponding to the current task. Among them, the target object observation information can be acquired by the RGB-D vision sensor, and the human grasping demonstration information is used to characterize the way a person grasps the cup, the contact area preference, the approach direction preference, and the functional part preference when performing drinking-related tasks.
[0030] As shown on the left side of Figure 3. Figure 3-a The diagram illustrates the result of a human grasping a cup. (By...) Figure 3-aIt is known that when performing tasks related to drinking water, humans typically prioritize touching the outer area of the cup or the handle, while avoiding the rim. This is because a cup has distinctly different functional areas: the rim, the body, and the handle. The rim is directly related to the subsequent drinking action; prioritizing the rim can easily interfere with the drinking process. The outer area and handle, on the other hand, are more conducive to maintaining gripping stability and ensuring the feasibility of subsequent actions. Therefore, human grasping demonstrations can provide effective task guidance information for generating robot task-oriented grasping mechanisms.
[0031] In this process, the state information captured by humans during the demonstration and the local geometric observation information of the target object are respectively encoded to obtain state features. and geometric features Then, a unified guidance representation is obtained through the multimodal fusion module. This information, as conditional guidance, is introduced into the diffusion backsampling process to constrain the direction of the grasping state update, thereby gradually generating a reasonable grasping pose that meets the requirements of the drinking task. For example... Figure 2 As shown.
[0032] Furthermore, information observed from the target object of the cup. and human-captured demonstration information To establish the distribution of robot task-oriented grasping targets, the following conditions are considered: (1) Furthermore, the grasping state is updated progressively during the backsampling process, thereby ensuring that the generated result simultaneously meets the requirements of geometric graspability of the target object and semantic consistency of the task.
[0033] After the pose is reasonably captured and generated, it is input into the comprehensive scoring module and then scored according to the comprehensive scoring function. (2) The grasping results generated through diffusion backsampling are comprehensively evaluated. The highest-scoring grasping poses are selected as output and sent to the robot control system to drive the robotic arm and gripper to complete the corresponding target grasping and subsequent operations. The task consistency score measures the degree of matching between the grasping posture and the drinking task requirements; the grasping stability score measures the cup's ability to remain stable during grasping; and the collision penalty score suppresses grasping postures that obstruct the cup opening or hinder subsequent drinking operations. Figure 3-b As shown, the grasping results generated by diffusion backsampling based on the local point cloud observation information of the target object and after comprehensive evaluation are mainly distributed in the outer area of the cup body or the cup handle area, indicating that the present invention can effectively guide the robot to generate task-oriented grasping results that take into account both task consistency and grasping stability.
Claims
1. A robot task-oriented grasping generation method based on a human-guided diffusion model, characterized in that... It includes the following steps: (1) Obtain the observation information of the target object and the human grasping demonstration information corresponding to the current task, and establish the robot grasping state representation to establish the local geometric observation information of the target object, the human demonstration information and the robot grasping variables; (2) Based on the target object observation information obtained in step (1) and the robot grasping state samples in the publicly available CONG dataset, establish a task-independent grasping diffusion prior model to learn the task-independent grasping prior distribution of the robot grasping state under the given target object observation conditions. (3) Based on the human grasping demonstration information and the local geometric observation information of the target object obtained in step (1), construct a unified guidance representation that matches the observation information of the target object obtained in step (2). ; (4) Based on the task-independent grasping diffusion prior model established in step (2), the unified guidance representation obtained in step (3) is applied. A diffusion backsampling process is introduced to impose task-related constraints on the grasping state in the sampling update corresponding to the prior model, thereby generating a reasonable grasping pose that meets the task requirements. (5) Perform a comprehensive quality assessment on the reasonable grasping pose generated by diffusion backsampling in step (4), and select one or more grasping poses with the highest scores as output results based on the assessment results, and send them to the robot control system, thereby realizing the robot's task-oriented grasping of the target object.
2. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... The observation information of the target object in step (1) is denoted as This includes at least one information item from the target object's point cloud, depth image, local geometric features, normal vector information, and 3D mesh model; the human grasping demonstration information is denoted as... It is used to characterize the grasping behavior characteristics of a person when performing a corresponding task, including at least one of the following information items: the hand contact area, approach direction, grasping orientation, functional part preference, and operation status information when performing the corresponding task.
3. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... The establishment of the robot grasping state representation in step (1) specifically refers to: in order to provide a unified parameterized representation of the robot's six-degree-of-freedom parallel gripper grasping state for subsequent diffusion modeling and backsampling updates, the robot grasping state is represented as: H=(t,g,w) (1) Where t represents the spatial translation parameter of the gripper, g represents the attitude parameter of the gripper, and w represents the opening width of the gripper. This indicates the gripping state of the robot's parallel gripper.
4. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... The robot grasping state sample includes at least the spatial position, orientation, and gripper opening width information of the robot's six-degree-of-freedom parallel gripper.
5. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... In step (2), establishing a task-independent grasping diffusion model to learn the robot grasping distribution under given target object observation conditions specifically refers to: ① Define the robot grasping distribution under given target object observation conditions as the task-independent grasping distribution. During training, the grasping state of the robot's parallel gripper was monitored. Add Gaussian noise to obtain a noisy capture state. Its expression is: (2) in, The noise scale is represented by ε, which represents standard Gaussian noise. Represents the identity matrix; ② Noisy capture state based on formula (2) Construct a noise conditional fractional network This approximates the noisy conditional distribution with respect to the fractional function of the grasping variable, i.e.: (3) in, The noise scale is represented as Distribution of noisy capture conditions at that time Represents a noise-conditional fractional network. Represents network parameters, Indicates the state of noisy capture. The gradient operator; ③ The noise conditional score network is trained using the denoising score matching loss function: (4) in, The noise scale-related weighting coefficients are: The denoising score matching loss function is represented; the task-independent grasping diffusion prior model is composed of formulas (2) to (4), which is a diffusion modeling network; ④ After training, the task-independent grasping diffusion prior model is used to learn and approximate the observation information of a given target object. Task-independent grasping prior distribution of robot grasping state under given conditions In the subsequent diffusion backsampling process, a noise-conditional fractional network is used. The conditional fractional function of the robot's grasping state is estimated, and the robot's grasping state is iteratively updated accordingly.
6. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... Step (3) specifically includes the following: (3-1) Extract state information from human capture demonstrations. and local geometric observation information of the target object Perform feature encoding to obtain state features and geometric features Its expression is: (5) in, and These represent the corresponding feature encoding networks. This represents the state characteristics of a human grasping demonstration. Represents the local geometric observation features of the target object; (3-2) Employing a multimodal fusion mechanism to integrate the state features With geometric features Integration into a unified guiding representation, denoted as ,Right now: ① State characteristics respectively With geometric features Feature mapping and dimension alignment are performed to give both features the same or compatible feature dimensions, resulting in aligned state features. With geometric features Its expression is: (6) in, and These represent the feature mapping matrices, and These represent the bias terms; ②Then, after completing the dimension alignment, the state features are... With geometric features Joint modeling is performed, and in this implementation, the fusion module is implemented by combining feature concatenation and multilayer perceptron mapping to output a unified guided representation. Its expression is: (7) Where [·;·] represents the feature concatenation operation, ψ represents the fusion network, and the unified guidance representation This is used to characterize the correspondence between human grasping demonstrations and the local geometry of the target object.
7. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... Step (4) specifically refers to: (4-1) The unified guiding representation obtained in step (3) After feature space alignment, it is embedded as conditional guiding information in the diffusion backsampling process, so that it participates in each step of the grasping state update. (4-2) Using the observation information of the target object obtained in step (1) and the human grasping demonstration information as joint conditions, establish the robot task-oriented grasping target distribution: (8) in, This indicates the distribution of targets that the robot is task-oriented in grasping. This represents the task-independent grasping distribution under the conditions of target object observation. This indicates the human's grasping demonstration information and the robot's grasping state. The guiding constraint distribution; (4-3) From the robot task-oriented grasping target distribution shown in formula (8), its logarithmic gradient expression with respect to the grasping state can be obtained as follows: (9) in, Indicates the state of the grabbing process. The gradient operator has a first term on the right that represents the task-independent capture score term and a second term on the right that represents the explicit guiding term. (4-4) Based on the logarithmic gradient expression of the grasping state obtained from formula (9), in each step of backsampling, the task-independent grasping score term and the explicit guidance term are weighted and fused as the update direction of the grasping state, and the current grasping state is iteratively updated in combination with the random perturbation term. The expression is as follows: (10) in, This represents the grasping posture at the k-th iteration. The grasping posture η represents the grasping posture at the (k-1)th iteration. k This represents the current step size parameter, where α is the guiding intensity coefficient. This represents a random perturbation term; after multiple iterations, a reasonable grasping pose that meets the task requirements is finally obtained.
8. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 7, characterized in that... The step (4-1) specifically refers to: ① Feature space alignment: through a conditional coding network, the unified guided representation is aligned. ① Transform to a feature space that is the same as or compatible with the robot's grasping state representation; ② Introduce a diffusion backsampling process: In each backsampling iteration, the aligned unified guided representation is transformed... With the current grasping state and target object observation Together, these serve as conditional information for updating the grasping state, guiding the grasping position and direction during the current grasping state update process. This ensures that the grasping state update process no longer relies solely on the task-independent grasping score, but is simultaneously influenced by the observation of the target object. Demonstration of human grasping Consistent guidance information constraints enhance the observation of target objects. Human grasping demonstration The coupling between the robot's grasping posture generation and the robot's grasping posture generation.
9. The robot task-oriented grasping generation method based on a human-guided diffusion model according to claim 1, characterized in that... Step (5) specifically refers to: Let the task consistency scoring function be... The capture stability scoring function is: The environmental collision penalty function is The comprehensive scoring function expression is: (11) in, , and These are the weight coefficients for the task consistency item, the grasping stability item, and the collision penalty item, respectively. After comprehensively scoring the reasonable grasping pose generated in step (4) according to formula (11), one or more grasping poses with the highest scores are selected as the output results and sent to the robot control system to drive the robotic arm and gripper to complete the corresponding target grasping and subsequent operation tasks.