Open vocabulary task guiding grabbing method and device, medium and program product

Abstract

The invention provides an open vocabulary task guiding grabbing method and device, a medium and a program product. The open vocabulary task guiding grabbing method comprises the steps that a user language instruction and a complex scene image are input into a preset visual language model; identifying a target task corresponding to the user language instruction through a preset visual language model, matching a target object associated with the target task in the complex scene image, and determining an optimal grabbing component of the target object based on a plurality of functional components of the target object; constructing a plurality of capture candidate schemes by adopting the optimal capture component and the complex scene image; and performing optimization solution on each grabbing candidate scheme to determine the optimal grabbing pose, so as to solve the problem that the existing method is poor in effect when dealing with an unseen object and a new task in a scene with open vocabularies due to the limited scale of an operability memory bank in the related technology. The invention aims at solving the problem that the existing method is poor in effect when dealing with the unseen object and the new task in the scene with the open vocabularies.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence technology, and particularly relates to an open-vocabulary task-oriented grasping method, device, medium, and program product. Background Art

[0002] Inferring the operability of an object and performing grasping in a task-oriented manner is the key for a robot to successfully complete an operation task. Operability combines the functional characteristics of an object to indicate where and how to grasp the object, laying the foundation for efficient task-oriented grasping. For example, when performing a pouring task, the robot must identify the handle of the cup as an operable area to ensure the success of the operation. This highlights the need for task-oriented grasping based on operability, which is similar to the way humans interact with objects (such as grasping the handle of a spoon or a racket), and can generalize such cognition to new objects.

[0003] Existing task-oriented methods usually rely on training data limited to specific tasks and objects, and it is difficult to generalize to new objects and complex scenarios. Some methods adopt object operability transfer to guide the robot to select the grasping position. For example, RAM, Robo-ABC, etc. build an "operability memory bank" to learn the operability of objects in the training data, and then transfer the operability by retrieving the object most similar to the current test object. However, these methods that rely on the retrieval of similar objects are usually very time-consuming; and due to the limited scale of the operability memory bank, these methods do not perform well in open-vocabulary scenarios when dealing with unseen objects and new tasks. Summary of the Invention

[0004] The purpose of this application is to provide an open-vocabulary task-oriented grasping method, device, medium, and program product, aiming to solve the problem that in related technologies, due to the limited scale of the operability memory bank, these methods do not perform well in open-vocabulary scenarios when dealing with unseen objects and new tasks.

[0005] According to the first aspect of this application, an open-vocabulary task-oriented grasping method is provided, including: inputting a user language instruction and a complex scene image into a preset vision-language model; identifying, by the preset vision-language model, a target task corresponding to the user language instruction, matching a target object associated with the target task in the complex scene image, and determining an optimal grasping component of the target object based on multiple functional components of the target object; constructing multiple grasping candidate solutions using the optimal grasping component and the complex scene image; and performing optimization solution on each of the grasping candidate solutions to determine an optimal grasping pose.

[0006] In an alternative embodiment, before the step of inputting the user language instruction and the complex scene image into the preset vision-language model, the method further includes: when receiving the user's ambiguous language instruction and the complex scene image, identifying the user's intention according to the user's ambiguous language instruction and the complex scene image, and generating a user language instruction.

[0007] In an alternative embodiment, the step of identifying the target task corresponding to the user language instruction through the preset vision-language model, matching the target object associated with the target task in the complex scene image, and determining the best grasping component of the target object based on multiple functional components of the target object includes: identifying the target task and task requirements corresponding to the user language instruction through the preset vision-language model; determining the object structure associated with the target task and task requirements based on the target task and task requirements; matching the object in the complex scene image that is consistent with the object structure to generate a target object; dividing each functional component of the target object according to a first preset rule and constructing a functional component set; identifying the operability of each functional component and constructing an operability set; and selecting the functional component with the highest grasping operability from the functional component set and the operability set based on the target task and the target object, and determining it as the best grasping component.

[0008] In an alternative embodiment, the open-vocabulary task-oriented grasping method further includes: dividing each object in the complex scene image according to a second preset rule and obtaining the bounding box of the target object; converting the image corresponding to the target object within the bounding box into an object mask image; and calculating the operability mask of the best grasping component on the object mask image.

[0009] In an alternative embodiment, the step of constructing multiple grasping candidate solutions by using the best grasping component and the complex scene image includes: obtaining the depth image corresponding to the complex scene image and the camera intrinsic parameters; inputting the operability mask corresponding to the best grasping component, the depth image corresponding to the complex scene image, and the camera intrinsic parameters into a preset grasping pose generation model; converting the depth image into a partial perspective point cloud by using the camera intrinsic parameters; filtering the partial perspective point cloud by using the operability mask corresponding to the best grasping component to generate a target point cloud; and constructing multiple grasping candidate solutions based on the target point cloud, where each grasping pose candidate is represented as the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

[0010] In an alternative embodiment, the step of optimizing and solving each of the grasping candidate solutions to determine the optimal grasping pose includes: obtaining the confidence scores corresponding to the grasping poses in each of the grasping candidate solutions; sorting all the grasping candidate solutions from high to low according to the ratio of the confidence score to the distance between the translation vector in the grasping candidate solution and the center of the manipulability mask; determining the optimal grasping pose based on the grasping pose in the grasping candidate solution with the highest ranking; and performing the operation of the optimal grasping pose in a preset scenario.

[0011] In an alternative embodiment, the open-vocabulary task-oriented grasping method further includes: constructing an object knowledge base according to the physical attributes of a preset object, the object tilt angle, and the object tilt angle threshold; constructing a robot grasping performance test set based on each object in the object knowledge base and a plurality of preset grasping scenarios; and using the robot grasping performance test set to select the objects in the complex scene image that are consistent with the structure of each object to determine the target object.

[0012] A control device is provided in a second aspect of the present application, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the open-vocabulary task-oriented grasping method described in any one of the above.

[0013] A computer-readable storage medium is provided in a third aspect of the present application, on which a computer program is stored, and the computer program, when executed, implements the open-vocabulary task-oriented grasping method described above.

[0014] A computer program product is provided in a fourth aspect of the present application. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the open-vocabulary task-oriented grasping method described above.

[0015] The present application utilizes the inference ability of a preset vision-language model to infer the target task and the manipulability of the object for achieving the target task in a user language instruction and a cluttered environment, and generate a task-specific grasping pose. By integrating context-based manipulability reasoning and visual manipulability localization, it can effectively parse user ambiguous instructions, identify task-related objects, and locate the component-level manipulability area to generate accurate grasping poses. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the related art. Obviously, the accompanying drawings in the following description are certain embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0017] Figure 1 is a schematic diagram showing the difference in steps between the present application and the existing task-oriented grasping method;

[0018] Figure 2 is a flowchart of the steps of an open-vocabulary task-oriented grasping method provided in Embodiment 1 of the present application;

[0019] Figure 3 is a flowchart of the steps of an open-vocabulary task-oriented grasping method provided in Embodiment 2 of the present application;

[0020] Figure 4 is a schematic diagram showing the generation of task-oriented grasping by AffordGrasp provided in Embodiment 2 of the present application;

[0021] Figure 5 is a schematic diagram of contextually operable reasoning provided in Embodiment 2 of the present application;

[0022] Figure 6 is a schematic diagram of grasping simulation in a cluttered scene provided in Embodiment 2 of the present application;

[0023] Figure 7 is a schematic diagram of grasping in a cluttered scene in a real-world experiment provided in Embodiment 2 of the present application;

[0024] Figure 8 is a schematic diagram of case analysis of a mug in a real-world experiment provided in Embodiment 2 of the present application. Detailed implementation manners

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0026] Please refer to Figure 1 , Figure 1 which is a schematic diagram showing the difference in steps between the present application and the existing task-oriented grasping method.

[0027] In related technologies, open-vocabulary grasping has received increasing attention. These methods generate grasping poses by jointly training a vision-language grasping framework or by leveraging the visual localization ability of vision-language models (VLMs). However, they only focus on generating grasping poses for the entire object, ignoring the task-related operability of the object. To achieve more refined grasping, some methods combine open-vocabulary object detection with part segmentation techniques and perform part-level grasping based on part prompts provided by the user. These methods usually rely on the explicit object names in the prompts, making it difficult to handle vague user instructions (such as "I'm thirsty") and lacking the ability to interpret task objectives and object operability in complex open-vocabulary environments. Therefore, this application designs an open-vocabulary task-oriented grasping method, device, medium, and program product to effectively solve the above problems.

[0028] Figure 2 It is a flowchart of the steps of an open-vocabulary task-oriented grasping method provided in the first embodiment of this application.

[0029] An open-vocabulary task-oriented grasping method provided in this application may include the following steps:

[0030] Step 101: Input the user's language instruction and the complex scene image into a preset vision-language model.

[0031] In the embodiment of this application, the complex scene image may refer to an RGB image obtained by a camera in a fixed location. The preset vision-language model (VLMs) may refer to a multi-modal artificial intelligence system that combines visual perception and natural language processing capabilities and can, through joint training of image and text data, achieve cross-modal tasks such as image description generation, visual question answering, text-guided image generation, etc. Its core advantage lies in significantly improving the generalization ability of the model through large-scale pre-training (especially unsupervised learning using vision-language correlation), reducing the dependence on manually labeled data, and supporting zero-shot transfer to new tasks.

[0032] For example, the preset vision-language model may use the GPT-4o large language model to parse the user's language instruction and the complex scene image.

[0033] In this embodiment, by the user issuing a user language instruction and obtaining the complex scene image corresponding to the user language instruction and inputting the user language instruction and the complex scene image into the preset vision-language model, it is convenient for the preset vision-language model to perform recognition.

[0034] Step 102: Identify the target task corresponding to the user's language instruction through the preset vision-language model, match the target object associated with the target task in the complex scene image, and determine the best grasping part of the target object based on multiple functional parts of the target object.

[0035] In the embodiments of the present application, the target task may refer to the task that needs to be implemented according to the user language instruction provided by the user, such as language instructions like "I want to scoop something up".

[0036] In this embodiment, when the preset vision-language model receives the user language instruction, it analyzes the user language instruction to obtain the meaning that the user wants to express, that is, the target task. After analyzing the user language instruction, it matches the target object associated with the target task in the complex scene image, that is, in the complex scene image, understands the user's intention and searches for an object close to the user's intention in the complex scene image, thereby improving the intelligent recognition effect.

[0037] After obtaining the target object, the target object is divided into multiple functional components, and among the multiple functional components, the preset vision-language model determines the component that is best to grasp according to these functional components.

[0038] Step 103: Use the best grasping component and the complex scene image to construct multiple grasping candidate solutions.

[0039] In this embodiment, after selecting the best grasping component, use the complex scene image to construct multiple grasping candidate solutions, and facilitate the final grasping according to the path of the grasping candidate solutions.

[0040] Step 104: Optimize and solve each grasping candidate solution to determine the best grasping pose.

[0041] The embodiments of the present application can use the 6D grasping generation model AnyGrasp trained based on a large-scale dataset to determine the best grasping pose.

[0042] In the embodiments of the present application, select the best grasping candidate solution among multiple grasping candidate solutions and determine the best grasping pose, so as to grasp the object.

[0043] Utilize the reasoning ability of the preset vision-language model to infer the target task and the operability of the object to achieve the target task in the user language instruction and the cluttered environment, and generate a task-specific grasping pose. By integrating context-based operability reasoning and visual operability localization, it can effectively parse the user's ambiguous instructions, identify the objects related to the task, and locate the component-level operable area to generate an accurate grasping pose.

[0044] Specifically, this application establishes the association between tasks, objects, and operability in complex scenarios by leveraging the inherent reasoning ability of a pre-set Visual Language Model (VLMs). Secondly, based on the operability reasoning results, the VLMs are used to locate the target object and its visually operable area for generating grasping poses, thus eliminating the need for additional training data. Finally, relying on the in-depth understanding of tasks and operability, the method of this application outperforms existing methods in cluttered scenarios, achieving robust and context-aware grasping.

[0045] Please refer to Figures 3 - 8 , Figure 3 FIG. [0000112] is a flowchart of the steps of an open-vocabulary task-oriented grasping method provided in the second embodiment of this application. This application provides an open-vocabulary task-oriented grasping method, which may include the following steps:

[0046] Step 201, when receiving a user's fuzzy language instruction and a complex-scene image, identify the user's intention according to the user's fuzzy language instruction and the complex-scene image, and generate a user language instruction.

[0047] It should be noted that the user's fuzzy language instruction refers to, including but not limited to, the language in a traditional knowledge base, and most of them are based on a user's casual and ambiguous utterance, such as a fuzzy language instruction like "I want to scoop something up", etc.

[0048] In this example, GPT-4o is used to perform context-based operability reasoning on the user's fuzzy language instruction. At the same time, according to the complex-scene image, identify the user's intention required by the task, and generate a user language instruction.

[0049] Step 202, input the user language instruction and the complex-scene image into a pre-set visual language model.

[0050] This step is the same as step 101 and will not be elaborated here.

[0051] Please refer to Figures 4 - 5 Please refer to Figure 4 , Figure 4 FIG. [0000127] is a schematic diagram of generating task-oriented grasping provided by the second embodiment of this application. Figure 5 FIG. [0000128] is a schematic diagram of context-based operability reasoning provided by the second embodiment of this application.

[0052] Step 203, identify the target task corresponding to the user language instruction through the pre-set visual language model, match the target object associated with the target task in the complex-scene image, and determine the best grasping part of the target object based on multiple functional parts of the target object.

[0053] In an alternative embodiment, step 203 includes the following steps S11-S16:

[0054] S11. Identify the target task and task requirements corresponding to the user's language instruction through a preset vision-language model;

[0055] S12. Based on the target task and task requirements, determine the object structure associated with the target task and task requirements;

[0056] S13. Match the objects in the complex-scene image that are consistent with the object structure to generate the target object;

[0057] S14. Divide each functional component of the target object according to the first preset rule and construct a set of functional components;

[0058] S15. Identify the operability of each functional component and construct a set of operabilities;

[0059] S16. Based on the target task and the target object, select the functional component with the highest grasping operability from the set of functional components and the set of operabilities, and determine it as the best grasping component.

[0060] In this embodiment, given the user's language instruction and the RGB image of the current scene , input the user's language instruction and the RGB image of the current scene into GPT-4o, and analyze the user's language instruction based on GPT-4o to extract the clear task objective and task requirements According to the given user's language instruction and the RGB image of the current scene First, analyze the user's language instruction through GPT-4o to extract the clear target task and task requirements

[0061] Exemplarily, the user inputs the vague language instruction "I want to scoop something, please help me select a suitable tool" to GPT-4o. GPT-4o identifies the user's vague language instruction, obtains the user's language instruction, and determines the target task as scooping something by analyzing or inferring the user's language instruction. After analyzing the target task, it is known that a tool with a concave structure is usually required to collect or transfer materials when scooping something, so the tool with a concave structure is determined as the associated object.

[0062] In this embodiment, after analyzing the target task and task requirements by GPT-4o, through the RGB image of the current scene determine the object structure (concave structure) that is most suitable for the target task (scooping something) and task requirements, that is, adapted to the instruction "I want to scoop something, please help me select a suitable tool". In the RGB image of the current scene Among them, according to the determination of the object structure associated with the target task and task requirements, the object to be operated is matched, and the target object is generated, such as the RGB image of the current scene "wooden spoon" in

[0063] Exemplarily, based on the target task and task requirements in the RGB image of the current scene identify the object most relevant to the task

[0064] In this embodiment, each selected functional component is decomposed into multiple component sets

[0065] In this example, according to the operability of the functional components, the operability set corresponding to each component is inferred

[0066] In this embodiment, according to the target task and task requirements for adaptability, select the best component p * and its operability a * . The whole reasoning process can be expressed as:

[0067]

[0068] In an alternative embodiment, the open-vocabulary task-oriented grasping method further includes the following steps S21-S23:

[0069] S21. Divide each object in the complex scene image according to the second preset rule, and obtain the bounding box of the target object;

[0070] S22. Convert the image corresponding to the target object within the bounding box into an object mask image;

[0071] S23. Calculate the operability mask of the best grasping component on the object mask image.

[0072] It should be noted that this application uses the open-vocabulary part segmentation method VLPart for visual operability localization, enabling it to handle open-vocabulary scenarios. Using the RGB image of the current scene and the operability inference result as input, output the object bounding box and the operability mask on the object part. This process is divided into the following two steps:

[0073] 1) Use VLPart to locate the bounding box of the object and generate an object mask image

[0074]

[0075]

[0076] 2) Based on the masked image, utilize the optimal component p * and its operability a * , and predict the operable component mask on the object masked image

[0077]

[0078] By the above two steps, the interference caused by other objects in the cluttered scene is effectively eliminated.

[0079] It is worth mentioning that the first preset rule refers to the rule for dividing the target object according to the functional parts of the target object, while the second preset rule refers to the rule for dividing all the objects in the image according to the various different objects in the complex scene image.

[0080] Step 204: Use the optimal grasping component and the complex scene image to construct multiple grasping candidate solutions.

[0081] In an alternative embodiment, step 204 includes the following steps S31 - S35:

[0082] S31: Obtain the depth image corresponding to the complex scene image and the camera internal parameters;

[0083] S32: Input the operable mask corresponding to the optimal grasping component, the depth image corresponding to the complex scene image, and the camera internal parameters into the preset grasping pose generation model;

[0084] S33: Use the camera internal parameters to convert the depth image into a partial perspective point cloud;

[0085] S34: Filter the partial perspective point cloud with the operable mask corresponding to the optimal grasping component to generate a target point cloud;

[0086] S35: Based on the target point cloud, construct multiple grasping candidate solutions, and each grasping pose candidate is represented as the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

[0087] It should be noted that AnyGrasp utilized in this application can generate grasping poses from the partial perspective point cloud.

[0088] The rotation matrix refers to a 3×3 orthogonal matrix that describes the rotation relationship between the world coordinate system and the camera coordinate system, and its core function is to convert points in the world coordinate system to the camera coordinate system.

[0089] ​The confidence score refers to a quantitative indicator that measures the reliability of a model or system's prediction results. It is usually represented by a score ranging from 0 to 1 (or 0% to 100%). The higher the score, the stronger the model's confidence in the result. In artificial intelligence and data analysis, it is widely used in scenarios such as question - answering systems, object detection, and document parsing. For example, in a question - answering system (such as Microsoft QnA), the confidence reflects the relevance between the user's query and the answer. A high score (e.g., >0.7) means that the answer highly matches the question. In object detection (such as YOLO), the confidence combines the probability of object existence and the accuracy of location. For example, 0.95 indicates that the model is confident that there is an object within the predicted bounding box and the location is accurate. In document processing, it evaluates the reliability of field extraction or table structuring. A low score may indicate data anomalies that require manual review.

[0090] The translation vector can refer to a vector that describes the direction and distance of translational motion. Its direction is the same as the direction of the object's movement, and its length is equal to the distance of movement. For example, after translating the point P(x, y, z) by the translation vector T=(a, b, c), the new coordinates are P′(x + a, y + b, z + c). It keeps the shape, size, and direction of the figure unchanged and only changes the position. It is widely used in fields such as physics (describing object displacement) and computer graphics (coordinate system transformation).

[0091] The center of the operability mask refers to a system or mechanism used to manage, configure, and coordinate mask - related operations. Its core function is to achieve precise control of data, networks, or hardware resources through binary bit - masking technology.

[0092] This application uses AnyGrasp to generate 6D grasping poses. First, the depth image is converted into a partial - view point cloud using the camera's internal parameters. Subsequently, the visual operability localization module is used to generate an operability mask to filter the point cloud and generate the target point cloud, thus achieving task - oriented grasping generation. This strategy constrains the grasping generation process and effectively avoids interference from surrounding objects.

[0093] This application uses AnyGrasp to process the object - part point cloud corresponding to grasping operability and generate grasping candidates. Each grasping pose g is represented as:

[0094]

[0095] where, is the rotation matrix, is the translation vector, represents the minimum width of the gripper. After generating a set of grasping - pose candidates the best grasping pose g is selected according to the following criteria * :

[0096]

[0097] Among them, score(g) and t(g) respectively represent the confidence score and translation vector of the grasping pose g. By selecting a grasping pose close to the center c = [x y z] of the manipulability mask and having a relatively high confidence score, the alignment between the grasping pose and the geometric center of the object manipulability region is ensured to improve the stability of grasping.

[0098] Step 205: Optimize and solve each grasping candidate solution to determine the optimal grasping pose.

[0099] In an alternative embodiment, step 205 includes the following steps S41 - S44:

[0100] S41: Obtain the confidence scores corresponding to the grasping poses in each grasping candidate solution;

[0101] S42: Sort all grasping candidate solutions from high to low according to the ratio of the confidence score to the distance between the translation vector in the grasping candidate solution and the center of the manipulability mask;

[0102] S43: Determine the optimal grasping pose based on the grasping pose in the grasping candidate solution with the highest ranking;

[0103] S44: Execute the operation of the optimal grasping pose in a preset scenario.

[0104] It should be noted that sorting all grasping candidate solutions from high to low means sorting them in descending order according to the score of the confidence score, and taking the grasping pose in the grasping candidate solution with the highest ranking order as the optimal grasping pose.

[0105] The preset scenario can refer to a set cluttered scenario.

[0106] In this embodiment, each grasping candidate solution has a corresponding confidence score for the grasping pose. After calculating all confidence scores and translation vectors through steps S31 - S35, all grasping candidate solutions are sorted in descending order according to the distance ratio of the manipulability mask center from high to low. From the sorting of all grasping candidate solutions, the grasping pose involved in the grasping candidate solution with the highest ranking is selected as the optimal grasping pose, and the object to be grasped for the specified task target and task requirements is grasped in the preset scenario through the optimal grasping pose.

[0107] In an alternative embodiment, the open - vocabulary task - oriented grasping method further includes the following steps S51 - S53:

[0108] S51: Construct an object knowledge base according to the physical properties of a preset object, the object tilt angle, and the object tilt angle threshold;

[0109] S52. Construct a robot grasping performance test set based on each object in the object knowledge base and multiple preset grasping scenarios;

[0110] S53. Use the robot grasping performance test set to select the objects in the complex scene image that are consistent with the structure of each object, and determine the target objects.

[0111] In this embodiment, by constructing a multi-modal object knowledge base, the physical properties of the objects (such as density, material, inertia parameters), dynamic structures (such as the movement range of articulated joints), and tilt angle thresholds are quantitatively integrated, and a structured knowledge graph (such as ArtiKG) is formed through 3D modeling, mechanical testing, and semantic association, providing accurate object characteristic references and dynamic reasoning bases for the robot; based on this knowledge base and preset diverse grasping scenarios, such as the stability test of tilted objects, the simulation of material friction interference, and the dynamic path planning of articulated structures, a test set (i.e., the robot grasping performance test set) containing complex conditions such as light changes and stacking occlusion is generated. By quantifying indicators such as grasping success rate and response time, the algorithm robustness is verified and the grasping strategy is optimized; finally, in the complex scene, by combining visual feature extraction (such as point cloud matching, texture analysis), physical property verification (such as real-time tilt angle monitoring), and semantic association reasoning (such as functional area localization), through hash index acceleration matching and dynamic priority sorting, the target objects are quickly determined and safe grasping paths are planned, reducing the false detection rate and improving the real-time performance, thus solving problems such as the risk of object dumping and multi-object interference in the complex scene environment.

[0112] In an alternative embodiment, an experimental result is provided. The specific experiment is as follows:

[0113] Implementation details: All experiments were conducted on a workstation equipped with a 24GB GeForce RTX 4090 GPU. The simulation environment was built based on PyBullet, including a UR5 robotic arm equipped with a ROBOTIQ-85 gripper and an Intel RealSense L515 camera. The original images were resized to 224×224 pixels for subsequent segmentation. The real-world experimental platform included a UR5 robotic arm equipped with an RS-485 gripper and an Intel RealSense L515 camera with eye-to-hand calibration. The captured RGB-D images had a resolution of 1280×720. In the simulation experiment, GraspNet (grasping network) was used to generate grasping poses; while in the real-world experiment, AnyGrasp (any grasping) was adopted in this application to generate grasping poses.

[0114] Baseline methods: The proposed method in this application is compared with the following methods: (1) GraspNet and AnyGrasp, both of which are 6D grasping methods trained based on large-scale real datasets; (2) RAM, which generates 2D manipulability maps and elevates them to 3D for operations; (3) ThinkGrasp (Intelligent Grasp), a vision-language grasping system that uses GPT-4o for reasoning in cluttered scenes.

[0115] Evaluation metrics: The Grasp Success Rate (GSR) is used to evaluate the performance of simulation and real-world experiments, and it is calculated as the percentage of the number of successful grasps to the total number of grasps.

[0116] Table 1 Simulation experiment results of different methods in single-object grasping tasks

[0117]

[0118] Table 2 Simulation experiment results of different methods in cluttered-scene grasping tasks

[0119]

[0120] In the simulation experiment, the proposed method was compared with GraspNet and ThinkGrasp, and GraspNet was used to generate grasping poses. The experiment included two tasks: single-object grasping and cluttered-scene grasping. In each test case, all methods were run 50 times under the same scene settings. In the single-object grasping task, the object was placed alone on the table. The results in Table 1 show that the average Grasp Success Rate (GSR) of the method in this application reached 0.85, which is better than GraspNet and ThinkGrasp. For simple objects (such as spoons, hammers, bowls), the method in this application performed similarly to ThinkGrasp; while for challenging objects such as scissors and wine glasses, the method in this application achieved GSRs of 0.62 and 0.66 respectively, which were 0.48 and 0.22 higher than GraspNet and 0.14 and 0.20 higher than ThinkGrasp. This indicates that the method in this application can adapt to objects with complex geometries.

[0121] In the cluttered-scene grasping task, the objects were randomly placed, and the graspable and manipulable positions of the target object were marked with red asterisks ( Figure 6 ). The quantitative results in Table 2 show that the average grasp success rate of the method in this application reached 0.77, which was significantly better than 0.54 of ThinkGrasp.

[0122] The method provided in this application performs excellently when grasping cups, spoons, hammers, and screwdrivers. For example, for a screwdriver, the GSR of ThinkGrasp is only 0.16, while the method of this application reaches 0.76. ThinkGrasp also has difficulties in handling wine glasses because it preferentially selects the most prominent object for grasping, resulting in grasping failures even when the target object is visible.

[0123] Table 3 Comparison results of different methods in real-world experiments

[0124]

[0125] In the experiment, task-oriented grasping in cluttered scenarios was focused on, and the proposed method was evaluated on a variety of objects. The results in Table 3 show that the method of this application achieved the highest average grasping success rate (GSR) of 0.83, outperforming RAM, AnyGrasp, and ThinkGrasp. For simple objects (such as bottles), all methods showed high grasping success rates. However, RAM performed poorly in handling complex geometries due to the limited scale of the operability memory bank; while ThinkGrasp showed competitiveness on specific objects (such as kettles and screwdrivers). The method of this application performed excellently in a wider range of objects, achieving grasping success rates of 0.80, 0.90, and 0.90 for cups, shovels, and pots respectively, demonstrating its robustness to objects with different shapes and functions.

[0126] Figure 7 Visualization results of the operability region localization and task-oriented grasping pose generation are shown. The relevant objects and their operability masks are annotated with cyan bounding boxes and masks in the RGB image. The grasping poses are generated on the point cloud corresponding to the operability mask and visualized on the object point cloud. As Figure 7 shown, the grasping poses are densely distributed on the object parts with grasping operability identified through GPT-4o inference.

[0127] Numerous simulations and real-world experiments highlight the key role of VLMs and operability localization in grasping tasks, demonstrating their task understanding ability and operability component recognition ability. In addition, VLMs can structure task understanding by parsing the user's vague instructions into specific tasks for target object and operability component recognition and generating grasping poses that meet the task requirements. These advantages effectively improve the grasping success rate of AffordGrasp.

[0128] For example analysis, to demonstrate the advantages of the proposed method, this application provides a case analysis of a mug, as Figure 8As shown. The first row shows the grasping pose candidates, and the second row depicts the optimal grasping poses selected by different methods. ThinkGrasp, due to its 3×3 grid strategy (dividing the image into numbered regions, 1: upper left corner, 9: lower right corner), tends to select grasping poses near the center of the cropped image. However, this method often fails when dealing with objects with curved edges (such as cups) because the center-based grasping poses lack stability due to insufficient contact points or improper force distribution. Similarly, AnyGrasp generates random grasping poses without operability reasoning, making task-oriented grasping unreliable ( Figure 8 (b) and (c)). In contrast, the present application combines operability reasoning with fine-grained grasping generation, capable of generating accurate and stable grasping poses. This highlights the importance of operability reasoning and proves the effectiveness of the present application in practical applications.

[0129] The third aspect of the present application provides a control device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement the open-vocabulary task-oriented grasping method of any one of the above embodiments.

[0130] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the open-vocabulary task-oriented grasping method is implemented.

[0131] The fifth aspect of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is made to execute the open-vocabulary task-oriented grasping method.

[0132] It can be understood that the circuit structures, names, and parameters described in the above embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above-mentioned multiple embodiments according to the usage needs, and should not limit the concept of the present application to the specific details of the above examples.

[0133] Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An open-vocabulary task-oriented grasping method, characterized in that, The open-vocabulary task-oriented grasping method includes: Inputting the user's language instruction and the complex scene image into a preset vision-language model; Identifying, through the preset vision-language model, the target task corresponding to the user's language instruction, matching the target object associated with the target task in the complex scene image, and determining the best grasping part of the target object based on multiple functional parts of the target object; Constructing multiple grasping candidate solutions by using the best grasping part and the complex scene image; Performing optimization solution on each of the grasping candidate solutions to determine the best grasping pose.

2. The open-vocabulary task-oriented grasping method according to claim 1, characterized in that Before the step of inputting the user's language instruction and the complex scene image into the preset vision-language model, it further includes: When receiving the user's fuzzy language instruction and the complex scene image, identifying the user's intention according to the user's fuzzy language instruction and the complex scene image, and generating a user's language instruction.

3. The open-vocabulary task-oriented grasping method according to claim 1, characterized in that, The step of identifying, through the preset vision-language model, the target task corresponding to the user's language instruction, matching the target object associated with the target task in the complex scene image, and determining the best grasping part of the target object based on multiple functional parts of the target object includes: Identifying, through the preset vision-language model, the target task and task requirements corresponding to the user's language instruction; Determining the object structure associated with the target task and task requirements based on the target task and task requirements; Matching the object in the complex scene image that is consistent with the object structure to generate a target object; Dividing each functional part of the target object according to a first preset rule and constructing a functional part set; Identifying the operability of each functional part and constructing an operability set; Selecting, based on the target task and the target object, the functional part with the highest grasping operability from the functional part set and the operability set and determining it as the best grasping part.

4. The open-vocabulary task-oriented grasping method according to claim 1, wherein The open-vocabulary task-oriented grasping method further includes: Dividing each object in the complex scene image according to a second preset rule and obtaining the bounding box of the target object; Converting the image corresponding to the target object within the bounding box into an object mask image; Calculating the operability mask of the best grasping part on the object mask image.

5. The open-vocabulary task-oriented grasping method according to claim 4, wherein The step of constructing multiple grasping candidate solutions by using the best grasping part and the complex scene image includes: Obtaining the depth image and camera internal parameters corresponding to the complex scene image; Inputting the operability mask corresponding to the best grasping part, the depth image corresponding to the complex scene image, and the camera internal parameters into a preset grasping pose generation model; Converting the depth image into a partial perspective point cloud by using the camera internal parameters; Filtering the partial perspective point cloud by using the operability mask corresponding to the best grasping part to generate a target point cloud; Constructing multiple grasping candidate solutions based on the target point cloud, and each grasping pose candidate is represented as the rotation matrix, translation vector, minimum width of the gripper, and confidence score of the gripper in the camera coordinate system.

6. The open-vocabulary task-oriented grasping method according to claim 5, characterized in that The step of performing optimization solution on each of the grasping candidate solutions to determine the best grasping pose includes: Obtain the confidence scores corresponding to the grasping poses in each of the grasping candidate solutions; Sort all the grasping candidate solutions from high to low according to the ratio of the confidence score to the distance between the translation vector and the center of the manipulability mask in the grasping candidate solution; Determine the optimal grasping pose based on the grasping pose in the grasping candidate solution with the highest ranking; Execute the operation of the optimal grasping pose in a preset scenario.

7. The open-vocabulary task-oriented grasping method according to claim 6, wherein The open vocabulary task-oriented grasping method further includes: Construct an object knowledge base according to the physical properties of a preset object, the object tilt angle, and the object tilt angle threshold; Construct a robot grasping performance test set based on each object in the object knowledge base and multiple preset grasping scenarios; Use the robot grasping performance test set to select the objects in the complex scene image that are consistent with the structure of each object, and determine the target object.

8. A control device, characterized in that The control device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement the open vocabulary task-oriented grasping method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the open vocabulary task-oriented grasping method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the open vocabulary task-oriented grasping method according to any one of claims 1-7.

Images